How to use the Data Uploader

Anton Kovalsky
  • Updated

Learn how to upload data files to workspace storage and a corresponding data table all at once, right in a Terra workspace. If metadata in your table matches the name of an uploaded file, the table will be updated with the full gs path to your file in workspace storage (Google bucket). This is critical since the gs path in the data table is needed to run workflows with inputs defined by a data table.

Why use the data uploader?

The data uploader automates and simplifies the process of uploading data and a corresponding data table to a workspace. While you can upload files directly in a workspace, you would usually need to add all of the URLs (the "gs://bucket-id/filename" file paths) that point to uploaded files in the table manually or with scripting. However, the Data Uploader does this for you, so you don't have to worry about programmatically generating the list of bucket URLs.

How to use the Data Uploader

Data-uploader_Steps-to-use_diagram.png
Steps: 1. Access data uploader 2. Create/choose data collection 3. Upload files 4. Upload data table (TSV)

You still need to create a TSV file to generate a table for the data To learn how, see How to make a data table from scratch or a template. Note that the TSV you upload to generate a table for your data will need to include the file name, just not the full path. In other words, the data uploader will replace your-data-file.vcf with gs:fc-3cf82dc8-62a0-4518-ac35-89e03debe3d7/your-data-file.vcf.

 


Video data uploader tutorial: Loading Illumina paired-end sequencing data to Terra

Step 1. Access data uploader and select a target workspace

You can access the Data Uploader directly, or from within a workspace.

To access directly

If you don't want to start from a workspace, go to https://app.terra.bio/#upload

To access from within a workspace

Click the Import Data button at the top of the Data page and select Open data uploader.
Data-Uploader_Access-the-data-uploader-in-the-data-tab_Screen_shot.png

When you first arrive at the data uploader you'll see a homescreen with the workspace you were in.
Data-uploader_Home-screen_Screen_shot.png

To change the destination workspace, click the Change link in the workspace card

In the selection screen, you can  search for workspaces available to you based workspace names, tags, or billing projects. Find the workspace to which you'd like to add data, and click on it to select it.
Data-uploader_Change-workspace_Screen_shot.png

Step 2. Create/choose a data collection

Once you've selected the target workspace, you'll be prompted to either create a "collection" or select an existing one.

What is a data collection?

Collections are a way to organize your data files into groups; sort of like files in local storage. You could use distinct collections if you are adding data for different organisms, different experimental methodologies, or different sequencing technologies to the same workspace. Each collection will have its own associated data table with metadata like the file ID, the URL of the data file in workspace storage, and any other useful details. 

Creating versus adding to a collection

If a new set of files is similar to an existing collection (i.e. its associated table includes roughly the same columns of metadata), you could just add to that collection. Otherwise, you can create a new collection. In the example shown in the screenshot below, the workspace doesn't have any existing collections to choose from, so we would create a new one.

2.1. Click "create a new collection" (or select a collection from an existing one, if you have any)
Data-uploader_Create-a-new-collection_Screen_shot.png

2.2. Name your collection and click "CREATE COLLECTION"
Data-uploader_name-new-collection_Screen_shot.png

How to delete a collection

Note that this will delete the data uploaded to the workspace bucket! 

1. In the workspace dashboard, click Open bucket in browser link (right column under Cloud Information).

2. In GCP console storage browser, click the Uploads file. 

3. Select the data collection to delete. 

4. Click the blue DELETE link and follow instructions. 

Delete-data-collection_Screen_capture.gif

Step 3. Upload files

Once you have a collection selected, if you scroll down you'll see an area prompting you to upload your files. 

3.1. Upload your files, either by dragging-and-dropping them on the page, or clicking the blue plus button at the bottom right to browse the files from your computer:
Data-uploader_Drag-and-drop-files_Screen_shot.png

3.2. Once the upload starts, you'll see a progress bar, along with an option to abort the upload:
Data-uploader_Upload-progress-bar_Screen_shot.png

Step 4. Upload data table (TSV)

The final step is to create a table in the workspace that lists the data and any associated metadata. You'll upload a TSV file that includes at minimum a unique ID for each file and the file names of the data you just uploaded.

To learn more, see How to make a data table from scratch or a template

4.1. After clicking "NEXT >" in the data file upload step, you'll see a prompt to upload your metadata TSV. You'll have the option to either drag-and-drop, or click the blue upload button to select files from your local machine. Note that Data Uploader will only accept .TSV or .TXT files.
Data-uploader_Drag-and-drop-TSV-files_Screen_shot.png

4.2. To complete this process, once the TSV file upload is complete, click CREATE TABLE (or UPDATE TABLE).

Data-uploader_Create-table-from-TSV-files_Screen_shot.png

What to expect in the target workspace Data page

The data files will be in workspace storage - in a directory called "uploads" in a folder named however you named the collection. You can see these files by navigating to the "Files" section under workspace Data page.

The metadata will be in a data table. 

Finding data and metadata in the workspace
Data-uploader_Files-in-storage-and-data-table_Screen_capture.gif
Data files are in workspace storage. Corresponding metadata (links to data files) are in the data table. 

 

 

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.