How to use the TDR support articles

Leyla Tarhan
  • Updated

If you're interested in using Terra on Azure, please email terra-enterprise@broadinstitute.org.

The Terra Data Repository helps you curate large datasets that can be integrated into Cloud analyses on Terra, while managing who can access your data. You can choose between multiple tools to interact with TDR, depending on your needs and background. This article guides you through which tools to use when, and where to find instructions for each step in your TDR journey.

Overview: using TDR

There are four main stages to using TDR:

  1. Set up a TDR billing profile
  2. Create a dataset and upload data
  3. Share the data
  4. Analyze the data

Different team members might work on each of these stages. For example, an administrator might set up the billing profile, a data manager might create the dataset and upload the data, a project lead might share the data, and an outside researcher might analyze the data on Terra. Because different team members have different needs and expertise, it's useful to have multiple tools available when navigating through these steps.

Tools for using TDR

If your data are stored on Azure, there are two ways to interact with TDR:

1. TDR's user interface (UI). Logging into https://data.terra.bio/ brings you to a graphical interface where you can create a dataset, view your data, create snapshots to share data, and more. This makes it easy to examine your data, and it's especially useful for those who don't have a background using API endpoints. However, the website doesn't support as many functions as the Swagger APIs.

2. Swagger API endpoints. TDR's full functionality is accessible through its Swagger API endpoints. This includes creating datasets, uploading data, creating and sharing snapshots, creating TDR billing profiles, managing permissions, creating assets, and checking the status of jobs launched from other interfaces. However, Swagger can be difficult to navigate if you're not already familiar with API endpoints.

If your data are stored on the Google Cloud, there is a third option: Zebrafish, a web-based tool that interfaces with the Swagger APIs so that you don't have to. Learn more about Zebrafish by reading How to create a TDR dataset and ingest data with Zebrafish.

Deciding which tool to use

Which of these tools should you use to manage your TDR data? The answer depends on what you’re trying to do in TDR – for example, whether you’re setting up billing or creating a dataset - and how familiar you are with APIs.

Constraints on your tools

  • Note that some tasks can only be done with Swagger APIs, while others can be done in Swagger, on the TDR web interface, or with Zebrafish.
  • The tools available to you will also depend on whether your data are stored on the Google Cloud or Azure.

Why does API familiarity matter?In general, the Swagger APIs allow you to do more things in TDR; however, if you’re not already familiar with APIs, these can be a bit challenging to use. While some functions are only available through the Swagger endpoints, we recommend using the TDR website when possible unless you've worked with APIs before. 

The rest of this article breaks down how to choose your tool for each stage of working in TDR. When different tools are available for a step, you'll find guidance on which tool to use, based on your cloud provider (Google or Azure) and your familiarity with APIs. 

Step 1. Set up a TDR billing profile

You will use APIs to set up a billing profile to cover the costs of working in TDR. The exact steps depend on whether your data are stored on Azure or Google (GCP). See How to create a TDR Billing Profile (Azure) or How to create a TDR Billing Profile (GCP) for step-by-step instructions. You can also add collaborators to your TDR billing profile.

Step 2. Create a dataset and upload data

Flow chart depicting how to decide which tool(s) to use to create a TDR dataset and upload data. The flow chart begins with the question, 'are you comfortable working with API endpoints?' If the answer is yes, the chart's next step is to write your dataset's schema in JSON, then create the dataset using APIs, then ingest data into the dataset using APIs, then update the data using APIs. If the answer is no, the chart asks another question: 'Are your data saved on the Azure Cloud or the Google Cloud?' If the answer is Azure, the chart's next step is to create a dataset and define schema through the TDR website, then ingest data into the dataset using APIs, then update the data using APIs. If the answer is Google, the next step is write your dataset's schema in JSON (including assets), then create a dataset and ingest data with Zebrafish, then update the data with Zebrafish.

2.1. Define a dataset schema

Once you’ve set up billing and are ready to upload data to TDR, the next step is to define your dataset’s schema. The schema sets up the tables that hold your data and metadata, the tables' columns and primary keys, and the relationships between tables. Setting up your schema is crucial for updating the tables later on. Learn more about schemas in Overview: Defining your TDR dataset schema.

2.2. Ingest and update data

Step 3. Share the data

To share TDR data, you'll create a snapshot — a subset of the dataset that you want to share with a particular researcher or group, frozen at the time when the snapshot was created.

Flow chart illustrating how to decide which tool(s) to use to share TDR data. The diagram starts with the question, 'are you comfortable working with API endpoints?'. If the answer is yes, the next step is to add assets using APIs, then create a snapshot using APIs. If the answer is no, the next step is to include assets in your schema when creating your dataset, then create a snapshot through the TDR web interface.

Assets are a prerequisite for creating a snapshot. Assets are subsets of the columns in your data tables that you want to include in snapshots. Learn more about assets in How to create dataset assets in TDR

Once you’ve created a snapshot, how do you decide who can access the data? See Streamlining access for approved requestors with DUOS & TDR to learn how to screen researchers who want to access your TDR dataset.

Step 4. Analyze the data

Currently, you can't export Azure-based TDR snapshots to Terra for further analysis. But our engineers are working on this functionality, so check back here for more information soon!

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.