This document explains how to create and use a public or private Google Container Registry (GCR) Docker image from inside Terra. This guide is useful for migrating or mirroring existing images from other repositories such as DockerHub. For a primer on what Docker containers are for and related terminology, see this Dictionary entry.
Some of these instructions were adapted from Google GCR documentation.
Step 1. Set up prerequisites
Before you begin there are a few setup steps to complete. Note that you will only need to do these steps the first time you publish a Docker container image.
- Install Docker (see this tutorial for guidance).
- Install Google Cloud SDK.
- Set up a Google Cloud Billing account and Project.
Setting up in Google first You need a Google Billing account to set up a new Google Project to store your images. This project must be different than your Terra Billing Project.
- Once you have a Google Billing account, follow the "Before you begin" section of the GCR Quickstart tutorial to create a Google project and a Google container repository.
Step 2. Create or obtain a container image
To create a new image, follow the first section of this Docker image tutorial to create and tag an image.
If you already have an image and local copy to use, continue to the next step (step 2. Tag image).
If your image is stored in DockerHub and you want to migrate it to GCR, first you need to pull it locally with this command:
docker pull <REPOSITORY-NAME>/<IMAGE>:<TAG>
where <REPOSITORY-NAME> is the name of the repository where the docker is stored, <IMAGE> is the name of the image you are pulling, and <TAG> is a keyword or version number that you want to attach to identify a specific image.
Why not use the default tag? Note that the default tag is latest. While this can be sometimes convenient, we don't recommend using the "latest" tag when pipelining commands in real work, unless you're sure that you're willing to take the chance that the version you're running might change without warning. It's better practice to use explicit tags. both for pulling and pushing images.
Step 3. Tag image with registry name
Before pushing the Docker image to GCR, you need to tag it with its registry name. This configures the docker push command to push the image to a specific location. The registry name format is: gcr.io/[PROJECT-ID]/[IMAGE] where [PROJECT-ID] is your Google Cloud Platform Console project ID and [IMAGE] is your image's name.
To tag your Docker image for Container Registry, run this command:
docker tag [IMAGE] gcr.io/[PROJECT-ID]/[IMAGE]
For example:
docker tag quickstart-image gcr.io/my-project/quickstart-image
You are now ready to push your image to GCR!
Step 4. Push your image to GCR (private)
To push your Docker image to Container Registry, run this command:
docker push gcr.io/[PROJECT-ID]/[IMAGE]
For example:
docker push gcr.io/my-project/quickstart-image
You can view your freshly-pushed image in GCR by visiting the image’s registry at http://gcr.io/<PROJECT-ID>/<IMAGE>. You can also view your image where it is stored in the Google Bucket. The directory will be named artifacts.<PROJECT-ID>.appspot.com for images pushed to gcr.io/<PROJECT-ID>.
By default, Google stores your images privately so only authorized users have permission to use them. You have the option of sharing your images publicly as well; see the next step, which is optional.
Step 5. Optional: make your image public
You can also make your images public, which allows you to use them in Terra or directly through Cromwell without following any more steps in this tutorial.
To do this through the Google Console, go to the Google bucket that you want to make public.
5.1. Check the box next to the name of the bucket.
5.2. Click SHOW INFO PANEL.
5.3. Type allUsers in the Add members field.
5.4. In the Select a role drop-down menu select “Storage Object Viewer”, and click Add.
This will give pull (read-only) permissions to all users. You can also change the bucket permissions with the Google Cloud SDK.
Choosing public or private bucket permissions Note that each Google bucket is either entirely public or not; it is not possible to publicly serve only specific images. If you have specific images you want to make public while keeping the rest private, create a separate bucket and GCR to make it public.
If in the future you change an image from public to private, be aware that any pipelines that use this formerly-public image will break for anyone who does not have access to the private image.
Step 6. Share your image with a Terra group
You can share your images with a new group or an existing group within Terra. We strongly recommend that you share with a group (as opposed to an individual) if you might share your images in the future; you can easily add new people to the Terra group. As an added benefit, they will have access to all of the images that you have ever shared with that group.
6.1. Go to Terra and view your groups.
6.2. To create a new group, click “Create New Group…” and follow the instructions to create and save a group.
Then, give that group access to your GCR bucket.
6.3. Copy the email address under “Group Email”, for example, <YOUR-GROUP-NAME>@firecloud.org.
6.4. Go to the Google bucket and check the box next to the name of the bucket with the private images you want to use in Terra.
6.5. Click SHOW INFO PANEL.
6.6. Paste the group email address in the Add members field.
6.7. In the Select a role drop-down menu, select Storage Object Viewer, and click Add.
This will give pull (read-only) permissions to Terra users who are in the group.
Step 7. Use the image in a workflow or interactive analysis
You can use a custom Docker image when setting up the VM that runs an analysis in Terra.
-
The path to your image in GCR is
gcr.io/<PROJECT-ID>/<IMAGE>, for example,gcr.io/my-project/quickstart-image; use this image path in your WDL workflow. -
To use your Docker image in an interactive analysis (Galaxy, Jupyter notebook or RStudio), follow the steps below.
7.1. Select "Custom" from the Environment drop down menu.
7.2. Input the container image, using the format <image name>:<tag>.
For more details, see Understanding and adjusting your Cloud Environment.