Google Cloud's current pricing structure for multi-regional storage buckets may mean paying more for Terra workspace Buckets, which were multi-regional by default until June 13, 2022. Read on to learn how to avoid potential data transfer (egress) costs in these older workspaces by migrating from multi-region to single-region storage.
Background: Google Cloud multi-region storage pricing changes
In October 2023, Google Cloud announced changes to the pricing structure of multi-regional storage buckets: data transfer fees — normally charged when transferring data stored in one region to another region for computing — now apply when analyzing data stored in multi-regional buckets with the default Terra VMs (in us-central1).
Before the price adjustment, using multi-regional buckets (Terra's previous default setting) let you access data from compute regions in the US without extra costs. But with the new prices, it's no longer beneficial to use these buckets for (most) computations in a Terra workspace. For this reason, the default region for workspace buckets was switched to us-central1 in June 2022.
See the blog post Moving away from multi-regional storage buckets for more details.
Pricing extension (Broad billing accounts)
Terra negotiated a pricing extension with Google to keep the previous pricing structure in place through March 15, 2024, for Broad billing accounts (see the blog post here).
Non-Broad accounts
Non-Broad billing account managers can reach out to their Google representatives to determine if they are eligible to receive the same pricing extension.
Action item: Migrate existing workspaces to single-region storage
Migrate before March 15, 2024 by following the link below!! This feature is available through Terra at https://app.terra.bio/#workspace-migration.
Impact on Terra costs (multi-region workspace Buckets)
Because analysis in Terra is done on us-central1 compute VMs, when the pricing structure takes effect (March 15, 2024 for Broad billing accounts - others should reach out to their Google representative), you may incur data transfer fees every time you reference a file or run any analysis on data stored in a multi-regional storage bucket.
To assist in mitigating this cost for workspaces created prior to June 13, 2022, we have developed a self-service migration feature that will allow you to convert existing workspaces to use single-region storage buckets without breaking any links or functionality within Terra.
Why migrate
- Avoid data transfer fees: Current pricing means potential costs for operations previously free.
- Use Autoclass feature: This is also an opportunity to take advantage of Google’s Autoclass feature, which automatically moves files to colder storage, saving on storage costs. To date, the feature has not been available for existing buckets, but since this migration process recreates the bucket, the Autoclass feature is enabled at the time of migration.
For details about when you might not want to migrate your workspace buckets and what happens in this case, see the FAQs.
Migration considerations (before December 15, 2023)While we have done our best to protect bucket migrations from data transfer charges through December 15, 2023, we cannot guarantee that there will be no data transfer or other operational charges when migrating buckets. If you have any concerns, you should contact your Google Billing Representative prior to migrating your buckets.
Large buckets with hundreds of thousands of files or more may see a one-time Storage Log operational charge equaling about $2.50 for every 1 million files in the bucket.
Migration of your workspace bucket is optional, but strongly recommended to save on storage costs and reduce the risk of future data transfer charges. However, there could be significant downtime to your workspaces depending on the size of your workspace buckets that should be understood and coordinated prior to migration. Migrations cannot be stopped or paused once they begin.
Frequently Asked Questions
How do I know if my workspace buckets are multi-region?
You can see the region of your workspace bucket under the Cloud Information section on the right side of the workspace dashboard.
The migration tool will not display any buckets that are already single-region. If you have access to your project’s Google Cloud console, you can see a list of your buckets and their associated regions there as well.
What’s the migration process?
- The workspace being migrated is locked and becomes unavailable for use.
- A temporary single-region bucket is created, and all data is transferred from the workspace multi-region bucket to that temporary bucket.
- The workspace multi-region bucket is deleted.
- The workspace bucket is recreated as a single-region bucket with the same name.
- All data is transferred from the temporary single-region bucket to the workspace single-region bucket.
- The temporary bucket is deleted, and the workspace is unlocked.
Since the bucket name remains the same, no other action is required in Terra. All file references, notebooks, workflow job history, etc., should remain fully functional once the migration is complete.
What happens if I choose not to migrate my workspaces?
Participation in this migration process is optional but strongly recommended to save on storage costs and reduce the risk of future data transfer charges.
You may choose not to migrate your workspaces to take advantage of multi-regional bucket geo-redundancy (the data they contain is copied into data centers located in multiple regions). This offers a degree of protection against data unavailability or loss in case of large-scale failure (such as a disaster affecting an entire region), which might be an attractive feature for some data storage use cases.
We cannot guarantee we will be able to migrate workspaces after the planned migration windowIf you think your data will be accessed at any point in the future and don’t have a need for geo-redundancy, we strongly recommend migrating your workspace buckets before December 15, 2023.
How long will my migration take? What should I expect?
Workspaces will be locked, and their buckets will be unavailable while they are being migrated.
Factors that will impact how long a migration will take
- The number of workspaces to migrate
- The size of each workspace bucket
- The number of files in each workspace bucket
Migration times based on bucket sizes (rough estimate)
Size of Bucket |
# of files |
Estimated duration |
| <100 GB | <1 million files | 2-4 hours |
| Between 100 GB and 1 TB | Between 1 million and 10 million files | 4-8 hours |
| Between 1 TB and 10 TB | Between 10 million and 20 million files | 8-16 hours |
| Between 10 TB and 100 TB | Between 20 million and 30 million files | 16-24 hours |
| >100 TB | >30 million files | 24 hours per 100 TB or 30 million files* |
* We strongly recommend you carefully consider migrating buckets of this size until you are able to plan for several days of downtime.
My migration failed! What should I do?
If your migration fails, please write to us at support@terra.bio. Please include screenshots of the error messages, as well as the names of your affected billing project and workspace(s).
Feedback
As always, we are undertaking these changes with the goal of serving the best interests of the research community, and we’re open to your suggestions, questions, and concerns. Don't hesitate to reach out to our support team in the Terra community forum or privately through the Helpdesk.