GCP Autoclass for Terra workspaces Answered
Hi Terra,
GCP recently came out with a feature called [Autoclass](https://cloud.google.com/storage/docs/autoclass) ([blog post from Lukas](https://lukwam.medium.com/google-cloud-storage-introduces-autoclass-60feca8f4a4e))that will manage the storage classes for objects based on access frequency. This was also included in an email sent to GCP-users at the Broad as a recommendation for new buckets.
Is this feature possible to use with Terra workspaces yet? It does not seem so yet but our group would be enthusiastic if so.
Comments
12 comments
Autoclass is now available for all newly created GCP Terra workspaces! Google does not currently support turning it on for existing buckets, but we plan to support that as well as soon as they allow it (ETA is Summer 2023).
Hi Brendan Reardon,
At this time, Terra does not officially support anything other than Standard storage class. That said, it is on our Product team's roadmap for 2023 but has yet to be prioritized or scheduled. We hope to share more concrete plans on this work early in the new year.
Best,
Samantha
Thank you, Samantha!
There's a Terra support article about saving money by nearlining and coldlining files. I wish there a way to change storage class of files on Terra without having to move them to out of the Terra workspace buckets.
https://support.terra.bio/hc/en-us/articles/360029748111-Overview-Controlling-Cloud-costs-on-Terra#heading-5
Us too, Mark. Hopefully the Terra developers are able to make this a reality eventually.
Hi Samantha (she/her), huzzah! Thank you! Is there documentation on how to enable AutoClass on a Terra workspace? There does not seem to be an option yet within the web browser.
Hi Brendan Reardon,
Autoclass is enabled by default for all newly created workspace buckets. Our product team chose to take the decision out of the users hands because in almost every scenario they analyzed, Autoclass was cheaper than Standard storage.
The scenarios where it's not cheaper are not likely to be possible within the next few months while we wait for Google to enhance the feature for existing buckets. Those scenarios being 10s of millions of files present in the workspace bucket with an average file size of less than 0.33 MB. This is really only possible for high data processing workspaces, like AoU and GATK mega workspaces. Regular use cases are highly unlikely to be able to achieve this.
If there are any doubts, users can verify their workspace bucket's storage class by opening the bucket in the Google console.
Best,
Samantha
Hi Samantha,
Is there an update on implementing into Terra the ability to convert current buckets to use AutoClass? I know that Google needs to implement this on their side first but I am having trouble finding the status of their feature implementation.
Thank you!
Hi Terra and Samantha,
Google released an announcement today that AutoClass will require opt-in to enable Coldline and Archive storage. Can the Terra team update users about how this will impact workspaces, especially since AutoClass is enabled by default?
In their email, they added,
So, if Terra chooses to opt-out workspaces that have objects in lower classes, users could face unexpected large charges upon interacting with archived data.
Following up on this. Thank you,
I emailed Terra support last week, they said they are rolling out a user interface to migrate our buckets to autoclass sometime next month.
Thank you, Mark!
Please sign in to leave a comment.