This feature will be leaving preview and be generally available by 12/6/24
What we're solving
This feature gives users better controls to delete unnecessary workspace bucket files and manage cloud storage costs.
Intermediate files generated as part of running WDL workflows cost money and users want to easily delete them once they no longer need them to decrease their costs. When running workflows, many files are generated such as output files, files generated as parts, task executions (intermediate files), log files, etc. Intermediate files, specifically, are used for call caching and are stored in the workspace bucket.
In most use cases, once you complete a successful run and after using successful runs for call caching, you do not need intermediate files anymore and these files become unnecessary expenses. Manual deletion of these files, however, is not easy since you may have many submissions, and the intermediate files are usually contained in several subdirectories under each submission. Additionally, you may want to have control over when to delete intermediate files—typically wanting to delete them after a certain time (e.g. 30 days).
What's changing for you
GCP workspace buckets now support the ability to set lifecycle rules to delete files in a defined location after a defined time. This functionality lets you clean up intermediate files generated from workflows or any other files you may want to delete without having to delete the entire workspace. You must be a workspace owner to set lifecycle rules on a workspace.
Benefit to the user experience
Lifecycle rules will both save money on storage costs and let you better manage files that are produced as a part of your analysis.
Try it Out
Go to https://app.terra.bio/#feature-preview to enable the “GCP Bucket Lifecycle Rules” feature and let us know what you think!
Step-by-step instructions
1. On a workspace dashboard or from the workspace list page, click the three-dot dropdown and click “Settings” to open the Workspace Settings dialog:
2. From here, you can enable and configure the Lifecycle Rules setting:
What to expect
Note that enabling lifecycle rules will also change the directory structure for future workflow submissions by separating files into submissions/intermediates and submissions/final-outputs directories. This lets you set up a lifecycle rule to automatically delete intermediate files in the submissions/intermediates directory. You may also choose to delete files in other directories by entering your own values.