To improve performance, Terra archives some workflow details (metadata) after a year. Read on to understand which workflow-level details are archived and how to retrieve them if you need them.
Archiving workflow details FAQs
What gets archived?
Only information currently found in Job Manager or on the Workflow Dashboard is archived.
- Records of which - and how many - tasks were run by the workflow
- Whether the workflows succeeded
- Error messages for workflows that failed
- Lists of workflow input and outputs (but note">: generated output files themselves are not deleted from GCS)
What isn't affected?
- Information on submission history pages, including workflow IDs, final status
- Generated output files (will still be in GCS)
- Call caching isn't affected. You can still cache results from old workflows.
Why archive these details?
Archiving allows Terra to deliver the best long-term experience for all users. Though some workflow-level details for older workflows are no longer instantly available in Terra, archiving allows Terra to retrieve details for newer workflows faster and more successfully. The feature also helps avoid occasional web errors and retrieval timeouts, which you may have experienced in the past while trying to see details for some large workflows.
What happens when workflow details are archived?
The information is still stored in our archives, but is no longer indexed. The details are not visible in Terra. You need to ask us to manually retrieve the files for you to view as a CSV file format. Scroll down for detailed instructions.
When does the archiving happen?
Details for workflows that completed more than one year ago are automatically archived.
Stay tuned for future updates on how we accelerate metadata access.
How to access archived metadata
If you need a copy of workflow details for an old workflow which has been archived, please contact Terra support at firstname.lastname@example.org. Your request should include:
1. Which workflow you would like workflow details for. Use one of the following two options:
The workflow ID, submission ID and billing project for the workflow details you need
A link to the workflow dashboard currently reporting the workflow has been archived
2. A GCS bucket into which the archived workflow details file can be copied
(Note: You may need an additional step to allow Terra support to have Write permission for copying the metadata archive into this bucket)
3. Proof that you are allowed to access the workflow(s) in question
The easiest way to do this is to make a support request from an account (your Terra user id) we can verify is allowed to access the metadata.
Please note that requests for archived metadata may take up to 5 business days.