To improve performance, Terra archives some workflow details (metadata) after a year. Read on to understand what workflow-level details are archived and how to retrieve them if you need them.
Archiving workflow details FAQs
What will be archived
Only information currently found in Job Manager or on the Workflow Dashboard will be archived.
- Records of which - and how many - tasks were run by the workflow
- Whether the workflows succeeded
- Error messages for workflows that failed
- Lists of workflow input and outputs (but note: generated output files themselves are not deleted from GCS)
What won't be affected
- Information on submission history pages, including workflow IDs, final statuses
- Generated output files (will still be in GCS)
- Call caching will not be affected. You will still be able to cache results from old workflows.
Why archive these details?
Archiving allows Terra to deliver the best long-term experience for all users. Though some workflow-level details for older workflows will no longer be instantly available in the Terra UI, archiving allows Terra to retrieve details for newer workflows faster and more successfully. The feature also helps avoid occasional web errors and retrieval timeouts, which you may have experienced in the past while trying to see details for some large workflows.
What happens when workflow details are archived
The information is still stored in our archives, but is no longer indexed. The details will not be visible in the UI. You will need to ask us to manually retrieve the files for you to view as a CSV file format. Scroll down for detailed instructions.
When does the archiving happen?
Details for workflows that completed more than one year ago will be automatically archived.
Stay tuned for future updates on how we are accelerating metadata access.
How to access archived metadata
If you need a copy of workflow details for an old workflow which has been archived, please contact Terra support at firstname.lastname@example.org. Your request should include:
1. Which workflow you would like workflow details for. Use one of the following two options:
The workflow ID, submission ID and billing project for the workflow details you need
A link to the workflow dashboard currently reporting the workflow has been archived
2. A GCS bucket into which the archived workflow details file can be copied
(note: you may need an additional step to allow Terra support to have write permission for copying the metadata archive into this bucket)
3. Proof that you are allowed to access the workflow(s) in question
The easiest way to do this is to make a support request from an account (your Terra user id) we can verify is allowed to access the metadata.
Please note that requests for archived metadata may take up to 5 business days.