What the title says. Backend log pasted below. Localization of the results from the preceding tasks works (file names redacted in the log), after that the VM is stuck at downloading additional files specified using DRS. This error occurs in some tasks and does not appear reproducible - for example, other tasks from the same workflow were able to localize the same DRS successfully. This particular task which the backend log is pasted for ran for almost two days with no progress. This kind of error appears periodically in our workflows.
Thank you for any help!
2023/11/03 22:35:34 Starting container setup.
2023/11/03 22:35:35 Done container setup.
2023/11/03 22:35:36 Starting localization.
2023/11/03 22:35:45 Localization script execution started...
2023/11/03 22:35:45 Localizing input gs://fc-secure-.../.../... -> /cromwell_root/some_file_name
2023/11/03 22:35:48 Localizing input gs://fc-secure-...redacted
2023/11/03 22:35:58 Localization script execution complete.
Using https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v3 to resolve DRS Objects
Attempting to download data
getm --manifest getm-manifest.json