Trouble localizing some files
I'm running a workflow on 18 samples and one sample failed. It shows PAPI error code 9 and when I look at the logs it appears there was a problem localizing the BAM file with a 503 error:
ResumableDownloadException: Transfer failed after 23 retries. Final exception: b"HttpError accessing <https://storage.googleapis.com/download/storage/v1/b/fc-ff0af1e8-0796-4eb6-a4fd-b84229ff7a3d/o/956fae1f-d1bb-46c5-8d80-8d2191559e0a%2FMergeBams%2Fbce3a906-008f-4744-82a0-fe94b37c0d04%2Fcall-MarkDuplicates%2Fattempt-2%2FDL-010_tumor_gDNA_FD31210086.md.deduplicated.bam?generation=1637638856226300&alt=media>: response: <{'x-guploader-uploadid': 'ADPycdtLP9ouW0k4ZpoODiMFhWHwAuSG-HTeNvxd0NkG7E7qEH3SV23ROotg-MjFqJv4sPzxubhZH7eXu1qALpO_wl2GPp-_nA', 'content-range': 'bytes 22544384000-39002903155/78005806311', 'etag': 'CPzrrpLIrfQCEAE=', 'content-type': 'text/plain; charset=utf-8', 'x-goog-generation': '1637638856226300', 'x-goog-metageneration': '1', 'x-goog-storage-class': 'STANDARD', 'pragma': 'no-cache', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'content-disposition': 'attachment', 'date': 'Mon, 06 Dec 2021 20:26:34 GMT', 'vary': 'Origin, X-Origin', 'content-length': '19', 'server': 'UploadServer', 'status': '503'}>, content <Service Unavailable>"
I tried rerunning and while it hasn't finished yet, it looks like it's having the same problem. Have you seen something like this before or have any ideas what may be causing it? It's just weird because the other 17 samples finished no problem. Thanks!
Comments
8 comments
Hi Justin Rhoades,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
We’ll be happy to take a closer look as soon as we can!
Best,
Samantha
Hi Samantha,
I've shared https://app.terra.bio/#workspaces/blood-biopsy/DLBCL_MRD with the account you've provided. The submission ID is df3adbf7-6d4d-46a9-85e9-f3bdd12a4c67 and the workflow ID is f9467a73-5c9d-4f29-932a-b98c26174c1b. Thanks for your help!
Best,
Justin
Hi Justin Rhoades,
The PAPI Error Code 9 usually indicates an issue with insufficient disk space and/or memory, and after Googling the error message "Caught ResumableDownloadException" I came across this thread that also seems to point to the issue being related disk space.
I also noticed that the sample that fails has a much larger BAM file size than other samples in your submission. Can you try rerunning the workflow for that sample with increased disk size and memory?
Best,
Samantha
Thanks for the suggestion, Samantha. I tried doubling the memory to 16GB and also doubled disk space. It appears the task failed during localization again. I also found at least 3 samples with larger BAMs that completed the workflow without any issues. Do you think I should continue to further increase disk/memory and see if that solves it?
Best,
Justin
Hi Justin Rhoades,
Thanks for trying that. I think we can rule out disk space/memory being an issue for now since you mentioned there were samples with larger BAM files that succeeded. I wonder if it's an issue with the file itself. Can you try copying the BAM file to a new location, then rerunning the workflow with that file path instead? If that doesn't work you may need to regenerate the file.
Best,
Samantha
Hi Samantha,
Copying the file to a new location and retrying seems to have done the trick. I'm still confused about what went wrong initially since I believe a previous task read in that same file no problem but must be some weird glitch on Google's end. Thanks for the help!
Best,
Justin
Good morning, Justin Rhoades.
When using the PAPI Error 9 specific to PAPI you are usually indicating the issue is related to insufficient disk space and/or memory and after Googling the error message "Caught Resumable Download Exception" I came across this thread that also seems to point to the issue being related to disk space.
In addition, I have noticed that the BAM file size of the sample that fails is much larger than that of the other samples on your submission. Is it possible to rerun the workflow for that sample using the increased disk size and memory?
Good morning, Justin Rhoades.
When using the PAPI Error 9 specific to PAPI you are usually indicating the issue is related to insufficient disk space and/or memory and after Googling the error message "Caught Resumable Download Exception" I came across this thread that also seems to point to the issue being related to disk space.
In addition, I have noticed that the BAM file size of the sample that fails is much larger than that of the other samples on your submission. Is it possible to rerun the workflow for that sample using the increased disk size and memory?
Please sign in to leave a comment.