Understand ErrorReport
Hi! I'm not sure what this error message mean. Can anyone help with it? Thanks!
ErrorReport(rawls,http error calling uri https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2,Some(502 Bad Gateway),List(),List(),None)
Comments
23 comments
Hi Sehyun,
We'll be happy to take a look at this. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in either the icon of your workspace in the workspace list or inside the workspace dashboard (see the icon with the three dots). Let us know the workspace name, as well as the relevant submission and workflow ID(s).
Kind regards,
Jason
Hi Jason,
I added the the support account, though this workspace is linked to authorization domain (TCGA-dbGaP-authorized), FYI.
workspace name: pilot-study-credit/CNVworkflow_TCGA_LUAD
submission: 99ca401b-f7f6-4483-95b9-71a54f16b5b5
workflow ID: 1_MuTect1_PON
Thanks,
Sehyun
Hi Sehyun,
Thank you for that information. The error message is a result of recently discovered issue with DRS URLs. In short, the overall problem lies in dataguids.org being unable to scale properly for large requests of DRS objects. We’re tackling the issue on both fronts by communicating with dataguids to see where we can help improve their app performance as well as identifying measures that will improve performance on our end.
If you urgently need progress on this, you are welcome to try batching the data in such a way that reduces the overall number of requests to DRS objects. However, we do not know what the "sweet spot" is for how many DRS objects can be used as input without error at this time. We may come to better understand where this is as we communicate with dataguids, but if you are able to wait a while longer, we are in motion to set up some testing and potentially incorporate some improvements on our end that will help in getting around the dataguids limitation.
I will create a post on the Known Issues section of the forum later today, and I can provide the link to you so that you can follow the latest updates on the situation.
Kind regards,
Jason
Thanks, Jason! It'll be nice to have the updates on this issue. Please let me know the link.
Hi
Do failures with a message:
"message: Failed to parse response from Martha into a case class. Error: RuntimeException: Unexpected response resolving DRS path through Martha url https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2. Error: 500 Internal Server Error."
indicate that my job missed the "sweet spot" for DRS objects? Should I re-submit this sort of failure after waiting an appropriate time?
An example workflow id for this failure type of failure is 97fb07c5-7614-4360-9ff6-1f4a15c5b265.
Chip
Hi Chip,
I believe the "sweet spot" was referring to the number of DRS URLs you would be requesting from dataguids.org. If you are requesting 1000 URLs at this time, you could try splitting it up into smaller quantities and re-running - however, we are not sure what the smaller batch size should be to avoid the error!
Hi Sehyun and Chip,
Here is the link to the Known Issues article—apologies for the wait on this.
https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-
We will be posting an update on that post once we have done some tests to see what number of DRS objects may be consistently successful. If either of you find success with a particular number of DRS calls, please post that information in the forum to assist other users running into this issue. We are also awaiting word from our developers on whether improvements have been made on the datastage.io side, or if we are able to find a usable workaround on our end for this scaling issue.
Please follow the post for the latest updates on the issue. If you have any questions, please let us know!
Kind regards,
Jason
Hi Sehyun,
I've done a test run of 100 samples using the 1_MuTect1_Variants_Calling workflow and it succeeded. I will do a test of 200 and post the results on the previously linked Known Issues article: https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-
Kind regards,
Jason
Hi Jason,
Which dataset did you use for your test? I just tried 100 samples from my dataset (TCGA BAM files) and it still failed with the same error.
- Sehyun
Hi Sehyun,
I used samples from the same dataset. It's possible that the success is only intermittent, even at 100 samples. I will do a couple more tests to see if I can run into a failure.
Kind regards,
Jason
Hi Broad folks Jason Cerrato - we are having this same issue when we try to run on the TCGA SKCM data, with only 3 samples:
Any update on this issue? We are 100% blocked on our analysis until it is fixed. Thank you!
Hi dannykwells, can you provide us the paths of the DRS objects you are trying to resolve? And can you confirm that dwells@parkerici.org is the address you use for logging into Terra? If it isn't please let me know what the your account address is.
Many thanks,
Jason
As an example:
drs://dataguids.org/98e64bc4-92a8-40d6-abfb-36dcfe03ae05
For sample: C828.TCGA-D3-A3C6-10A-01D-A198-08.3.bam
Which links to: gs://gdc-tcga-phs000178-controlled/SKCM/DNA/WXS/BI/ILLUMINA/C828.TCGA-D3-A3C6-10A-01D-A198-08.3.bam
This is for patient: SKCM-D3-A3C6
Hi dannykwells,
Thanks for getting back to us.
Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?
1. Add GROUP_FireCloud-Support@firecloud.org to the User email field
2. Click Add User
3. Click Save
Let us know the workspace name, as well as the relevant submission and workflow IDs.
Kind regards,
Jason
Hi Jason Cerrato
Sorry for the delay. This is done.
Workspace: pici-firecloud/TCGA_SKCM_DATA copy
Hi Danny,
We'll take a look at this. Can you confirm that your DCP Framework account is linked in your Profile and hasn't expired?
https://support.terra.bio/hc/en-us/articles/360038086332
Many thanks,
Jason
Hi Jason,
Here is what I see in my profile:
That is, I have authorized access to TCGA, but no DCP access. I am actually not even sure what that is. Is it required to access TCGA through Terra?
Hi Jason,
An update, I did log into the DCP framework, but got the same error. Could the log in take time to propagate?
Hi Danny,
Many thanks for those screenshots. I'll be taking a look at this with one of our engineers and I'll get back to you as soon as I can.
Kind regards,
Jason
Hi Danny,
I've been informed that because this is TCGA, it's actually the DCF Framework that you'll want to connect to. You can read more about that here: https://support.terra.bio/hc/en-us/articles/360037648172-Accessing-TCGA-Controlled-Access-workspaces-in-Terra
Let us know if you are able to access the objects after connecting to the DCF Framework.
Many thanks,
Jason
Hi Sehyun Oh and Chip,
Can you both let me know whether you are still blocked due to this issue? We have made a number of adjustments in the interest of improving performance here. If you are still experiencing this issue please let us know the submission and workflow IDs that showcase the issue and the name of the workspace where this occurred (including sharing the workspace if possible).
Many thanks,
Jason
Hi Jason Cerrato
Thanks for the update. I checked/rerun my workflow and still got the same error.
workspace name: pilot-study-credit/CNVworkflow_TCGA_LUAD
submission: d349549e-1d90-4927-bfec-c8f446479e2f
workflow ID: 1_MuTect1_PON
Thanks,
Sehyun
Hi Jason Cerrato,
I got the same error and I shared my workspace to GROUP_FireCloud-Support@firecloud.org
Submission ID : e776bd05-22c9-466b-aa3e-3525a7ce356b
Can you suggest the solution?
Thanks!
Please sign in to leave a comment.