Understand ErrorReport

Post author
Sehyun Oh

Hi! I'm not sure what this error message mean. Can anyone help with it? Thanks!

ErrorReport(rawls,http error calling uri https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2,Some(502 Bad Gateway),List(),List(),None)

Comments

23 comments

  • Comment author
    Jason Cerrato

    Hi Sehyun,

    We'll be happy to take a look at this. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in either the icon of your workspace in the workspace list or inside the workspace dashboard (see the icon with the three dots). Let us know the workspace name, as well as the relevant submission and workflow ID(s).

    Kind regards,

    Jason

    0
  • Comment author
    Sehyun Oh
    • Edited

    Hi Jason,

    I added the the support account, though this workspace is linked to authorization domain (TCGA-dbGaP-authorized), FYI.

    workspace name: pilot-study-credit/CNVworkflow_TCGA_LUAD

    submission: 99ca401b-f7f6-4483-95b9-71a54f16b5b5

    workflow ID: 1_MuTect1_PON

    Thanks,

    Sehyun

    0
  • Comment author
    Jason Cerrato

    Hi Sehyun,

    Thank you for that information. The error message is a result of recently discovered issue with DRS URLs. In short, the overall problem lies in dataguids.org being unable to scale properly for large requests of DRS objects. We’re tackling the issue on both fronts by communicating with dataguids to see where we can help improve their app performance as well as identifying measures that will improve performance on our end.

    If you urgently need progress on this, you are welcome to try batching the data in such a way that reduces the overall number of requests to DRS objects. However, we do not know what the "sweet spot" is for how many DRS objects can be used as input without error at this time. We may come to better understand where this is as we communicate with dataguids, but if you are able to wait a while longer, we are in motion to set up some testing and potentially incorporate some improvements on our end that will help in getting around the dataguids limitation.

    I will create a post on the Known Issues section of the forum later today, and I can provide the link to you so that you can follow the latest updates on the situation.

    Kind regards,

    Jason

     

    0
  • Comment author
    Sehyun Oh

    Thanks, Jason! It'll be nice to have the updates on this issue. Please let me know the link. 

    0
  • Comment author
    Chip

    Hi

    Do failures with a message: 

    "message: Failed to parse response from Martha into a case class. Error: RuntimeException: Unexpected response resolving DRS path through Martha url https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2. Error: 500 Internal Server Error."

    indicate that my job missed the "sweet spot" for DRS objects?   Should I re-submit this sort of failure after waiting an appropriate time?  

    An example workflow id for this failure type of failure is 97fb07c5-7614-4360-9ff6-1f4a15c5b265

    Chip

    0
  • Comment author
    Sushma Chaluvadi

    Hi Chip,

    I believe the "sweet spot" was referring to the number of DRS URLs you would be requesting from dataguids.org. If you are requesting 1000 URLs at this time, you could try splitting it up into smaller quantities and re-running - however, we are not sure what the smaller batch size should be to avoid the error! 

    0
  • Comment author
    Jason Cerrato

    Hi Sehyun and Chip,

    Here is the link to the Known Issues article—apologies for the wait on this. 

    https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-

    We will be posting an update on that post once we have done some tests to see what number of DRS objects may be consistently successful. If either of you find success with a particular number of DRS calls, please post that information in the forum to assist other users running into this issue. We are also awaiting word from our developers on whether improvements have been made on the datastage.io side, or if we are able to find a usable workaround on our end for this scaling issue.

    Please follow the post for the latest updates on the issue. If you have any questions, please let us know!

    Kind regards,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Sehyun,

    I've done a test run of 100 samples using the 1_MuTect1_Variants_Calling workflow and it succeeded. I will do a test of 200 and post the results on the previously linked Known Issues article: https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-

    Kind regards,

    Jason

    0
  • Comment author
    Sehyun Oh

    Hi Jason,

    Which dataset did you use for your test? I just tried 100 samples from my dataset (TCGA BAM files) and it still failed with the same error. 

    - Sehyun

    0
  • Comment author
    Jason Cerrato

    Hi Sehyun,

    I used samples from the same dataset. It's possible that the success is only intermittent, even at 100 samples. I will do a couple more tests to see if I can run into a failure.

    Kind regards,

    Jason

    0
  • Comment author
    dannykwells

    Hi Broad folks Jason Cerrato - we are having this same issue when we try to run on the TCGA SKCM data, with only 3 samples: 

    Any update on this issue? We are 100% blocked on our analysis until it is fixed. Thank you!

    0
  • Comment author
    Jason Cerrato

    Hi dannykwells, can you provide us the paths of the DRS objects you are trying to resolve? And can you confirm that dwells@parkerici.org is the address you use for logging into Terra? If it isn't please let me know what the your account address is.

    Many thanks,

    Jason

    0
  • Comment author
    dannykwells

    As an example: 

    drs://dataguids.org/98e64bc4-92a8-40d6-abfb-36dcfe03ae05

    For sample: C828.​TCGA-D3-A3C6-10A-01D-A198-08.​3.​bam

    Which links to: gs://gdc-tcga-phs000178-controlled/SKCM/DNA/WXS/BI/ILLUMINA/C828.TCGA-D3-A3C6-10A-01D-A198-08.3.bam 

     

    This is for patient: SKCM-D3-A3C6

     

     

    0
  • Comment author
    Jason Cerrato

    Hi dannykwells,

    Thanks for getting back to us.

    Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?

    1. Add GROUP_FireCloud-Support@firecloud.org to the User email field
    2. Click Add User
    3. Click Save

    Let us know the workspace name, as well as the relevant submission and workflow IDs.

    Kind regards,

    Jason

    0
  • Comment author
    dannykwells

    Hi Jason Cerrato 

     

    Sorry for the delay. This is done. 

     

    Workspace: pici-firecloud/TCGA_SKCM_DATA copy

    Both of hte jobs failed, but here are their submission IDs: 
     
    Thanks!!
     
    danny
     
     
    0
  • Comment author
    Jason Cerrato

    Hi Danny,

    We'll take a look at this. Can you confirm that your DCP Framework account is linked in your Profile and hasn't expired?

    https://support.terra.bio/hc/en-us/articles/360038086332

    Many thanks,

    Jason

    0
  • Comment author
    dannykwells
    • Edited

    Hi Jason, 

     

    Here is what I see in my profile: 

     

    That is, I have authorized access to TCGA, but no DCP access. I am actually not even sure what that is. Is it required to access TCGA through Terra? 

     

     

    0
  • Comment author
    dannykwells

    Hi Jason, 

     

    An update, I did log into the DCP framework, but got the same error. Could the log in take time to propagate? 

    0
  • Comment author
    Jason Cerrato

    Hi Danny,

    Many thanks for those screenshots. I'll be taking a look at this with one of our engineers and I'll get back to you as soon as I can.

    Kind regards,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Danny,

    I've been informed that because this is TCGA, it's actually the DCF Framework that you'll want to connect to. You can read more about that here: https://support.terra.bio/hc/en-us/articles/360037648172-Accessing-TCGA-Controlled-Access-workspaces-in-Terra

    Let us know if you are able to access the objects after connecting to the DCF Framework.

    Many thanks,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Sehyun Oh and Chip,

    Can you both let me know whether you are still blocked due to this issue? We have made a number of adjustments in the interest of improving performance here. If you are still experiencing this issue please let us know the submission and workflow IDs that showcase the issue and the name of the workspace where this occurred (including sharing the workspace if possible).

    Many thanks,

    Jason

    0
  • Comment author
    Sehyun Oh

    Hi Jason Cerrato

    Thanks for the update. I checked/rerun my workflow and still got the same error. 

    workspace name: pilot-study-credit/CNVworkflow_TCGA_LUAD

    submission: d349549e-1d90-4927-bfec-c8f446479e2f

    workflow ID: 1_MuTect1_PON

    Thanks,

    Sehyun

    0
  • Comment author
    Sehong Theragen

    Hi Jason Cerrato,
    I got the same error and I shared my workspace to GROUP_FireCloud-Support@firecloud.org
    Submission ID : e776bd05-22c9-466b-aa3e-3525a7ce356b

    Can you suggest the solution?

    Thanks!

    0

Please sign in to leave a comment.