Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Understand ErrorReport

Comments

10 comments

  • Avatar
    Jason Cerrato

    Hi Sehyun,

    We'll be happy to take a look at this. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in either the icon of your workspace in the workspace list or inside the workspace dashboard (see the icon with the three dots). Let us know the workspace name, as well as the relevant submission and workflow ID(s).

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Sehyun Oh

    Hi Jason,

    I added the the support account, though this workspace is linked to authorization domain (TCGA-dbGaP-authorized), FYI.

    workspace name: pilot-study-credit/CNVworkflow_TCGA_LUAD

    submission: 99ca401b-f7f6-4483-95b9-71a54f16b5b5

    workflow ID: 1_MuTect1_PON

    Thanks,

    Sehyun

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Sehyun,

    Thank you for that information. The error message is a result of recently discovered issue with DRS URLs. In short, the overall problem lies in dataguids.org being unable to scale properly for large requests of DRS objects. We’re tackling the issue on both fronts by communicating with dataguids to see where we can help improve their app performance as well as identifying measures that will improve performance on our end.

    If you urgently need progress on this, you are welcome to try batching the data in such a way that reduces the overall number of requests to DRS objects. However, we do not know what the "sweet spot" is for how many DRS objects can be used as input without error at this time. We may come to better understand where this is as we communicate with dataguids, but if you are able to wait a while longer, we are in motion to set up some testing and potentially incorporate some improvements on our end that will help in getting around the dataguids limitation.

    I will create a post on the Known Issues section of the forum later today, and I can provide the link to you so that you can follow the latest updates on the situation.

    Kind regards,

    Jason

     

    0
    Comment actions Permalink
  • Avatar
    Sehyun Oh

    Thanks, Jason! It'll be nice to have the updates on this issue. Please let me know the link. 

    0
    Comment actions Permalink
  • Avatar
    Chip

    Hi

    Do failures with a message: 

    "message: Failed to parse response from Martha into a case class. Error: RuntimeException: Unexpected response resolving DRS path through Martha url https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2. Error: 500 Internal Server Error."

    indicate that my job missed the "sweet spot" for DRS objects?   Should I re-submit this sort of failure after waiting an appropriate time?  

    An example workflow id for this failure type of failure is 97fb07c5-7614-4360-9ff6-1f4a15c5b265

    Chip

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hi Chip,

    I believe the "sweet spot" was referring to the number of DRS URLs you would be requesting from dataguids.org. If you are requesting 1000 URLs at this time, you could try splitting it up into smaller quantities and re-running - however, we are not sure what the smaller batch size should be to avoid the error! 

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Sehyun and Chip,

    Here is the link to the Known Issues article—apologies for the wait on this. 

    https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-

    We will be posting an update on that post once we have done some tests to see what number of DRS objects may be consistently successful. If either of you find success with a particular number of DRS calls, please post that information in the forum to assist other users running into this issue. We are also awaiting word from our developers on whether improvements have been made on the datastage.io side, or if we are able to find a usable workaround on our end for this scaling issue.

    Please follow the post for the latest updates on the issue. If you have any questions, please let us know!

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Sehyun,

    I've done a test run of 100 samples using the 1_MuTect1_Variants_Calling workflow and it succeeded. I will do a test of 200 and post the results on the previously linked Known Issues article: https://support.terra.bio/hc/en-us/community/posts/360056741832-Error-Calling-URI-Rawls-Martha-DRS-issue-

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Sehyun Oh

    Hi Jason,

    Which dataset did you use for your test? I just tried 100 samples from my dataset (TCGA BAM files) and it still failed with the same error. 

    - Sehyun

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Sehyun,

    I used samples from the same dataset. It's possible that the success is only intermittent, even at 100 samples. I will do a couple more tests to see if I can run into a failure.

    Kind regards,

    Jason

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk