Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Comments

18 comments

  • Avatar
    Jason Cerrato

    Hi Tet,

    Thanks for writing in. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?

    1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press Enter on your keyboard
    2. Click Save

    Let us know the workspace name, as well as the relevant submission and workflow IDs. Please also add any details about how you arrived at this error that would be helpful for our investigation. We'll be happy to take a look and get back to you as soon as we can.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

     

    I appreciate your help.

     

    workspace: natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling

    workflow: NGSPCA_WF
     
    I could run exactly the same workflow successfully for a sample_set contains a small number of samples. But I always get the error for a sample_set consists of 50k samples. The error occurs without even launching anything. I could only get the error message above.
     
    We added the address to share the workspace.
     
    Best regards,
    Tet
    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    Sounds good—let me know once you are able to get the workspace shared and I'll be happy to take a closer look. If the workspace is in an authorization domain, I may need to be added to that as well.

    If it exists in an auth domain and I cannot be added, let me know.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

     

    We added the account.

    I would appreciate your help with this!

     

    Best,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tetsushi,

    Can you provide a screenshot of the error message that pops up when you try to submit, including the "Details" section if there is one (click Details to see it).

    Can you also provide some screenshots of the developer tools console when you try to submit a job with 50k samples?

    1. Navigate to the page where you would submit the job
    2. Open the Developer Console on your browser. In Chrome, you go to View > Developer > Developer Tools


    3. Click on the Console tab in the Developer Console
    4. Reload the page
    5. Try to kick off a submission with 50k samples
    6. Take a screenshot of what you see in the Console page

     

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

    Thank you for your help.

    Actually, nothing pops up. Just Status turns from Qued -> Failed.

    I am trying to upload the screenshot but somehow the page refuses. So I am just copy-pasting that.

    --------------------------------------------------------------------------------------------------------------------------

    A cookie associated with a cross-site resource at http://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
    /#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF:1

    A cookie associated with a cross-site resource at https://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.

    DevTools failed to load SourceMap: Could not load content for https://fast.appcues.com/generic/main/4.9.4/appcues.main.42ce620362c114fbcfacc93c6f2260968e34d854.js.map:

    DevTools failed to load SourceMap: Could not load content for https://fast.appcues.com/generic/main/4.9.4/container.42ce620362c114fbcfacc93c6f2260968e34d854.css.map:
    VM918 react_app.js:32742 Download the React DevTools for a better development experience: https://fb.me/react-devtools
    /#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF:1

    A cookie associated with a cross-site resource at https://accounts.google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
    (index):1

    POST https://terra-bard-prod.appspot.com/api/syncProfile 503
    t.<computed> @ (index):1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    s @ 2.0e6e1db1.chunk.js:2
    Mr @ 2.0e6e1db1.chunk.js:2
    g @ 2.0e6e1db1.chunk.js:2
    next @ 3.03616164.chunk.js:1
    s @ 3.03616164.chunk.js:1
    update @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    s @ 2.0e6e1db1.chunk.js:2
    Mr @ 2.0e6e1db1.chunk.js:2
    g @ 2.0e6e1db1.chunk.js:2
    next @ 3.03616164.chunk.js:1
    s @ 3.03616164.chunk.js:1
    update @ 3.03616164.chunk.js:1
    t @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    n @ 2.0e6e1db1.chunk.js:2
    6 @ 3.03616164.chunk.js:1
    i @ (index):1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    i @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    i @ (index):1
    r @ (index):1
    t @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    Show 66 more frames
    (index):1

    GET https://api.firecloud.org/api/nih/status 404
    t.<computed> @ (index):1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    Jr.getNihStatus @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    s @ 2.0e6e1db1.chunk.js:2
    Mr @ 2.0e6e1db1.chunk.js:2
    g @ 2.0e6e1db1.chunk.js:2
    next @ 3.03616164.chunk.js:1
    s @ 3.03616164.chunk.js:1
    update @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    s @ 2.0e6e1db1.chunk.js:2
    Mr @ 2.0e6e1db1.chunk.js:2
    g @ 2.0e6e1db1.chunk.js:2
    next @ 3.03616164.chunk.js:1
    s @ 3.03616164.chunk.js:1
    update @ 3.03616164.chunk.js:1
    t @ 3.03616164.chunk.js:1
    (anonymous) @ 3.03616164.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    n @ 2.0e6e1db1.chunk.js:2
    6 @ 3.03616164.chunk.js:1
    i @ (index):1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    u @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    forEach.t.<computed> @ main.8fa76fce.chunk.js:1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    s @ (index):1
    Promise.then (async)
    s @ (index):1
    n @ main.8fa76fce.chunk.js:1
    c @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    s @ (index):1
    r @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    (anonymous) @ main.8fa76fce.chunk.js:1
    i @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    i @ (index):1
    r @ (index):1
    t @ (index):1
    (anonymous) @ main.8fa76fce.chunk.js:1
    Show 88 more frames
    VM1052:1

    Uncaught SyntaxError: Identifier 'tab_url' has already been declared
    react_app.js:32742

    Download the React DevTools for a better development experience: https://fb.me/react-devtools

    >

    -----------------------------------------------------------------------------------------------------

    Best wishes,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    Are these DRS URI you are trying to access for this workflow? There is a known scaling issue on the dataguids hosting service that doesn't allow for large numbers of requests. We have been working with them to try and improve scaling, as well as make modifications on our end to work around the issue. What is the highest number you have successfully run with?

    If you are not using DRS, please let me know.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

    I just tried up to 24 samples. I can try more but eventually, we need 50k samples at once.

    I am not sure what is DRS but I did not intentionally use that at least.

    Do I need to change the workspace to solve this?

    Thank you,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    Looking through the data table I'm not seeing drs:// paths, only gs://, so there may be something else going on here. Thanks for the info about what previously worked. I'll investigate with one of our Cromwell engineers and let you know if we have any questions.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    I would appreciate it!

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    Can you add jcerrato@broadinstitute.org and rasch@broadinstitute.org to the workflow so we can examine the contents and configuration?

    https://app.terra.bio/#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    I added them. Thanks!

    Best,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

    I would appreciate it if there is any update with this!

    Thank you,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    Thank you for checking back in. The engineer investigating fell ill yesterday—I will see if they are available to continue investigating today or if another engineer will pick up the investigation.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

    Thank you. I appreciate it.

    Best regards,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    I've received an update from the engineer investigating. Here are the key points:

    • We believe that if you re-do your WDL to input a file of file names (FOFN), then manually create-then-list that FOFN as an field on your sample_set, you should be able to move forward.
    • Our engineer created 60k individual input files, put them in a FOFN, added the FOFN to a new sample_set. Everything made it to Cromwell and ran as it was supposed to.

    I'll email you an example WDL the engineer ran and found success with—you can use this as reference (for example, noting how the input is handled). Just don't use read_string() as they erroneously did in the output; this won't work.

    If you have any questions, please let us know. And please continue correspondence here rather than through email, to keep everything tidy. :)

    Kind regards,

    Jason

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Tetsushi Nakao

    Hi Jason,

    This seems to be working. I appreciate it!

    Was this a known issue?

    Best regards,

    Tet

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Tet,

    I do not believe it was—we don't often see workflows with this number of inputs.

    I'm glad to hear it's working! If there's anything else we can assist with, please let us know!

    Kind regards,

    Jason

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk