500 Internal Server Error
Hi,
I am experiencing an error below;
ErrorReport(rawls,http error calling uri https://cromwell1-int-lb.dsde-prod.broadinstitute.org/api/workflows/v1/batch,Some(500 Internal Server Error),List(),List(),None)
This occurs only for big sample sets.
I would appreciate your help with this.
Best,
Tet
Comments
18 comments
Hi Tet,
Thanks for writing in. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?
1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press Enter on your keyboard
2. Click Save
Let us know the workspace name, as well as the relevant submission and workflow IDs. Please also add any details about how you arrived at this error that would be helpful for our investigation. We'll be happy to take a look and get back to you as soon as we can.
Kind regards,
Jason
Hi Jason,
I appreciate your help.
workspace: natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling
Hi Tet,
Sounds good—let me know once you are able to get the workspace shared and I'll be happy to take a closer look. If the workspace is in an authorization domain, I may need to be added to that as well.
If it exists in an auth domain and I cannot be added, let me know.
Kind regards,
Jason
Hi Jason,
We added the account.
I would appreciate your help with this!
Best,
Tet
Hi Tetsushi,
Can you provide a screenshot of the error message that pops up when you try to submit, including the "Details" section if there is one (click Details to see it).
Can you also provide some screenshots of the developer tools console when you try to submit a job with 50k samples?
1. Navigate to the page where you would submit the job
2. Open the Developer Console on your browser. In Chrome, you go to View > Developer > Developer Tools
3. Click on the Console tab in the Developer Console
4. Reload the page
5. Try to kick off a submission with 50k samples
6. Take a screenshot of what you see in the Console page
Many thanks,
Jason
Hi Jason,
Thank you for your help.
Actually, nothing pops up. Just Status turns from Qued -> Failed.
I am trying to upload the screenshot but somehow the page refuses. So I am just copy-pasting that.
--------------------------------------------------------------------------------------------------------------------------
A cookie associated with a cross-site resource at http://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
/#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF:1
A cookie associated with a cross-site resource at https://google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
DevTools failed to load SourceMap: Could not load content for https://fast.appcues.com/generic/main/4.9.4/appcues.main.42ce620362c114fbcfacc93c6f2260968e34d854.js.map:
DevTools failed to load SourceMap: Could not load content for https://fast.appcues.com/generic/main/4.9.4/container.42ce620362c114fbcfacc93c6f2260968e34d854.css.map:
VM918 react_app.js:32742 Download the React DevTools for a better development experience: https://fb.me/react-devtools
/#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF:1
A cookie associated with a cross-site resource at https://accounts.google.com/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
(index):1
POST https://terra-bard-prod.appspot.com/api/syncProfile 503
t.<computed> @ (index):1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
s @ 2.0e6e1db1.chunk.js:2
Mr @ 2.0e6e1db1.chunk.js:2
g @ 2.0e6e1db1.chunk.js:2
next @ 3.03616164.chunk.js:1
s @ 3.03616164.chunk.js:1
update @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
s @ 2.0e6e1db1.chunk.js:2
Mr @ 2.0e6e1db1.chunk.js:2
g @ 2.0e6e1db1.chunk.js:2
next @ 3.03616164.chunk.js:1
s @ 3.03616164.chunk.js:1
update @ 3.03616164.chunk.js:1
t @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
n @ 2.0e6e1db1.chunk.js:2
6 @ 3.03616164.chunk.js:1
i @ (index):1
s @ (index):1
Promise.then (async)
s @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
i @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
i @ (index):1
r @ (index):1
t @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
Show 66 more frames
(index):1
GET https://api.firecloud.org/api/nih/status 404
t.<computed> @ (index):1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
Jr.getNihStatus @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
s @ 2.0e6e1db1.chunk.js:2
Mr @ 2.0e6e1db1.chunk.js:2
g @ 2.0e6e1db1.chunk.js:2
next @ 3.03616164.chunk.js:1
s @ 3.03616164.chunk.js:1
update @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
s @ 2.0e6e1db1.chunk.js:2
Mr @ 2.0e6e1db1.chunk.js:2
g @ 2.0e6e1db1.chunk.js:2
next @ 3.03616164.chunk.js:1
s @ 3.03616164.chunk.js:1
update @ 3.03616164.chunk.js:1
t @ 3.03616164.chunk.js:1
(anonymous) @ 3.03616164.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
n @ 2.0e6e1db1.chunk.js:2
6 @ 3.03616164.chunk.js:1
i @ (index):1
s @ (index):1
Promise.then (async)
s @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
u @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
forEach.t.<computed> @ main.8fa76fce.chunk.js:1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
s @ (index):1
Promise.then (async)
s @ (index):1
n @ main.8fa76fce.chunk.js:1
c @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
s @ (index):1
r @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
(anonymous) @ main.8fa76fce.chunk.js:1
i @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
i @ (index):1
r @ (index):1
t @ (index):1
(anonymous) @ main.8fa76fce.chunk.js:1
Show 88 more frames
VM1052:1
Uncaught SyntaxError: Identifier 'tab_url' has already been declared
react_app.js:32742
Download the React DevTools for a better development experience: https://fb.me/react-devtools
>
-----------------------------------------------------------------------------------------------------
Best wishes,
Tet
Hi Tet,
Are these DRS URI you are trying to access for this workflow? There is a known scaling issue on the dataguids hosting service that doesn't allow for large numbers of requests. We have been working with them to try and improve scaling, as well as make modifications on our end to work around the issue. What is the highest number you have successfully run with?
If you are not using DRS, please let me know.
Kind regards,
Jason
Hi Jason,
I just tried up to 24 samples. I can try more but eventually, we need 50k samples at once.
I am not sure what is DRS but I did not intentionally use that at least.
Do I need to change the workspace to solve this?
Thank you,
Tet
Hi Tet,
Looking through the data table I'm not seeing drs:// paths, only gs://, so there may be something else going on here. Thanks for the info about what previously worked. I'll investigate with one of our Cromwell engineers and let you know if we have any questions.
Kind regards,
Jason
I would appreciate it!
Tet
Hi Tet,
Can you add jcerrato@broadinstitute.org and rasch@broadinstitute.org to the workflow so we can examine the contents and configuration?
https://app.terra.bio/#workspaces/natarajan-aric-ukb-chip/natarajan-aric-ukb-chip-calling/workflows/tnakao/NGSPCA_WF
Many thanks,
Jason
I added them. Thanks!
Best,
Tet
Hi Jason,
I would appreciate it if there is any update with this!
Thank you,
Tet
Hi Tet,
Thank you for checking back in. The engineer investigating fell ill yesterday—I will see if they are available to continue investigating today or if another engineer will pick up the investigation.
Kind regards,
Jason
Hi Jason,
Thank you. I appreciate it.
Best regards,
Tet
Hi Tet,
I've received an update from the engineer investigating. Here are the key points:
I'll email you an example WDL the engineer ran and found success with—you can use this as reference (for example, noting how the input is handled). Just don't use read_string() as they erroneously did in the output; this won't work.
If you have any questions, please let us know. And please continue correspondence here rather than through email, to keep everything tidy. :)
Kind regards,
Jason
Hi Jason,
This seems to be working. I appreciate it!
Was this a known issue?
Best regards,
Tet
Hi Tet,
I do not believe it was—we don't often see workflows with this number of inputs.
I'm glad to hear it's working! If there's anything else we can assist with, please let us know!
Kind regards,
Jason
Please sign in to leave a comment.