Jobs claim to be running, but no activity creating a stdout, stderr, or log file
We've started some jobs, and firecloud / terra claims that the jobs have started running, but from what I can see (mostly from firecloud) the bucket has been created, but there's be no activity to move files, or start up the docker. The folder for the workflow bucket is empty, no script, stdout, stderr, or log file. It's been this way for over an hour, which is significantly longer than this usually takes.
Comments
9 comments
Hi Brian, thanks for bringing this to our attention. Can you share your workspace with GROUP_FireCloud-Support@firecloud.org so that we can take a look?
I've given that email access to broadtagteam/CNVcaller. One of the jobs has since started, but I believe the latest is still waiting to run.
Hi Brian,
You may be running into a quotas issue with not enough cpus for your amount of jobs. Are you the billing project owner for broadtagteam? If so, you can go to this link and see the current usage column to see if you are getting close to your max quotas: https://console.cloud.google.com/iam-admin/quotas?project=broadtagteam
Here are instructions on requesting more quota. Let us know if this is the issue.
If it is, we should look into providing a better way of displaying this as a warning earlier on before a workflow gets launched.
Hi Tiffany,
Thanks for looking into it. I definitely wouldn't mind an earlier warning for anything that might go wrong, so I think working on that regardless would be good.
However, I don't believe we're running into quota issues. I could be wrong because this is the first I've looked at these (seems like a great resource!), but everything looks pretty well within our limits besides Cloud Key Management Service (KMS) API. Here's a screenshot of the top "peak usage" metrics.
Hi Brian,
How does it look when you scroll down to the CPUs section? All good?
I limited it to CPU metrics and we have this now:
I've been sorting them either by 7 day peak or current usage, so the things further down the page shouldn't hit as high.
Ok, thanks. We will look further.
Hi Brian, your workflow is running and I can see the timing diagram getting updated, so that is good news. We are still investigating why there seemed to have been a pause starting the workflow. I will respond later once we know more.
Hi Brian - Did increasing the quotas resolve the issue?
Thanks for any info.
Please sign in to leave a comment.