Jobs claim to be running, but no activity creating a stdout, stderr, or log file

Comments

9 comments

  • Tiffany Miller

    Hi Brian, thanks for bringing this to our attention. Can you share your workspace with GROUP_FireCloud-Support@firecloud.org so that we can take a look?

    0
    Comment actions Permalink
  • Brian Granger

    I've given that email access to broadtagteam/CNVcaller. One of the jobs has since started, but I believe the latest is still waiting to run. 

    0
    Comment actions Permalink
  • Tiffany Miller

    Hi Brian, 

    You may be running into a quotas issue with not enough cpus for your amount of jobs. Are you the billing project owner for broadtagteam? If so, you can go to this link and see the current usage column to see if you are getting close to your max quotas: https://console.cloud.google.com/iam-admin/quotas?project=broadtagteam

    Here are instructions on requesting more quota. Let us know if this is the issue. 

    If it is, we should look into providing a better way of displaying this as a warning earlier on before a workflow gets launched.

    0
    Comment actions Permalink
  • Brian Granger

    Hi Tiffany, 

    Thanks for looking into it. I definitely wouldn't mind an earlier warning for anything that might go wrong, so I think working on that regardless would be good. 

    However, I don't believe we're running into quota issues. I could be wrong because this is the first I've looked at these (seems like a great resource!), but everything looks pretty well within our limits besides Cloud Key Management Service (KMS) API. Here's a screenshot of the top "peak usage" metrics.

    0
    Comment actions Permalink
  • Tiffany Miller

    Hi Brian,

    How does it look when you scroll down to the CPUs section? All good?

    0
    Comment actions Permalink
  • Brian Granger

    I limited it to CPU metrics and we have this now:

     

    I've been sorting them either by 7 day peak or current usage, so the things further down the page shouldn't hit as high.

    0
    Comment actions Permalink
  • Tiffany Miller

    Ok, thanks. We will look further. 

    0
    Comment actions Permalink
  • Tiffany Miller

    Hi Brian, your workflow is running and I can see the timing diagram getting updated, so that is good news. We are still investigating why there seemed to have been a pause starting the workflow. I will respond later once we know more. 

    0
    Comment actions Permalink
  • Adelaide Rhodes

    Hi Brian - Did increasing the quotas resolve the issue?

    Thanks for any info.

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk