Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

CGA pipeline scatter getting stuck at localization step for hours

Comments

11 comments

  • Avatar
    Sushma Chaluvadi

    Hi Luda,

    We have opened a ticket with Google to investigate this as it seems to be a problem across many users. Can you please share the stdout, stderr, and **Task*.log files as well aas the Operation ID so that we can pass it along to the team looking into this issue? If you cannot post on the forum, feel free to email your information to Terra-support@broadinstitute.zendesk.com.

     

    Sushma

    0
    Comment actions Permalink
  • Avatar
    Liudmila Elagina

    Hello Sushma,

     

    I emailed log file to the email you provided, only the log file was generated. No stdout or stderr files.

     

    Thank you,

    Luda

    0
    Comment actions Permalink
  • Avatar
    Liudmila Elagina

    Hello Sushma,

     

    I shared workspace broad-firecloud-ibmwatson/Wu_Richters_IBM with GROUP_FireCloud-Support@firecloud.org . The CGA_WES_Characterization_Pipeline_v0.2_Jun2019 pipeline has been running since yesterday (~23 hours) and the current estimated cost is > $100 for 6 pairs. It is only completed 2 out of 6 pairs. 

    In the same workspace, you can find runs from the same pipeline where the cost of running it was  $1-5 per pair just a few months ago. What has changed? Why scatter tasks hang indefinitely?

    Is it possible to reimburse the billing project for this type of issue? 

     

    Thank you,

    Luda

    0
    Comment actions Permalink
  • Avatar
    breardon

    Hi Terra,

    We have also observed similar hang ups but have been unable to verify the scary cost increase that Liudmila describes. We would love to hear something about this time sensitive issue. Thank you, 

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hello Brendan,

    We are updating this thread with new information as we hear it: https://support.terra.bio/hc/en-us/community/posts/360056045911-Hanging-Localization-Step

    Sushma

    0
    Comment actions Permalink
  • Avatar
    breardon

    Thank you, Sushma! I'll keep watch of that thread. 

    0
    Comment actions Permalink
  • Avatar
    Liudmila Elagina

    Hello Sushma,

     

    I just restarted the pipeline on 6 pairs and see the same issue where scatter tasks get stuck for an hour at the localization step. You can see it in the same workspace I shared with you.

     

    Thank you,

    Luda

    0
    Comment actions Permalink
  • Avatar
    Liudmila Elagina

    I shared workspace with GROUP_FireCloud-Support@firecloud.org. It is called broad-firecloud-ibmwatson/Wu_Richters_IBM. Let me know if you have any issues accessing it.

     

    The pipeline is still running from yesterday and it is stuck on MuTect1 and MuTect2 scatter tasks. 

     

    Thank you,

    Luda

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hello Luda,

    We are attempting to re-collect new information to pass back to our Google partners to help determine what is happening. You shared your workspace but it seems that there is an Authorization Domain protecting the workspace so we are unable to access it. Would you also add us to the Authorization Domain?

     

    Thank you,

    Sushma

    0
    Comment actions Permalink
  • Avatar
    Liudmila Elagina

    Hello Sushma,

     

    I am not an owner of this authorization domain. I will request access for you. Are you able to replicate the issue in your workspace?

     

    Thank you,

    Luda

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hello Luda,

    We were hoping to look at your workspace but since we are waiting on access to the Auth Domain, we can try and replicate in our test workspace. Once you can get access, having details that are specific to your run would be great information to pass back!

     

     

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk