Unable to determine reason for Mutect2 failure in Terra from log file

Post author
Elise Valkanas (she/her)

I am running Mutect2 (us.gcr.io/broad-gatk/gatk:4.1.7.0) on Terra using in PON mode for a single hg38 WGS CRAM. I have cloned the example Terra workspace and updated the reference info and all somatic workspace files to hg38, but am having a problem with Funcotator failing and not writing an error to the log file so I cannot determine why. 

The entire error log is too long to paste:

gs://fc-d34f52dc-ec7a-4221-8d6b-36eb90094803/1a8ee538-c5f2-41f6-a3ea-31acc6a99402/Mutect2/4842f753-56b2-4140-860c-f0b8b609fdd9/call-Funcotate/attempt-5/Funcotate.log

It seems like the Funcotate task estimates the disk it will need and requests a VM of that size - it's estimating 154 GB. I don't see a way to override the memory or disk for Funcotate in the inputs (and I am not sure that memory is the problem), but I would like to know what is causing this task to fail or if there is a way to get more information/error output from the Mutect2 tasks when running in Terra. I have had other attempts at this workflow similarly fail without error in the log file and run once given more memory so I am not sure how to determine what the cause of failure is for each job. 

 

Comments

13 comments

  • Comment author
    Emil Furat

    Hi Elise,

     

    Thanks for reaching out! We are going to look into this issue now and let you know once we have more details.

     

    Kind regards,

    Emil

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press enter on your keyboard.
    2. Click Save.

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

    We'll be happy to investigate further once we have access to the workspace.

    Kind regards,

    Jason

    0
  • Comment author
    Elise Valkanas (she/her)

    Hello!

    I just shared my workspace and here is the other info:

    https://app.terra.bio/#workspaces/prenatal-wgs/Somatic-SNVs-Indels-GATK4_20211019 

    Workflow ID: 4842f753-56b2-4140-860c-f0b8b609fdd9 

    Submission ID: 1a8ee538-c5f2-41f6-a3ea-31acc6a99402 

    Thanks!

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    Taking a look through the logs there are a couple of things I'm noticing. First, I'm noticing that your log has a few ERROR messages with this type of message:

    I'm not sure if this is expected for you, but I figured I'd point it out in case it's something you want to take action on.

    Secondly, I compared your log to one of our previously successful runs of Mutect2 and noticed that the end of your run looks a bit different from ours. Here is yours:

    And ours:

    When Funcotator finishes successfully, you should see a message saying

    org.broadinstitute.hellbender.tools.funcotator.Funcotator done.

    with elapsed time and total memory information. When this message doesn't appear it's more often than not because the task failed abruptly due to inadequate memory. In general, when task output stops abruptly without reaching the end it's due to inadequate memory!

    You can specify how much RAM you want to allocate for the Funcotate task using the optional input default_ram_mb in your method configuration.

    You can see in the WDL that it specifies 3000 MB as the default, but you might need more depending on the size of your inputs. I'd recommend trying with more memory and seeing if the task ends up getting farther and/or completing. You can re-run with call caching enabled to save time and money on the already successful task runs.

    If you have any questions about any of this, please let me know!

    Kind regards,

    Jason

    0
  • Comment author
    Elise Valkanas (she/her)

    Hi Jason, 

    Thanks! I tried running this job again with increasing the Funcotate disk and ram variables you pointed me to above, but my job failed and looks to have used the same disk and ram resources as the previous attempt where I used the lower default values. After looking at the WDL for this workflow, it seems to me like the defaults cannot be overwritten by those input variables, but please let me know if I am missing something. I have included a screenshot of the only location in the WDL those variables are mentioned to show why this is my line of thinking. Please let me know what you think.


    Thanks,
    Elise

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    Thanks for following up. Let me take a closer look here and get back to you!

    Kind regards,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    I agree that the WDL is written in such a way that the defaults can't be overwritten by the input variables. I'm going to try and track down whether this is a known issue with the WDL and if there are any versions where this is fixed.

    If you urgently need to make progress here I would recommend making your own copy of the WDL and modifying it so that you can specify your preferred memory and disk space. I can help with this if needed.

    Kind regards,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Elise,

    I've walked through this WDL with a colleague and found that there is a way to increase memory for the Funcotate task, in something of a roundabout fashion. The Funcotate task's memory is defined by this line: https://github.com/broadinstitute/gatk/blob/4.1.7.0/scripts/mutect2_wdl/mutect2.wdl#L1104

    This is using a dictionary called standard_runtime.

    This dictionary uses a variable called machine_mem which is calculated using the workflow's small_task_mem input, which is configurable.

     

    So to allocate more memory for the Funcotate task, you'll want to define this small_task_mem variable. For example, if you wanted to allocate 8000MB of memory, you would set set_task_mem to 8.

    I don't think this is very intuitive, and the extra optional variables I mentioned earlier don't seem to serve a purpose, so I'm going to file an Issue on the GATK Github repo to see if we can get this cleaned up in a subsequent version of the WDL.

    Kind regards,

    Jason

    0
  • Comment author
    Elise Valkanas (she/her)

    HI Jason, 

    Thanks for the advice. I re-ran this workflow twice with increased set_task_mem (20GB and 40GB) like you recommended, although this increases memory to other tasks too, not just Funcotate. It does look like the jobs went further than the one above, but they didn't complete either time and there seems to be no difference between the two runs with increased memory. It looks like Funcotate isn't properly handling the case where the variant is close to the start of the contig and is making the start of the reference window negative rather than the start position of the contig. Here is part of the error with Funcotate:

    java.lang.IllegalArgumentException: Invalid interval. Contig:chrUn_KI270468v1 start:-1 end:18 
    at org.broadinstitute.hellbender.utils.Utils.validateArg(Utils.java:733) 
    at org.broadinstitute.hellbender.utils.SimpleInterval.validatePositions(SimpleInterval.java:59) 
    at org.broadinstitute.hellbender.utils.SimpleInterval.<init>(SimpleInterval.java:35) 
    at org.broadinstitute.hellbender.tools.funcotator.FuncotatorUtils.createReferenceSnippet(FuncotatorUtils.java:1439) 
    at org.broadinstitute.hellbender.tools.funcotator.dataSources.gencode.GencodeFuncotationFactory.createIgrFuncotation(GencodeFuncotationFactory.java:2437) 
    at org.broadinstitute.hellbender.tools.funcotator.dataSources.gencode.GencodeFuncotationFactory.createIgrFuncotations(GencodeFuncotationFactory.java:2363)

    If you are curious, I ran this in the same project and workflow ID mentioned above

    20 GB submission ID: 5b3990f5-1485-4cb2-9a4b-f64898064e51

    40 GB submission ID: b9ce95c3-1169-4180-b1b5-631a9df9151a

    Please let me know what you think or if there is someone on the development team I should speak to about this instead.

    Thanks for the help!
    Elise

     

    0
  • Comment author
    Jason Cerrato

    Hi Elise,

    Thanks for following up. It looks like this question is more about the functionality of the Funcotator tool, which would be better suited for the GATK Support team. I'd highly recommend bringing this to the GATK forum if you haven't already. If you don't get a response about this in a couple of days, let me know and I'll see if I can help figure out what's going on!

    Kind regards,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    We haven't heard from you in a while so I just wanted to check in to see whether you were able to get your questions answered.

    Kind regards,

    Jason

    0
  • Comment author
    Elise Valkanas (she/her)

    Hi Jason, 

    I was able to get this issue resolved through the GATK forum. I updated the version of GATK (the one featured on the workspace had a bug in this tool) and some Funcotator files and it worked perfectly. 

    Thanks so much for all the help!

    Elise

    0
  • Comment author
    Jason Cerrato

    Hey Elise,

    Great to hear! If we can help with anything else, please let us know.

    Kind regards,

    Jason

    0

Please sign in to leave a comment.