Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Error editing a wdl from a featured workspace

Comments

22 comments

  • Avatar
    Jason Cerrato

    Hi Ick,

    We'll be happy to take a closer look at this. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?

    1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press enter on your keyboard
    2. Click Save

    Let us know the workspace name, as well as the relevant submission and workflow IDs. Please also add jcerrato@broadinstitute.org to the workflow if it's not currently public.

    Can you also include some details about why you are continuing on return code 1? Is it the case that this doesn't actually indicate failure?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Hi Jason -- unfortunately, I can't share the data I used under our current IRB. I'm sorry, I know this makes it difficult. I did copy the workflow to this workspace and shared access, but I can't recreate the error without the bam file. If it helps, the error we're getting is the known bug referenced in these issues:

    https://github.com/broadinstitute/gatk/issues/5783

    https://github.com/broadinstitute/gatk/issues/6289

    Although it's not ideal, we can still use the output generated up until the time of the error, and it would make downstream scripts simpler if we were able to continue. Right now I have to deal with successes, true failures, and failures due to this bug three separate ways. Thank you!

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Ick,

    We'll do what we can! Can you provide the submission and workflow IDs for the job with the issue?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    They're in a different workspace, with the data, but the submission ID is 021efe5a-8eb3-4c67-b9d5-a8ec2b6409fe and the worflow ID is 64eb99b6-1421-486b-b6aa-a19fdb0bc300. Thanks!

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Ick,

    To confirm, the standard_runtime variable worked fine before you added "continueOnReturnCode," is that correct? Do you have a submission and workflow ID for a previously successful run before this was added?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Yes, that's correct. The only thing I changed was the "continueOnReturnCode" part of the standard_runtime variable. One that worked before this change (up until the funcotate error) is workflow ID 155bcbd7-1bed-4c14-bc8a-23ff2cb8a4b1, submission ID d93c5f64-a373-439f-8dbf-b68f6a63dd1b. Workflow fa3607c9-508c-4340-bd5e-301474fb8c21 in the same submission ID is an example of a sample that runs fine to completion, without the funcotate error, if that helps at all. (May just confuse things...)

    Thank you so much for looking into this.

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Ick,

    Just wanted to update you and let you know that I discovered some strange statuses in the metadata, and one of our engineers will be taking a closer look once they are available.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Oh, that's encouraging! Thanks for the update.

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    One of our engineers was able to isolate the issue to the configuration of the standard_runtime block with the addition of the continueOnReturnCode key. The way it's currently set, the configuration is of type map. This led to issues of mixed value types (array type was added in with "continueOnReturnCode" addition). This should be resolved by changing the type to object, which should be able to handle the mixed value types. So something like the following:

    Runtime standard_runtime = object {
    gatk_docker: gatk_docker,
    gatk_override: gatk_override,
    max_retries: max_retries_or_default,
    preemptible: preemptible_or_default,
    cpu: small_task_cpu,
    machine_mem: small_task_mem * 1000,
    command_mem: small_task_mem * 1000 - 500,
    disk: small_task_disk + disk_pad,
    boot_disk_size: boot_disk_size,
    continueOnReturnCode: [0, 1]
    }

    Note where you set the Runtime standard_runtime to be equal to object. I seem to have lost access to the method you were using, but I think this is relatively straightforward to implement.

    If you have any questions about this, please let us know!

    Kind regards,

    Jason

     

     

    0
    Comment actions Permalink
  • Avatar
    lck

    Thank you, Jason! I was able to change the code as you described above, and the job no longer hangs -- but unfortunately, it still fails the way it did before I added the "continueOnReturnCode: [0, 1]" piece. I get this error: "Job Mutect2.Funcotate:NA:5 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."

    Thanks for working on this -- I'll just try to figure out a workaround until the Funcotator bug can be addressed. I appreciate it!

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    Glad to hear it's no longer hanging! Was the funcotator task being fed the standard runtime as well?

    Feel free to write to us as a new inquiry for help troubleshooting Funcotator if needed.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    As far as I can tell, the Funcotator task is being fed the standard runtime -- it appears that way from the wdl (which is the standard somatic variant calling pipeline from the GATK featured workspace).

    I'm hoping the Funcotator issues are being investigated, given the github issues I linked earlier. Not sure if there's something else I should do?

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    Would you be able to share the submission and workflow IDs from your latest run after making the change?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Sure -- the submission ID is e93c6e1a-6343-4679-b9eb-01b0343e7ff8 and the workflow ID is 002caa1c-d7e2-4ce9-ba84-e3332c565787.

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    I can see from your WDL that you're passing in the standard_runtime to the Funcotate task call,

    but in the runtime block for the task, you don't have the continueOnReturnCode attribute to use that value you're passing in.

    I think this might be why it's still failing with code 1. Can you see if adding something like "continueOnReturnCode: runtime_params.continueOnReturnCode" allows it to work?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Not sure if I'm doing something wrong, but I get an error when I try to save a snapshot with that edit:

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hmm, I'm curious to know if it's not working because there isn't a continueOnReturnCode element for the Runtime struct.

    Perhaps it doesn't know how to handle the input. Would you be able to add a line to the struct Runtime block like "Array[Int] continueOnReturnCode"? If it doesn't work, let me know and I'll follow up with one of our workflow engineers.

    0
    Comment actions Permalink
  • Avatar
    lck

    The good news is: that change allowed the wdl to load successfully and run.

    The bad news is:

    So, the same error as before. If it helps, this was submission ID 7f2ec587-c77f-4acf-9d4b-eeb1fd7f7ea9, workflow ID 9aae779a-c81e-4652-91e0-6704b5496e2a. Thank you!

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    I'm glad we've gotten farther. I'll follow up with one of the workflow engineers to see what's going on here.

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    I noticed that the WDL you used for the workflow still doesn't have a continueOnReturnCode item in the runtime block for the Funcotate task.

    Were you having any difficulty adding one? If not, can you add continueOnReturnCode: runtime_params.continueOnReturnCode to this block (starts at line 1113) to see if it resolves your issue?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    lck

    Thank you! I must have messed up while editing and uploaded the wrong one. The job completes without exiting after the error now -- thank you so much for all of your help and patience!

    Now I'll just cross my fingers that the Funcotator bug is fixed at some point, and all of this will be irrelevant. ;)

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Lydia,

    Glad to hear it completes! And yes, I'm hoping for the same that the bug will be fixed soon. :)

    If we can help with anything else, please let us know!

    Kind regards,

    Jason

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk