New error (Validation errors: Extra inputs) running somatic calling pipeline: problem with Cromwell 48?
I'm running the GATK Somatic Variant calling pipeline on batches of samples, and something has broken between 1/20/2020 and today. My last batch ran fine, but today when I try to run the next batch, using the exact same script, inputs, workspace, and workflow (just a new set of samples), I get this error in red in the "Launching analysis" box:
Validation errors: Extra inputs: Mutect2.Mutect2.TumorCramToBam.mem,Mutect2.Mutect2.Funcotate.default_ram_mb,Mutect2.Mutect2.Funcotate.interval_list,Mutect2.Mutect2.Funcotate.default_disk_space_gb
All of these inputs are blank in my workflow (as they were previously). This is being launched using Cromwell 48, and I feel like maybe that's new -- was this the version in place on 1/20?
Thanks for any help.
Comments
24 comments
Hi Mark Fleharty
Earlier in this thread, Beri copied my response from another thread—it is the appropriate answer to the issue you are facing.
This is related to an older bug that was patched late January. This bug inadvertently prepended the workflow name to the input in specific cases, which resulted in extraneous inputs errors with task variables that have duplicated qualifiers. Previously, Cromwell both emitted and accepted incorrect method configs. If someone generated a method config before the fix and put it in their workspace, it would still be in the incorrect state today. However, today Cromwell neither emits nor accepts incorrect method configs, so while it may have run in the past, it will not today.
To resolve the issue, you will need to reimport the workflow to the workspace and fill out your method configuration manually—uploading the old method configuration json will put the method in an incorrect state once again. After doing this, you should be able to run the workflow.
If you have any questions, please let me know.
Kind regards,
Jason
Hi lck,
Would you be willing to share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in either the icon of your workspace in the workspace list or inside the workspace dashboard (see the icon with the three dots). Let us know the workspace name, as well as the submission and workflow ids for both the previously successful run and the recent failure. We'll be happy to take a closer look.
Many thanks,
Jason
Jason: I'm totally new to Terra. The workspace is "test". There are three submissions. All failed. But take a look at 157ddecb-0614-4458-8769-79a2a5d13290, because that one used the intervals from Picard. The submission 993f343b-4476-4fbe-aabf-f4a4bd4e1a48 was with a handmade GATK style interval list.
Thanks!
Jason Cerrato, the workspace has bam files that can't be shared -- is there some other option by any chance, or a way I can share without sharing the workspace data? Thanks.
(Also, Matthew Miller, I think you meant to leave your comment on your thread instead...)
Hi Ick,
Are the files unable to be shared due to an authorization domain? If so, even if the workspace is shared with the GROUP, only members who are part of the authorization domain will be able to access the workspace. If the workspace is under an authorization domain, please let me know which.
Kind regards,
Jason
Hi Jason Cerrato, it's a regulatory/data security issue for the project -- we're not supposed to share the files with people outside of my institution, although we have permission to upload them to Terra. If necessary I could delete the bam files and then share the workspace? I just hate to delete them then have to reupload later... Or is there any other way I could share troubleshooting information with you?
Thanks!
Hello, I'm having the same error. For a pipeline that previously worked perfectly fine. So it must be a bug in Terra. Please fix it.
Validation errors: Extra inputs: WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.CollectRawWgsMetrics.memory_multiplier,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.BaseRecalibrator.gatk_docker,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.ApplyBQSR.gatk_docker,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.AggregatedBamQC.AggregatedBamQC.CollectAggregationMetrics.collect_gc_bias_metrics,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.MarkDuplicates.read_name_regex,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.ApplyBQSR.memory_multiplier,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.GatherBqsrReports.gatk_docker,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.BamToGvcf.make_bamout,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.MarkDuplicates.memory_multiplier,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.BamToGvcf.make_gvcf,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.BamToGvcf.VariantCalling.HaplotypeCallerGATK4.gatk_docker,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.AggregatedBamQC.AggregatedBamQC.CollectReadgroupBamQualityMetrics.collect_gc_bias_metrics,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.CheckContamination.disable_sanity_check,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.BamToCram.BamToCram.ValidateCram.memory_multiplier,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.BamToGvcf.VariantCalling.ValidateVCF.gatk_docker,WholeGenomeGermlineSingleSample.WholeGenomeGermlineSingleSample.UnmappedBamToAlignedBam.UnmappedBamToAlignedBam.SplitRG.reads_per_file
GE, I ended up getting it to run by switching the version to 2.7 -- whatever new was introduced in Cromwell v48 was handled appropriately by version 2.7 of the workflow. My previous runs used version 2.6, and this was still selected by default when the Cromwell version was updated. So you might try that. (I also confirmed that my results using 2.6 and 2.7 were identical.)
Hi Ick,
Thank you for reporting that using the newer version worked. Gilad, please see if Ick's suggestion resolves your issue. If you run into any further difficulties, please let us know.
KInd regards,
Jason
Hello,
I am having the same issue with a different workflow imported from Dockstore. Each time I update that workflow and the inputs change, I see a similar error (screen shot attached). Any ideas what might be going on?
Deleting the workflow and re-importing from Dockstore fixes the issue, but naturally it would be nice not to have to do this each time the workflow is updated.
(If this belongs in a new thread, please just let me know!)
Kenneth Westerman
Jason C. made this comment to another user with a related error.
"This sounds related to an older bug that was patched late January. This bug inadvertently prepended the workflow name to the input in specific cases, which resulted in extraneous inputs errors with task variables that have duplicated qualifiers. Previously, Cromwell both emitted and accepted incorrect method configs. If someone generated a method config before the fix and put it in their workspace, it would still be in the incorrect state today. However, today Cromwell neither emits nor accepts incorrect method configs, so while it may have run in the past, it will not today. "
So after you re-imported the workflow to a workspace this should have permanently fixed the error. Do you still receive the error after making updates to the workflow?
I haven't seen the error recur yet but will continue to monitor. Thank you!
I have a workspace, that worked months ago, but I simply tried to rerun an analysis, expecting it to call-cache, but I got this error.
Jason Cerrato
I was able to circumvent the issue on my own. But it's good to know the solution as well of reimporting the workflow.
Thank you!
Mark
Just checking in here to report that this issue seems to still persist. I updated a workflow today that is linked through Dockstore (that involved changing the names of input parameters), and upon trying to kick it off, got a similar error (below). Once again, deleting and re-importing the workflow was the only way I got it to work.
Hi Kenneth,
Thanks for writing in. Am I right in understanding that you've needed to delete and re-import the workflow multiple times since the change took effect in January?
Kind regards,
Jason
Yes, correct. Seems to continue recurring each time there is a workflow input that is no longer part of the updated Dockstore workflow.
Hi Kenneth,
Thanks for clarifying. Our Cromwell team is currently investigating a similar report, so I'll tie this thread to the investigation and add these details. I'll update you as I hear more!
Kind regards,
Jason
Hi Kenneth,
The team is going to be investigating this more closely over the coming week. I'll reply here with any helpful updates.
Kind regards,
Jason
Hi Kenneth Westerman,
You can reproduce this every time you try, correct? Can you provide a details breakdown for how to reproduce the issue so that I can pass the steps along to the engineers for their investigation? If you can add anichols@broadinstitute.org and jcerrato@broadinstitute.org to the workflow (if it's not public) and provide a link to it for testing purposes, that would be very helpful.
Many thanks,
Jason
Just made a test workspace and workflow that reproduces the issue.
I have a simple test workflow that used to take two File inputs ("A" & "B") and worked fine. I then updated the workflow such that there is no input B and a new input C, and upon running it, get the same "Extra inputs" error that I have been describing.
Workspace on Terra is "kw-bdc-strides-credits/test-workspace", and I have invited those two addresses to share/compute in that workspace.
Public workflow is on Dockstore here: https://dockstore.org/workflows/github.com/kwesterman/test-workflow:master?tab=info
Let me know if you need anything else!
Hi Kenneth,
Terra developer here. Thank you for providing that information.
We have pinpointed the defect and will roll out a fix soon. I will ask Jason to update this thread when it lands.
Best,
Adam
Hello Kenneth Westerman,
The fix is released and I verified it in a workspace of my own. Please let us know if you have any further concerns.
Thank you for reporting and providing the repro case.
Best,
Adam
Please sign in to leave a comment.