Problems running Clara Parabricks Workflows

I have experienced several problems running Clara Parabricvs Workflows (fw2bam and germline variant calling).

See below one of the log files. I think there are issues with docker images.

Can you help us?

Thank you

2023-05-15 13:16:15,606 INFO  - MaterializeWorkflowDescriptorActor [UUID(65bde925)]: Parsing workflow as WDL 1.0
2023-05-15 13:16:15,623 INFO  - MaterializeWorkflowDescriptorActor [UUID(65bde925)]: Call-to-Backend assignments: ClaraParabricks_Germline.deepvariant -> PAPIv2-CloudNAT, ClaraParabricks_Germline.haplotypecaller -> PAPIv2-CloudNAT
2023-05-15 13:16:15,624 WARN  - PAPIv2-CloudNAT [UUID(65bde925)]: Key/s [hpcMemory, hpcRuntimeMinutes, hpcQueue] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2023-05-15 13:16:15,624 WARN  - PAPIv2-CloudNAT [UUID(65bde925)]: Key/s [hpcMemory, hpcRuntimeMinutes, hpcQueue] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2023-05-15 13:16:19,906 INFO  - WorkflowExecutionActor-65bde925-d388-4ae9-aba0-57cef3135590 [UUID(65bde925)]: Starting ClaraParabricks_Germline.haplotypecaller
2023-05-15 13:16:37,596 WARN  - BackendPreparationActor_for_65bde925:ClaraParabricks_Germline.haplotypecaller:-1:1 [UUID(65bde925)]: Docker lookup failed
java.lang.Exception: Registry nvcr.io is not supported
    at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
    at akka.actor.FSM.processEvent(FSM.scala:707)
    at akka.actor.FSM.processEvent$(FSM.scala:704)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40)
    at akka.actor.LoggingFSM.processEvent(FSM.scala:847)
    at akka.actor.LoggingFSM.processEvent$(FSM.scala:829)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40)
    at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
    at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
    at akka.actor.Actor.aroundReceive(Actor.scala:539)
    at akka.actor.Actor.aroundReceive$(Actor.scala:537)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614)
    at akka.actor.ActorCell.invoke(ActorCell.scala:583)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
    at akka.dispatch.Mailbox.run(Mailbox.scala:229)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2023-05-15 13:16:37,708 WARN  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: Unrecognized runtime attribute keys: hpcRuntimeMinutes, hpcQueue, hpcMemory
2023-05-15 13:16:37,757 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: `mv /cromwell_root/clara-parabricks-public-workflows/References/Homo_sapiens_assembly38.fasta.tar Homo_sapiens_assembly38.fasta.tar && \
time tar xvf Homo_sapiens_assembly38.fasta.tar && \
time pbrun haplotypecaller \
--in-bam /cromwell_root/fc-46202b64-a343-4c28-b50d-49dfed22434e/submissions/a530d733-f909-41d9-af50-61f566682f7e/ClaraParabricks_fq2bam/ec154f5e-87b6-4f81-8945-c59470d6fe97/call-fq2bam/HG003.hiseqx.pcr-free.30x.R1.pb.bam \
--ref Homo_sapiens_assembly38.fasta \
--out-variants HG003.hiseqx.pcr-free.30x.R1.pb.haplotypecaller.vcf \
--in-recal-file /cromwell_root/fc-46202b64-a343-4c28-b50d-49dfed22434e/submissions/a530d733-f909-41d9-af50-61f566682f7e/ClaraParabricks_fq2bam/ec154f5e-87b6-4f81-8945-c59470d6fe97/call-fq2bam/HG003.hiseqx.pcr-free.30x.R1.pb.BQSR-REPORT.txt \
 \
--haplotypecaller-options "" \
 \
 \
 \`
2023-05-15 13:16:38,065 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: Adjusting boot disk size to 11 GB: 10 GB (runtime attributes) + 0 GB (user command image) + 1 GB (Cromwell support images)
2023-05-15 13:16:41,020 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: job id: projects/973593093490/locations/us-central1/operations/10136365186267029699
2023-05-15 13:16:41,607 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: Status change from - to Initializing
2023-05-15 13:17:30,548 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: Status change from Initializing to Running
2023-05-15 13:52:48,920 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:1]: Status change from Running to Preempted
2023-05-15 13:52:52,037 INFO  - WorkflowExecutionActor-65bde925-d388-4ae9-aba0-57cef3135590 [UUID(65bde925)]: Retrying job execution for ClaraParabricks_Germline.haplotypecaller:NA:2
2023-05-15 13:52:52,513 INFO  - WorkflowExecutionActor-65bde925-d388-4ae9-aba0-57cef3135590 [UUID(65bde925)]: Starting ClaraParabricks_Germline.haplotypecaller
2023-05-15 13:53:07,426 WARN  - BackendPreparationActor_for_65bde925:ClaraParabricks_Germline.haplotypecaller:-1:2 [UUID(65bde925)]: Docker lookup failed
java.lang.Exception: Registry nvcr.io is not supported
    at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88)
    at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
    at akka.actor.FSM.processEvent(FSM.scala:707)
    at akka.actor.FSM.processEvent$(FSM.scala:704)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40)
    at akka.actor.LoggingFSM.processEvent(FSM.scala:847)
    at akka.actor.LoggingFSM.processEvent$(FSM.scala:829)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40)
    at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701)
    at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
    at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270)
    at akka.actor.Actor.aroundReceive(Actor.scala:539)
    at akka.actor.Actor.aroundReceive$(Actor.scala:537)
    at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614)
    at akka.actor.ActorCell.invoke(ActorCell.scala:583)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
    at akka.dispatch.Mailbox.run(Mailbox.scala:229)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2023-05-15 13:53:07,506 WARN  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: Unrecognized runtime attribute keys: hpcRuntimeMinutes, hpcQueue, hpcMemory
2023-05-15 13:53:07,508 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: `mv /cromwell_root/clara-parabricks-public-workflows/References/Homo_sapiens_assembly38.fasta.tar Homo_sapiens_assembly38.fasta.tar && \
time tar xvf Homo_sapiens_assembly38.fasta.tar && \
time pbrun haplotypecaller \
--in-bam /cromwell_root/fc-46202b64-a343-4c28-b50d-49dfed22434e/submissions/a530d733-f909-41d9-af50-61f566682f7e/ClaraParabricks_fq2bam/ec154f5e-87b6-4f81-8945-c59470d6fe97/call-fq2bam/HG003.hiseqx.pcr-free.30x.R1.pb.bam \
--ref Homo_sapiens_assembly38.fasta \
--out-variants HG003.hiseqx.pcr-free.30x.R1.pb.haplotypecaller.vcf \
--in-recal-file /cromwell_root/fc-46202b64-a343-4c28-b50d-49dfed22434e/submissions/a530d733-f909-41d9-af50-61f566682f7e/ClaraParabricks_fq2bam/ec154f5e-87b6-4f81-8945-c59470d6fe97/call-fq2bam/HG003.hiseqx.pcr-free.30x.R1.pb.BQSR-REPORT.txt \
 \
--haplotypecaller-options "" \
 \
 \
 \`
2023-05-15 13:53:10,163 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: Adjusting boot disk size to 11 GB: 10 GB (runtime attributes) + 0 GB (user command image) + 1 GB (Cromwell support images)
2023-05-15 13:53:11,023 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: job id: projects/973593093490/locations/us-central1/operations/4895429569537341910
2023-05-15 13:53:11,363 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: Status change from - to Initializing
2023-05-15 13:53:47,253 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: Status change from Initializing to Running
2023-05-15 15:00:32,874 INFO  - PipelinesApiAsyncBackendJobExecutionActor [UUID(65bde925)ClaraParabricks_Germline.haplotypecaller:NA:2]: Status change from Running to Failed
2023-05-15 15:00:38,582 INFO  - $j [UUID(65bde925)]: Copying workflow logs from /cromwell-workflow-logs/workflow.65bde925-d388-4ae9-aba0-57cef3135590.log to gs://fc-a81f359c-2985-4160-8a88-1da30b8038e5/submissions/20ce9e5d-df29-41b2-b204-7e44e78a6386/workflow.logs/workflow.65bde925-d388-4ae9-aba0-57cef3135590.log