Summary
We've identified an issue that impacts users of workflows and Cromwell service.
See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system.
Timeline
November 20, 2024 2:00 PM - Issue Remediated - Our engineers have released a fix to the issue that should increase the default disk size for workflows
November 20, 2024 10:00 AM - Issue Discovered - Our engineers had gotten user reports of workflows running out of disk space.
Impact
Affected users who are using the default boot disk for runtime workflows may see this error message:
Task call_cnvs.PreprocessIntervals:NA:1 failed. The job was stopped before the command finished. PAPI error code 9. Execution failed: generic::failed_precondition: pulling image: docker pull: running ["docker" "pull" "broadinstitute/gatk:4.6.1.0"]: exit status 1 (standard error: "failed to register layer: write /opt/miniconda/envs/gatk/lib/python3.10/site-packages/pandas/tests/window/__pycache__/test_apply.cpython-310.pyc: no space left on device\n"
While this can be resolved by increasing the bootDiskSizeGb in the WDL, we are also working on an over all solution to the problem.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.