Summary
We have identified an issue in which users are receiving the following error when running workflows even when the associated Google buckets do not have requestor pays enabled:
StorageException: Bucket is a requester pays bucket but no user project provided
Initially, this appeared as a transient error that could be resolved by re-running failed samples but has increased in frequency.
Timeline
March 7, 2022 - Google fix released - Google followed through with our request. The GATK engineers are now finalizing the GATK release to fully resolve the issue.
March 1, 2022 - Request submitted to Google - The GATK engineers determined that fixing the issue for GATK workflows required a fix on the Google side.
February 24, 2022 5:40 AM ET - Issue remaining with GATK workflows - Users reported that the same error is still occurring in GATK workflows. Terra engineers are working to resolve this remaining issue.
February 23, 2022 2:04 PM ET - Fix implemented - Terra engineers implemented a fix that should resolve the issue. The same error is still occurring in GATK workflows. Engineers are working to solve the remaining error.
February 22, 2022 2:15 PM ET - Fix to be implemented - The Terra engineers have identified a fix that should resolve the issue. This fix is scheduled to go out on February 23, 2022.
February 22, 2022 9:00 AM ET - Issue Worsened - Several users began writing in about experiencing the error for all or nearly all samples when running workflows. The initial workaround is less effective. Terra engineers are working to resolve the issue.
February 18, 2022 3:00 PM ET - Cause Identified - It was determined that Google tweaked an error message causing Cromwell not to recognize buckets as requestor pays.
February 17, 2022 11:27 AM ET - Issue First Identified - A user first wrote in about experiencing the transient error and the Terra engineers began looking into it. Re-running failed samples resolved the error.
Impact
Users are receiving errors when attempting to run workflows. A possible workaround is to re-run the failed samples until they succeed.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.
