We have identified an issue in which users are receiving the following error when running workflows even when the associated Google buckets do not have requestor pays enabled:
StorageException: Bucket is a requester pays bucket but no user project provided
Initially, this appeared as a transient error that could be resolved by re-running failed samples but has increased in frequency.
March 7, 2022 - Google fix released - Google followed through with our request. The GATK engineers are now finalizing the GATK release to fully resolve the issue.
March 1, 2022 - Request submitted to Google - The GATK engineers determined that fixing the issue for GATK workflows required a fix on the Google side.
February 24, 2022 5:40 AM ET - Issue remaining with GATK workflows - Users reported that the same error is still occurring in GATK workflows. Terra engineers are working to resolve this remaining issue.
February 23, 2022 2:04 PM ET - Fix implemented - Terra engineers implemented a fix that should resolve the issue. The same error is still occurring in GATK workflows. Engineers are working to solve the remaining error.
February 22, 2022 2:15 PM ET - Fix to be implemented - The Terra engineers have identified a fix that should resolve the issue. This fix is scheduled to go out on February 23, 2022.
February 22, 2022 9:00 AM ET - Issue Worsened - Several users began writing in about experiencing the error for all or nearly all samples when running workflows. The initial workaround is less effective. Terra engineers are working to resolve the issue.
February 18, 2022 3:00 PM ET - Cause Identified - It was determined that Google tweaked an error message causing Cromwell not to recognize buckets as requestor pays.
February 17, 2022 11:27 AM ET - Issue First Identified - A user first wrote in about experiencing the transient error and the Terra engineers began looking into it. Re-running failed samples resolved the error.
Users are receiving errors when attempting to run workflows. A possible workaround is to re-run the failed samples until they succeed.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.
I'm still getting errors. 😢
Has the fix already been implemented?
No, the fix has not been implemented yet. It is scheduled to go out today (February 23).
Hi Jeremie Kalfon,
Thanks for letting us know that you are still experiencing errors. I'm going to look further into why this is happening and update this post as soon as I can.
Now it seems that all GATK workflows using the --gcs-project-for-requester-pays options are not working... same workflow, same parameters I was using before but the command returns the typical: "message" : "Bucket is a requester pays bucket but no user project provided."
Is there any updates on this. We can chat offline also if useful.
Hi Jeremie Kalfon,
The Terra engineers are looking into why this issue is still occurring. Could you share the workspace where you're seeing this issue with Terra-Support@firecloud.org and provide a link to the workspace so that we can gather more information?
Here is the workspace: https://app.terra.bio/#workspaces/broad-firecloud-ccle/DepMap_WES_CN_hg38/job_history all last 3 jobs have the same issue. They should have access now.
Thank you 🙏
Thanks for checking in. There is expected to be a GATK release sometime this week that will resolve the issue. I am unsure yet when exactly this is happening but I will let you know when I have any updates. Feel free to reach out to Terra support through email with any questions and we can chat there!
Thank you Jeremie!
Please sign in to leave a comment.