Service Incident - July 7, 2020
Summary
The issue was found at 12:46 PM on July 7 and impacts some users launching workflows. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system.
Timeline
July 9 - 10:57 AM ET - Issue resolution - No new instances of this error have cropped up since yesterday. This issue should now be resolved.
July 8 - 4:23 PM ET - Behavior improvements - The issue is not yet fully resolved, but we are seeing significant improvements in behavior, with fewer jobs running into the error.
July 7 - 10:10 PM ET - Further escalated issue - Raised the priority of this issue with our partners and continuing investigation.
July 7 - 2:13 PM ET - Issue investigation - Confirmed that this issue is impacting multiple users and a banner should be created to notify users. Continuing investigation of impact.
July 7 - 12:46 PM ET - Issue discovered - Raised issue with our Google Cloud Partners for further investigation as a few users were reporting similar workflow failures on workflows that previously ran successfully.
Impact
Users running workflows may see a failure message like this:
GcsBatchFlow.BatchFailedException: java.io.IOException: Error writing request body to server
After reviewing our logs, we believe this became an issue on July 6 19:30 EST. We are still investigating the impact and have evidence that not all workflows are impacted.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.
Comments
4 comments
I think it's now resolved, I run workflows paired-fastq-to-ubam, preprocessing for variant discovery and haplotype-caller on WES 216 samples yesterday evening without any errors (except user input errors from my side). can someone else check to confirm?
Best
S
Hi Samuel Terkper Ahuno,
Happy to hear you were able to run your workflows successfully! We are seeing big improvements, with many users and internal folks no longer running into this issue. We will update the article to mark it as fully resolved after some final checks.
Kind regards,
Jason
Hi Jason,
I'm getting a similar error now: "GcsBatchFlow.BatchFailedException: java.io.IOException: Error writing request body to server".
It fails in a task upon trying to estimate size(my_input_bam) before the command section.
Thanks,
Binyamin
Hi Binyamin Knisbacher,
Thank you for letting us know. Can you write us at support@terra.bio with the submission and workflow IDs for the run that got this error? Please also share your workspace with GROUP_FireCloud-Support@firecloud.org and provide us a link. We'll be happy to investigate!
Kind regards,
Jason
Please sign in to leave a comment.