Service Incident - July 7, 2020

Tiffany Miller

Summary

The issue was found at 12:46 PM on July 7 and impacts some users launching workflows. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system. 

Timeline

July 9 - 10:57 AM ET - Issue resolution - No new instances of this error have cropped up since yesterday. This issue should now be resolved.

July 8 - 4:23 PM ET - Behavior improvements - The issue is not yet fully resolved, but we are seeing significant improvements in behavior, with fewer jobs running into the error.

July 7 - 10:10 PM ET - Further escalated issue - Raised the priority of this issue with our partners and continuing investigation. 

July 7 - 2:13 PM ET - Issue investigation - Confirmed that this issue is impacting multiple users and a banner should be created to notify users. Continuing investigation of impact.

July 7 - 12:46 PM ET - Issue discovered - Raised issue with our Google Cloud Partners for further investigation as a few users were reporting similar workflow failures on workflows that previously ran successfully.

Impact

Users running workflows may see a failure message like this: 

GcsBatchFlow.BatchFailedException: java.io.IOException: Error writing request body to server

After reviewing our logs, we believe this became an issue on July 6 19:30 EST.  We are still investigating the impact and have evidence that not all workflows are impacted. 

For more information

Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page

Was this article helpful?

4 out of 4 found this helpful

Have more questions? Submit a request

Comments

4 comments

  • Comment author
    Jason Cerrato

    Hi Samuel Terkper Ahuno,

    Happy to hear you were able to run your workflows successfully! We are seeing big improvements, with many users and internal folks no longer running into this issue. We will update the article to mark it as fully resolved after some final checks.

    Kind regards,

    Jason

    1
  • Comment author
    Samuel Terkper Ahuno

    I think it's now resolved, I run workflows paired-fastq-to-ubam, preprocessing for variant discovery and haplotype-caller on WES 216 samples yesterday evening without any errors (except user input errors from my side). can someone else check to confirm? 

     

    Best

    S

    0
  • Comment author
    Jason Cerrato

    Hi Binyamin Knisbacher,

    Thank you for letting us know. Can you write us at support@terra.bio with the submission and workflow IDs for the run that got this error? Please also share your workspace with GROUP_FireCloud-Support@firecloud.org and provide us a link. We'll be happy to investigate!

    Kind regards,

    Jason

    0
  • Comment author
    Binyamin Knisbacher

    Hi Jason, 

    I'm getting a similar error now: "GcsBatchFlow.BatchFailedException: java.io.IOException: Error writing request body to server". 

    It fails in a task upon trying to estimate size(my_input_bam) before the command section. 

    Thanks,
    Binyamin

     

    0

Please sign in to leave a comment.