Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Find out about service incidents and scheduled maintenance

  • Service Incident - April 1, 2021 plus-icon minus-icon

    Summary We've identified an issue impacting the creation of new billing projects in Terra. Timeline April 2, 2021 12:02 PM ET - Issue remediated - After investigation, our engineers have remediated the issue.  April 1, 2021 7:55 PM ET - Issue esca...
    See more

  • Service Incident - March 23, 2021 (Galaxy on AnVIL) plus-icon minus-icon

    Summary We've identified a Google issue impacting the use of Galaxy on AnVIL. Timeline  March 23, 2021 4:28 PM ET - Google outage - Our engineers identified an issue with launching Galaxy instances in AnVIL. We believe this is likely due to a curr...
    See more

  • Scheduled Maintenance - March 23, 2021 6:00PM - 11:00PM ET plus-icon minus-icon

    Summary The Terra team will be performing important migrations to backend systems. All Terra services will be unavailable during this migration. Tuesday, March 23, 2020 Start: 6:00 PM ET End: 11:00 PM ET Users Impacted All users of the Terra platf...
    See more

  • Service Incident - March 1, 2021 (AoU Notebooks Service) plus-icon minus-icon

    Summary We've identified and remediated an issue affecting All of Us users on March 1, 2021. AoU users were unable to access the notebooks service after the weekly release at 4 PM ET on March 1. See the Timeline section for the latest troubleshoot...
    See more

  • Service Incident - March 1, 2021 (PipelinesApiRequestHandler & long queue times) plus-icon minus-icon

    Summary We've identified an issue affecting some user workflows. User workflows are failing with the following error: PipelinesApiRequestHandler actor termination caught by manager Some user have also reported that their submissions are stuck in Q...
    See more

  • Service Incident - February 10, 2021 plus-icon minus-icon

    Summary We've identified an issue with our ID service that's impacting the platform. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system. ...
    See more

  • Scheduled Maintenance - Weekly Release plus-icon minus-icon

    Our engineers are constantly working to make our platform the best it can be, and we aim to release improvements to the platform on a weekly basis. While our goal is to release these updates with minimal-to-no disruption, there is still a chance t...
    See more

  • Scheduled Maintenance - November 12, 2020 12:00PM - 2:00PM ET (NCI CRDC Framework Services) plus-icon minus-icon

    Summary Terra team will be releasing external authentication enhancements which may impact workflows and data access at the time of release on Thursday, November 12, 2020. Start: 12:00 PM ET End: 2:00 PM ET Users Impacted Users who utilize NCI CRD...
    See more

  • Google Service Incident - October 27, 2020 (Google Cloud Storage) plus-icon minus-icon

    Summary Google has reported issues for its Google Cloud Storage service, which may impact functionality of the Terra platform. Refer to Google's Status Dashboard for the latest on this issue directly from Google. See this article for the latest on...
    See more

  • Service Incident - October 26, 2020 (Cromwell) plus-icon minus-icon

    Summary We've discovered issues related to the aborting, submitting, and completion of workflows in Terra.  There may be delays updating the status of aborted workflows. Newly submitted workflows may take longer than normal to launch. These submi...
    See more

  • Service Incident - October 15, 2020 plus-icon minus-icon

    Summary Engineers discovered a Bond outage Thursday (10/15) evening causing intermittent problems with external account linking. Timeline October 15, 2020 11:20 PM - Issue remediated - Issue resolved itself October 15, 2020 6:40 PM - Issue discove...
    See more

  • Service Incident - October 6, 2020 plus-icon minus-icon

    Summary Engineers have identified an outage that is causing Terra to produce "Failed to fetch" messages for link status in the Terra profile. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to...
    See more

  • Service Incident - September 24, 2020 plus-icon minus-icon

    Summary Cromwell engineers have identified an issue with one of the workflow execution runner instances which caused potential slowness for some user workflows. See the Timeline section for the latest troubleshooting and resolution updates and the...
    See more

  • Service Incident - September 23, 2020 plus-icon minus-icon

    Summary The workflow execution cluster slowed down over the course of a couple hours and eventually stopped processing workflows altogether for around half an hour Wednesday night (9/23). The cluster was restarted and most workflows have continued...
    See more

  • Service Incident - September 8, 2020 plus-icon minus-icon

    Summary We've discovered an issue where the terminal in the Terra UI is currently not loading for users.  See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact you...
    See more

  • Scheduled Maintenance - August 27, 2020 8:00PM - 9:00PM EDT plus-icon minus-icon

    Summary Terra team will be releasing security enhancements which will have some impact for workflows and Methods Repository users at the time of release on Thursday, August 27, 2020. Start: 8:00 PM EDT End: 9:00 PM EDT Users Impacted Workflows use...
    See more

  • Service Incident - August 12, 2020 plus-icon minus-icon

    Summary We've identified an issue that is causing runtimes to be stuck in Updating status when updating Dataproc runtimes. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this...
    See more

  • Scheduled Maintenance - July 27, 2020 plus-icon minus-icon

    Summary Terra team will be releasing a hotfix for a Notebooks issue which will have some impact for Notebooks users at the time of release on Monday, July 27, 2020. Start: 9:00 PM EDT End: 9:30 PM EDT Users Impacted Notebook users Expected Behavio...
    See more

  • Service Incident - July 17, 2020 (Workflows Issue) plus-icon minus-icon

    Summary We've identified an issue that is disallowing users from running new workflows on Terra. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of ...
    See more

  • Service Incident - July 7, 2020 plus-icon minus-icon

    Summary The issue was found at 12:46 PM on July 7 and impacts some users launching workflows. See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the...
    See more

  • Service Notification - July 2, 2020 (Free Credits Projects) plus-icon minus-icon

    Summary Due to activity that is against the Terra Terms of Service, we have suspended the free credits program as of July 2, 2020. The Terra team is looking into the matter, and this article will be updated with any developments. Please follow thi...
    See more

  • Service Notification - June 29, 2020 (Free Credits Projects) plus-icon minus-icon

    Summary The service to sign up for free credits is currently unavailable. Service is scheduled to be restored tomorrow, June 30. Users who try to create a new free credits project will see an error similar to the following: Trial has no available...
    See more

  • Service Incident - June 29, 2020 plus-icon minus-icon

    Summary   We are seeing issues associated with linking to Identity & External Servers across a number of Terra domains. This issue is due to certain URLs not being whitelisted on Gen3, and missing aliases on our whitelist. Known issues: Error (re...
    See more

  • Service Incident - June 16, 2020 plus-icon minus-icon

    Summary An external configuration change has caused the import of cohort data feature in the Data Explorer to fail. Trying to export a cohort will result in an Import Error. See the Timeline section for the latest troubleshooting and resolution u...
    See more

  • Service Incident - June 11, 2020 plus-icon minus-icon

    Summary Our external ID linking service is currently returning errors on request with Bearer Tokens. This is impactful for users who link to DCF and DCP Frameworks. A Terra UI bug is resulting in existing or newly linked DCP or DCF Framework links...
    See more

  • Service Incident - June 9, 2020 plus-icon minus-icon

    Summary A configuration error led to Cromwell being down for approximately 25 minutes. The issue was swiftly resolved and Cromwell is operating as it normally would.   See the Timeline section for the latest troubleshooting and resolution updates ...
    See more

  • Service Incident - June 8, 2020 plus-icon minus-icon

    Summary On the evening of June 5, a member of our notebooks team identified a bug related to dataproc multi-node spark clusters. When users who have created a multi-node spark cluster stop their runtime in the Terra UI, it's possible that only som...
    See more

  • Scheduled Maintenance - June 4, 2020 9:00-10:00 PM ET plus-icon minus-icon

    Summary A second series of important updates are scheduled for Thursday, June 4. Changes include updates to the SSL certificate and wildcard certificates. These updates will be highly impactful for users of the platform during the maintenance wind...
    See more

  • Scheduled Maintenance - May 28, 2020 9:00-10:00 PM ET plus-icon minus-icon

    Summary A series of important updates are scheduled for Thursday, May 28. Changes include updates to the load balancer, openDJ, and SSL certificate. These updates will be highly impactful for users of the platform during the maintenance window. Th...
    See more

  • Service Incident - May 6, 2020 plus-icon minus-icon

    Summary On the morning of May 6, 2020, we reported massive spikes of jobs receiving PAPI error code 2 to Google. They have concluded this to be an error on their end, and are working to put out a fix within the week. See the Timeline section for t...
    See more

Powered by Zendesk