Service Incident - April 29, 2021 (Lock wait timeout exceeded)

Jason Cerrato

Summary

We are investigating an increase in the number of Lock wait timeout exceeded / MySQLTransactionRollbackException errors affecting some users of the platform. Users are running into this error when trying to make changes to their workspace dashboard or when trying to update their Data tables, particularly when those tables are large.

Users trying to update their workflow dashboard are seeing the message Error saving workspace.

image__1_.png

Users trying to update their tables are seeing the message Error uploading entities.

screen_shot_2021-04-29_at_2.11.33_pm.png

Users trying to add new or edit currently existing workspace variables are seeing the message Error saving change to workspace variables.

Screen_Shot_2021-04-30_at_1.12.32_PM.png

 

These errors are a factor of concurrent load across the system. Retrying the edit or upload typically succeeds under normal circumstances. We are investigating the matter because users are reporting consistent failures.

See the Impact section below to see how you might be affected by this issue.

Timeline

May 10, 2021 9:30 AM ET - Monitoring resolution -  Our engineers have applied a fix on Saturday, May 8 that appears to have resolved the root cause of the issue. Users should be able to update their workspace dashboards, variables, and data tables without issue at this time. If you are still experiencing "Lock wait timeout" errors, please let us know.

May 7, 2021 5:00 PM ET - Planned maintenance - Our engineers will be performing scheduled maintenance this evening from 8-10pm with the goal of reducing the frequency of these errors.

May 7, 2021 11:00 AM ET - Working on optimization - Our engineers have been working on an optimization that they hope to release on Monday.

May 3, 2021 5:00 PM ET - Improvements - We are seeing performance improvements, particularly around the updating of data tables. The errors associated with updating workspace dashboards and variables should be more intermittent, but may still happen. Our engineers are investigating current bottlenecks and areas for performance improvement. 

May 3, 2021 4:30 PM ET - Fix released - A potential fix has been released by our engineers. We are currently gathering information to discern whether users are still being impacted.

April 30, 2021 11:40 AM ET - Fix planned - A potential fix is scheduled to go out on Monday with Terra's weekly monolith release.

April 29, 2021 1:30 PM ET - A growing number of reports - A member of our Frontline Support team noticed a sizeable uptick in the number of reports for this issue. Internal members are also experiencing these errors. The appropriate engineering team has been notified and is currently investigating.

April 28, 2021 10:32 AM ET - User report - A Terra user wrote in reporting having run into an issue when trying to upload entities to their data table. 

Impact

Some users may be unable to update their workspace dashboard, update their data tables, update their workspace variables, or perform other workspace-related changes until the issue is resolved.

In particular, the updating of already large tables can potentially fail with this error. A workaround for this situation is currently available. You can either

  1. Delete the table and upload the relevant TSV again, generating it anew
  2. Upload the TSV with a different table name

For more information

Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

1 comment

  • Comment author
    Brendan Reardon

    Thank you for this detailed service incident and updates!

    1

Please sign in to leave a comment.