Preventing runaway costs with Cloud Environment autopause

Allie Hajian
  • Updated

To prevent runaway charges when no computational work is being done in a cloud environment, the VM will automatically pause when there is no web browser or kernel activity (notebooks only) for 30 minutes. Note that kernel activity will only prevent autopause for up to 24 hours and only for notebook analyses. This article explains how autopause on Terra works by default, when you might want to override it, and how to manually override the default settings.

Autopause overview

Autopause (default) is based on browser inactivity. Browser inactivity includes:

  • Closing the browser tab that contains the still running Terra cloud environment
  • Closing your computer, which causes your laptop to go to sleep
  • If your laptop goes to sleep on its own (due to configuration settings)

Once you do any of the above and 30 minutes passes, the cloud environment will autopause.

What actions prevent the default autopause based on browser inactivity?

  • If you come back (click into the browser) before 30 minutes pass, you will prevent autopause.
  • If you are running a notebook analysis and the kernel is active.
  • Your cloud environment will never pause if you don't close your cloud environment browser tab and your computer is set to never go to sleep. Note that this can be dangerous, as it could incur large and unanticipated costs. 

What happens after auto-pausing?

If you come back after 30 minutes of inactivity, you will have to un-pause your cloud environment, which can take up to 2 minutes.

You will not lose any output that had been already explicitly written to a file, since the cloud environment (aka "boot disk") is not deleted. And of course, you will not lose any output you copied to an external Google bucket. However, notebook output generated while the browser is inactive will not show up in the notebook cells. 

Environment variables (such as workspace name or billing project) will be reset. In order to do queries within a notebook, for example, you will need to rerun the cells that grab the environment variables.

How does autopause work?

  • Autopause will attempt to trigger every 30 minutes when no browser activity is detected.
  • (Notebooks only) If the kernel is still busy after 30 minutes, the notebook will continue to run.
    • When the kernel shuts off, autopause will occur at the end of the next 30-minute interval.
    • Because the notebook is not connected to the browser, any outputs written to cells will not be visible in notebook user interface. However, any output written to external files can be accessed directly.
    • Kernel activity will only prevent autopause for 24 hours

Kernel activity (notebooks only) prevents autopause for up to 24 hours

To allow long-running notebook jobs that generate output files to finish, kernel activity will prevent autopause, even if the browser is inactive for 30 minutes.

To protect from out-of-control costs associated with executing an operation that unintentionally keeps the kernel active, autopause will pause the cloud environment - even if the kernel is active - after 24 hours. 

Accessing analysis output generated when the browser is inactiveNote that in this case, because the browser is inactive, outputs written to notebook cells will not be visible in the notebook user interface. Only outputs written to files external to the notebook will be retained.

How do I know the kernel is running (notebooks only)?

When the kernel autopauses, the cloud environment will go from Running -> Stopping -> Stopped in the user interface. In the screenshot below you will see that the cloud environment is running. You can click the pause button to pause the cloud environment

Note - The kernel indicator icon circled in orange is only filled in when a cell is running in the notebook, and empty when no cells are running (i.e. when the kernel is "idle"):

oct9.png

Adjusting/overriding autopause

You may want to override the default settings for autopause.

  • If you want a longer or shorter autopause threshold.
  • If you are running a long RStudio or Galaxy analysis and will be away from your browser for more than half an hour.

You can adjust the minutes of activity that trigger autopause when you customize your Cloud Environment.

Autopause_Screen_shot.png

Overriding autopause can leave you vulnerable to accruing considerable costsIt is important to be very careful when adjusting the default settings.

For more information on adjusting autopause using Swagger, see this article.

Was this article helpful?

1 out of 1 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.