To prevent runaway charges when no computational work is being done in a cloud environment, they will automatically pause when there is no web browser or kernel activity for 30 minutes. Note that kernel activity will only prevent autopause for up to 24 hours. This article explains how autopause on Terra works by default and links to how you can manually override the default settings.
- Autopause (default) is based on browser inactivity
1.1. What actions prevent the default autopause due to inactivity?
1.2. What happens after auto-pausing?
- Kernel activity prevents autopause for up to 24 hours
2.1. How does it work?
2.2. How do I know the kernel is running?
- How to adjust autopause manually
1. Autopause (default) is based on browser inactivity
Browser inactivity includes:
- Closing the browser tab that contains the still running Terra cloud environment
- Closing your computer, which causes your laptop to go to sleep
- If your laptop goes to sleep on its own (due to configuration settings)
Once you do any of the above and 30 minutes passes, the cloud environment will autopause.
1.1 What actions prevent the default autopause based on browser inactivity?
- If you come back before 30 minutes pass, you will prevent autopause from occurring
- Your cloud environment will never pause if you don't close your cloud environment browser tab and your computer is set to never go to sleep. Note that this can be dangerous, as it could incur large and unanticipated costs.
1.2 What happens after auto-pausing?
If you come back after 30 minutes of inactivity, you will have to un-pause your cloud environment, which can take up to 2 minutes.
When a cloud environment is autopaused, you will not lose any output that had been already explicitly written to a file, since the cloud environment (aka "boot disk") is not deleted. And of course, you will not lose any output you copied to an external Google bucket.
Note that your environment variables (such as workspace name or billing project) will be reset. In order to do queries within a notebook, for example, you will need to rerun the cells that grab the environment variables.
2. Kernel activity prevents autopause for up to 24 hours
To allow long-running jobs that generate output files to finish, kernel activity will prevent autopause, even if the browser is inactive for 30 minutes.
To protect from out-of-control costs associated with executing an operation that unintentionally keeps the kernel active, autopause will pause the cloud environment - even if the kernel is active - after 24 hours.
Warning about accessing analysis output generated when the browser is inactive
Note that in this case, because the browser is inactive, outputs written to notebook cells will not be visible in the notebook user interface. Only outputs written to files external to the notebook will be retained.
2.1. How does it work?
- Autopause will attempt to trigger every 30 minutes when no browser activity is detected
- If the kernel is still busy after 30 minutes, the notebook will continue to run. When the kernel shuts off, autopause will occur at the end of the next 30-minute interval.
- Because the notebook is not connected to the browser, any outputs written to cells will not be visible in notebook user interface. However, any output written to external files can be accessed directly.
- Kernel activity will only prevent autopause for 24 hours
2.2 How do I know the kernel is running?
When the kernel autopauses, the cloud environment will go from Running -> Stopping -> Stopped in the user interface. In the screenshot below you will see that the cloud environment is running. You can click the pause button to pause the cloud environment.
Note - The kernel indicator icon circled in orange is only filled in when a cell is running in the notebook, and empty when no cells are running (i.e. when the kernel is "idle"):
3. Adjusting autopause
In some rare cases, you may want to override the default settings for autopause (if you want a longer or shorter autopause threshold, for example). Note that because overriding autopause can leave you vulnerable to accruing considerable costs, it is important to be very careful when adjusting the default settings.
For more information on adjusting autopause using Swagger, see this article.