To prevent runaway charges when no computational work is being done in a notebook, runtime environments will automatically pause when there is no web browser or kernel activity for 30 minutes . This article explains how auto-pause on Terra works by default, and links to how you can manually override the default settings.
- Auto-pause (default) is based on browser inactivity
- What actions prevent the default auto-pause due to inactivity?
- What happens after auto-pausing?
- Kernel activity prevents auto-pause
- How does it work?
- How to know if your kernel is running
- How to adjust auto-pause manually
1. Auto-pause (default) is based on browser inactivity
Browser inactivity includes:
- Closing the browser tab that contains the still running Terra notebook
- Closing your computer, which causes your laptop to go to sleep
- If your laptop goes to sleep on its own (due to configuration settings)
Once you do any of the above and 30 minutes passes, the notebook runtime will auto-pause.
1.1 What actions prevent the default auto-pause based on browser inactivity?
- If you come back before 30 minutes passes, you will prevent auto-pause from occurring
- Your runtime will never auto-pause if you never close your notebook browser tab and your computer is set to never go to sleep. Note that this can be dangerous, as it could incur large and unanticipated costs.
1.2 What happens after auto-pausing?
If you come back after 30 minutes of inactivity, you will have to un-pause your notebook runtime, which can take up to 2 minutes.
When a notebook is auto-paused, you will not lose any output that had been already explicitly written to a file or Google bucket, since the runtime environment (aka "boot disk") is not deleted.
Note that your environment variables (such as workspace name or billing project) will be reset. In order to do queries within the notebook, for example, you will need to rerun the cells that grab the environment variables.
2. Kernel activity prevents auto-pause
To allow long running jobs that generate output files to finish, kernel activity will prevent auto-pause, even if the browser is inactive for 30 minutes.
Note that in this case, because the browser is inactive, outputs written to notebook cells will not be visible in the notebook user interface. Only outputs written to files external to the notebook will be retained.
2.1 How does it work?
- Auto-pause will attempt to trigger every 30 minutes when no browser activity is detected
- If the kernel is still busy after 30 minutes, the notebook will continue to run. When the kernel shuts off, auto-pause will occur at the end of the next 30 minute interval.
- Because the notebook is not connected to the browser, any outputs written to cells will not be visible in notebook user interface. However, any output written to external files can be accessed directly.
2.2 How to know when your kernel is running
Since a notebook disconnected from the browser does not write any cell output, how will you know whether the kernel is running if you return to the notebook?
- There is a kernel activity indicator in the top right hand corner of the notebook. If the kernel is active, the small circle is filled in solid:
- If the kernel is paused, the small circle is transparent or white:
3. Adjusting auto-pause
In some rare cases, you may want to override the default settings for auto-pause (if you want a longer or shorter auto-pause threshold, for example). Note that because overriding auto-pause can leave you vulnerable to accruing considerable costs, it is important to be very careful when adjusting the default settings.
For more information on adjusting auto-pause using Swagger, see this article.
And for more information on adjusting auto-pause using gsutil, see this article.