Occasionally, users may need to extend the default autopause time (30 minutes) to accommodate long-running jobs.
To do this without restarting the cluster (simultaneously losing outputs and any newly loaded programs) is to send a PATCH command on a currently running cluster.
This article will describe how send a PATCH command using Swagger (an API management system that uses a web browser interface to facilitate cluster creation and management).
Alternatively, you can modify autopause using gsutil in a terminal.
For step-by-step instructions on that process, check out this article.
A note of caution: Autopause protects users from additional charges when a job is done and they are not actively working in the notebook. Turning off autopause or extending the time limit may lead to additional charges, since the notebook will continue to bill until the user manually pauses or deletes the cluster.
Step 1: Find the notebook cluster name
You can do this by opening a notebook, right clicking the Jupyter logo in the top left, and selecting "Copy Link Address":
The unique cluster id (part of the link address) is created when you make your cluster. The link will have the format below. The cluster id is the part highlighted in purple:
Step 2: Open the Swagger interface
Swagger contains a set of scripts that allow you to configure your cluster directly from the browser.
Clicking on the link above will bring you to this screen (screenshot below). Scroll down to the PATCH topic and follow the instructions highlighted in red.
Step 3: To turn off autopause, set to false
A note of caution when turning off autopause
Note that the notebook will continue to run (and incur charges) until you manually pause or delete the cluster.
Step 4: To adjust the threshold, enter a new value (in minutes)
Cautions about adjusting the autopause threshold:
You will want to strike a balance between an autopause value that is too short (not long enough to complete your analysis) and one that is too long (so your notebook sits there running up changes without actually doing any work).
The safest practice may be to adjust the threshold to a large number (e.g. 600 minutes) for long jobs and then adjust it back to 30 when done. That way, you won't add extra charges your notebook longer than you need to.
Your browser window must remain active in order for autopause to work correctly. Be sure to adjust the sleep functions on your monitor to not turn off during job execution. This is not required if autopause is turned off.