Occasionally, users may need to extend the default autopause time (30 minutes) to accommodate long-running jobs.
To do this without restarting the cluster (simultaneously losing outputs and any newly loaded programs) is to send a PATCH command on a currently running cluster.
This article will describe how send a PATCH command using Swagger (an API management system that uses a web browser interface to facilitate cluster creation and management).
A note of caution: Autopause protects users from additional charges when a job is done and they are not actively working in the notebook. Turning off autopause or extending the time limit may lead to additional charges, since the notebook will continue to bill until the user manually pauses or deletes the cluster.
Step 1: Find the notebook cluster name
You can do this by opening a notebook, right clicking the Jupyter logo in the top left, and selecting "Copy Link Address":
The unique cluster id (part of the link address) is created when you make your cluster. The link will have the format below. The cluster id is the part highlighted in purple:
Step 2: Open the Swagger interface & authenticate
Swagger contains a set of scripts that allow you to configure your cluster directly from the browser.
Clicking on the link above will bring you to this screen (screenshot below). Scroll down to the PATCH topic and follow the instructions below
1. Click on the open lock on the right-hand side to authenticate. It will open up the dialog box shown below.
In that box, you'll want to check all three authorization boxes and click Authorize to close it. The lock will be closed when this is complete.
2. Click Try it out
Step 3: Fill in the Parameters section
Fill in the following:
googleProjectis the Terra billing project you are using for the notebook in question.
clusterNameis the one you found in Step 1 of this article.
Request bodyspecifies what you want the cluster to do. There are a bunch of pre-filled options in the body that you can remove. Make sure your request is specific to auto-pause by only leaving the options shown below. To adjust the threshold enter a new value (in minutes).
Step 4: Click Execute
Under the Responses section, you will find out whether the request went through. After clicking execute it will be the first response listed at the top. If your response doesn't make sense, please contact the Support team for help.
Cautions about adjusting the autopause threshold:
You will want to strike a balance between an autopause value that is too short (not long enough to complete your analysis) and one that is too long (so your notebook sits there running up changes without actually doing any work).
The safest practice may be to adjust the threshold to a large number (e.g. 600 minutes) for long jobs and then adjust it back to 30 when done. That way, you won't add extra charges your notebook longer than you need to.
Your browser window must remain active in order for autopause to work correctly. Be sure to adjust the sleep functions on your monitor to not turn off during job execution. This is not required if autopause is turned off.