Copying notebook output to a Google bucket
Data generated by running an analysis in a Jupyter notebook is saved to the disk associated with the virtual notebook runtime. When the runtime is deleted, the data are as well. To transfer data generated within a notebook to more permanent storage, use steps 1 and 2 ("a" for Python and "b" for R) below.
Step 1 ("a" and "b") include the code to capture environment variables (in Python and R, respectively). Adding these code cells lets the notebook grab variables such as the workspace name and Google bucket directly, making for cleaner and more flexible notebooks that don't require hardcoding these variables in.
Note that because the workspace variables are part of the runtime environment (and not the virtual disk associated with the cluster), they will go away when the notebook cluster is stopped or paused. You will need to rerun the code cells that set the environment variables after pausing or stopping the notebook runtime environment.
=========================================
Contents
1. Setting environment variables
- Python kernel
- R kernel
2. Saving notebook files with bash commands
- Python kernel
- R kernel
=========================================
1. Setting environment variables
Python kernel
To set the environment variables in a Python kernel, use the following syntax:
import os
BILLING_PROJECT_ID = os.environ['WORKSPACE_NAMESPACE']
WORKSPACE = os.environ['WORKSPACE_NAME']
bucket = os.environ['WORKSPACE_BUCKET']
R kernel
To set the environment variables in an R kernel, use the following syntax:
project <- Sys.getenv('WORKSPACE_NAMESPACE')
workspace <- Sys.getenv('WORKSPACE_NAME')
bucket <- Sys.getenv('WORKSPACE_BUCKET')
2. Saving output files permanently to a bucket with bash commands
Note: the bucket is a Google bucket, so basic bash commands in the notebooks need to be preceded by "gsutil."
These commands will only work if you have run the commands above to set the environment variables. Once you execute these cells, the data files should be visible in the workspace bucket.
Python kernel
To save all generated files after the notebook runs, use the Python command below. If you want to copy individual files, you can replace `*` with the file name to copy.
# Copy all files in the notebook into the bucket
!gsutil cp ./* $bucket
# Run list command to see if file is in the bucket
!gsutil ls $bucket
R kernel
To save all generated files after the notebook runs, use the following R command. If you want to copy individual files, you can replace `*` with the file name to copy.
#Copy all files generated in the notebook into the bucket
system(paste0("gsutil cp ./* ",bucket),intern=TRUE)
#Run list command to see if file is in the bucket
system(paste0("gsutil ls ",bucket),intern=TRUE)
Comments
5 comments
Note that the above new functionality does not currently work for getting the WORKSPACE_NAME for workspaces with that have space in the name:
https://github.com/DataBiosphere/leonardo/issues/864
Also an alternative way to get the environment variables is through the Python `os` module:
This removes the need to pull out the 0th element from the SList returned from the "!command".
Issue 864 should be released this week.
I like the python approach better than shelling out `echo` commands. Here's an R version:
project <- Sys.getenv('WORKSPACE_NAMESPACE')
workspace <- Sys.getenv('WORKSPACE_NAME')
bucket <- Sys.getenv('WORKSPACE_BUCKET')
It looks like this will be made permanent when the issue is closed this week.
@Matt Bookman -- Thanks for an update to the code, I will update that in the documentation.
I recreated my cluster today and confirmed that WORKSPACE_NAME variable was set correctly for a notebook whose workspace name has a space in it.
So https://github.com/DataBiosphere/leonardo/issues/864 is fixed.
Okay, a little more information was provided in your email that I am sharing here.
If the cluster has not been "recreated" recently, meaning completely replaced, this might not work.
Network clusters can be on "pause" and "restarted" but that is not the same as "recreated".
"Recreating" a cluster will cause any newly installed programs and notebook outputs to be cleared out, so please back up your work to the workspace bucket. [I am working on a blog post on some simple commands that facilitate this.]
Please sign in to leave a comment.