Possible to clone a notebook VM?
Hello!
I want to enable somebody else on my billing project to work in the exact same notebook environment I am working in.
I realize that the real goal is to have a nice docker image so that you can spin up a new notebook "cluster" using that image and you're ready to go. Same every time. Reproducible.
But in reality, it's often the case that we spend days and days getting a setup just the way we want it (above and beyond the docker image itself... things are constantly changing). And then when somebody else in the group wants to work on the same notebook VM setup, we cannot recreate the magic, and struggle for a long time.
Is it possible to just clone an existing notebook VM exactly as is? (Same billing project.) This kind of thing is possible on the Google Cloud backend, I think...? Is it possible through Terra?
Thanks,
Stephen
Comments
4 comments
Hi Stephen,
Let me look into this for you and see what I can find out. I'll get back to you as soon as I can.
Kind regards,
Jason
Hi Stephen,
Can you provide some insight for our engineers by letting us know what you mean when you say "things are constantly changing?" We want to know whether those things that are changing are things we can address and help with or if it's something outside of our control.
When you say it's possible to clone a VM on Google Cloud, are you referring to the
CREATE SIMILARbutton on the Google Cloud Console? This function chooses the same hardware and the machine image. Your work shouldn’t depend on the machine image because what defines your environment is the Docker container you use, so if your collaborators are using the same Docker image, they’re practically cloning your notebook environment.It may be the case, as you said, that the real goal is to have a nice docker image, so continually tweaking it to incorporate changes/additions made while in the notebook. But we would like to get more information just to make sure there isn't something we can't help with here.
Kind regards,
Jason
Hi Jason,
Yeah, I was a bit vague there. I was mainly referring to the fact that we have several workflows internally that all require particular versions of particular tools. As some of the tools get updated to new versions, they do not maintain backwards compatibility and they break some of our pipelines.
But I guess this can be addressed perfectly well by just making a docker image with every version of every tool specified.
And you're right as well about the `CREATE SIMILAR` button in the Google Cloud Console. It only copies the hardware and machine image, and I guess it wouldn't be cloning my R+python+conda environment anyway. I don't know if there's a way to clone everything.
I think this really is a problem I need to solve on my end with a tailored Docker image.
Thanks!
Stephen
Hi Stephen,
No problem! If there's anything you think would be helpful from our end, please let us know!
Kind regards,
Jason
Please sign in to leave a comment.