Running concurrent notebooks from a single Workspace under separate tmux sessions
Hello,
I have two Jupyter notebooks which I've tried running simultaneously from the same Workspace environment by launching two separate tmux sessions. The notebooks are essentially identical, just use different input file lists, and are both using hail to pull GWAS vcf files from a remote repo, process them and upload into an s3 bucket.
When I launch one notebook instance from the terminal I use: jupyter nbconvert --stdout --to markdown --execute notebook_1_name.ipynb &
and it works fine. However, when I detach the current tmux session, launch a new one and launch the second notebook, both runs grind to a halt.
When I look at the hail log files I see this:
2021-06-16 15:56:00 YarnScheduler: WARN: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2021-06-16 15:56:15 YarnScheduler: WARN: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2021-06-16 15:56:15 YarnScheduler: WARN: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Any suggestions on how I can get around this issue?
Thank you very much!
Comments
12 comments
Hi Konstantin,
Thank you for writing in about this issue! It seems like the runs are failing because tmux isn't yet fully supported in Terra cloud environments. I'd be happy to file tmux support as a feature request with the appropriate engineering team and follow up with you if that support gets built.
Please let us know if there is anything else we can help you with!
Kind Regards,
Anika
Thank you very much for the quick response, Anika! Do you think that using nohup instead of tmux would work?
Hi Konstantin,
I'm not certain about the functionality of nohup in Terra, but I will look into it and let you know what I find!
Kind Regards,
Anika
Hi Konstantin,
Based on this article, https://www.maketecheasier.com/nohup-and-uses/ , it sounds like nohup is a command that is used to keep something running in the terminal even if the user logs out; the user can run process they want, but the terminal needs to be open in the browser while they’re using it.
You mentioned you are launching your notebooks from the terminal. Is there a reason you are doing this, rather than launching them from the Notebooks tab?
Kind Regards,
Anika
Hi Anika,
Thank you very much for looking into this. I'm running from a terminal because I need to analyze thousands of input files and have concurrent runs on different sets of data. In the terminal, the notebook output gets redirected into the STDOUT and does not clog up the browser screen. I'm currently using tmux for running a single notebook in the background. Being able to run in the background from tmux is not the issue, the issue arises when I launch the second notebook from another tmux session in the same terminal. That's when I get messages like:
2021-06-16 15:56:00 YarnScheduler: WARN: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2021-06-16 15:56:15 YarnScheduler: WARN: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
So, the question is, does nohup provide for the possibility to run multiple notebooks from the same terminal without clashes for available resources? Does my question make sense?
Thank you again!
Best regards,
Konstantin
Hi Anika,
Bingo!
I can confirm that I can run two notebooks concurrently via tmux as long as only one of them is using hail. Since our last exchange, we have split the analysis such that hail is only used by one of the notebooks. It would be great if one could run hail concurrently from two notebooks, I don't know if this is possible in principle.
Thank you very much again for looking into this!
Best regards,
Konstantin
Hi Konstantin,
Yay! We're so glad that worked for you! According to our engineers, running hail concurrently should be possible in principle -- the issue is that each time
hl.init()runs, it tries to allocate a SparkContext, and there aren't enough resources on the VM to run 2SparkContexts concurrently. It may be possible to manually instantiate aSparkContextwith fewer resources and dohl.init(sc=mySparkContext)This hasn't been tested though :)
Please let us know if you have further questions!
Kind Regards,
Anika
Hi Anika,
Thank you very much for the suggestion! I'll definitely look into this and see if it works.
Best regards,
Konstantin
Dear Konstantin and Anika,
With nohup or tmux am I able to run the environment and leave cells in my notebook running even if I close the actual browser window? If I wanted to run something overnight, I would like to do it through my notebook while still being able to close the lid down. I am essentially following these steps here:
https://stackoverflow.com/questions/47331050/how-to-run-jupyter-notebook-in-the-background-no-need-to-keep-one-terminal-for
Kumar
Hi Kumar Thurimella,
Unfortunately, closing the browser or your laptop will stop the VM. If you are using a notebook/RStudio/terminal, or other interactive analysis application where you create a cloud environment, you need to keep the session active (hence, interactive analysis).
It sounds like a better option for you would be to run a batch job in Cromwell within Terra. You would just need to represent the script as a WDL. If you're new to running workflows on Terra, I would suggest taking a look at our support documentation to help you get started.
Please let me know if you have any questions.
Best,
Samantha
One thing to add - if you are okay with keeping your browser/lid open, you can actually change the autopause time so the VM doesn't stop overnight. See Adjusting autopause for Cloud Environments using Swagger for more information.
Please sign in to leave a comment.