Multiple runtime environments per project

• 

  Sushma Chaluvadi

  • August 21, 2019 16:49

  Thanks Elena, will keep you posted on status of the request.

  0
• 

  Sushma Chaluvadi

  • August 28, 2019 17:38

  Elena,

  Would you be able to provide a bit more context on how you would like to use the ability to have multiple runtime environments per billing-project in Terra? We are trying to understand what you find is lacking in current functionality that would be resolved with this Feature to see if there are better methods or modifications to current methods.

   

  Sushma

  0
• 

  Benjamin Doran

  • September 04, 2019 14:21

  Not sure if this is the reason Elena wants multiple run times.

  But for me a big use case, and part of our reason for moving into the cloud is that our analyses can take a long time to run. It is really common to set a script/notebook running and then in another kernel work on another aspect that needs more interaction.

  Having everything (installed packages, data, scripts) in the same environment is functional but not optimal. I often have to use larger environments than are usually necessary for the analysis because I need bandwidth to run the larger analysis, along with the smaller ones. If we had multiple environments we could use the resources necessary for that particular analysis rather than needing to over estimate. 

  These longer jobs don't really work as workflows (wdl) because for the most part you are just trying something out that you might never run again, and writing the wdl has too much boilerplate to easily test new things and drastically refactor every day. I also have tried dsub which seemed like a nice middle ground between writing a workflow and testing smaller things in jupyter notebooks, but at the time I tried it, it was not well integrated to the terra ecosystem
  1
• 

  Sushma Chaluvadi

  • September 04, 2019 14:24

  Thanks Benjamin, I have passed your input along to the team for further investigation.

  0
• 

  Elena Torlai Triglia

  • September 04, 2019 15:51

  I agree with what Benjamin wrote. The case use for me as well would be to run scripts/ notebook on parallel kernels  - with a single environment, I need to estimate the 'total' resources needed rather than being able to individually allocate them.

  0
• 

  Adrian

  • August 03, 2020 21:41

  Additional use cases:

  • Sometimes there are cases when 2 python packages that are incompatible because they require different versions of the same package.
  • Sometimes I have multiple docker images that I want to deploy at the same time, without having to delete the existing machine

   
  1
• 

  Matt Bookman

  • November 05, 2020 01:23

  Adrian's use case is the one that comes up the most in discussions I have had with Terra users. It would be really nice if I could create a sane environment for a workspace (typically via a docker image) and not have to worry about the environment setup conflicting with other workspaces.

  0
• 

  Samantha (she/her)

  • November 05, 2020 18:37

  Thanks for your input, Matt Bookman. I've shared your comment with our product team to review.

  0

Please sign in to leave a comment.