Multiple runtime environments per project

I would be great to allow multiple runtime environment to be created per billing-project - at the moment, only one is allowed. 

Comments

8 comments

  • Comment author
    Benjamin Doran

    Not sure if this is the reason Elena wants multiple run times.

    But for me a big use case, and part of our reason for moving into the cloud is that our analyses can take a long time to run. It is really common to set a script/notebook running and then in another kernel work on another aspect that needs more interaction.

    Having everything (installed packages, data, scripts) in the same environment is functional but not optimal. I often have to use larger environments than are usually necessary for the analysis because I need bandwidth to run the larger analysis, along with the smaller ones. If we had multiple environments we could use the resources necessary for that particular analysis rather than needing to over estimate. 

    These longer jobs don't really work as workflows (wdl) because for the most part you are just trying something out that you might never run again, and writing the wdl has too much boilerplate to easily test new things and drastically refactor every day. I also have tried dsub which seemed like a nice middle ground between writing a workflow and testing smaller things in jupyter notebooks, but at the time I tried it, it was not well integrated to the terra ecosystem
    1
  • Comment author
    Adrian

    Additional use cases:

    • Sometimes there are cases when 2 python packages that are incompatible because they require different versions of the same package.
    • Sometimes I have multiple docker images that I want to deploy at the same time, without having to delete the existing machine
     
    1
  • Comment author
    Sushma Chaluvadi

    Thanks Elena, will keep you posted on status of the request.

    0
  • Comment author
    Sushma Chaluvadi

    Elena,

    Would you be able to provide a bit more context on how you would like to use the ability to have multiple runtime environments per billing-project in Terra? We are trying to understand what you find is lacking in current functionality that would be resolved with this Feature to see if there are better methods or modifications to current methods.

     

    Sushma

    0
  • Comment author
    Sushma Chaluvadi

    Thanks Benjamin, I have passed your input along to the team for further investigation.

    0
  • Comment author
    Elena Torlai Triglia

    I agree with what Benjamin wrote. The case use for me as well would be to run scripts/ notebook on parallel kernels  - with a single environment, I need to estimate the 'total' resources needed rather than being able to individually allocate them.

    0
  • Comment author
    Matt Bookman

    Adrian's use case is the one that comes up the most in discussions I have had with Terra users. It would be really nice if I could create a sane environment for a workspace (typically via a docker image) and not have to worry about the environment setup conflicting with other workspaces.

    0
  • Comment author
    Samantha (she/her)

    Thanks for your input, Matt Bookman. I've shared your comment with our product team to review.

    0

Please sign in to leave a comment.