scaling terra: threads vs CPU how to set quota
I am trying to figure out how to run big job on terra. I want to be fast, efficient, and cheap. My goal is to create an easy-to-use pipeline for my lab.
many bioinformatic tools allow you to specify the number of threads to use as a command-line argument. How is the number of threads and the workflow "runtime cpu" related?
I started going through https://support.terra.bio/hc/en-us/articles/360059028911-Scaling-your-workflow-submissions
I have a little test workflow I have testing with 10 samples. It runs however I do not think it will work well if I had to run a couple of 100 samples.
on google cloud quota page. it looks like cpu regional default is set to 24. The documentation says I have to contact terra support to change the value. This seems odd. to me as I have direct access to the google admin page. I am new to terra and google and am using Google's free $300 offer. I am afraid of misconfiguration and blowing my account
what are configuration best practices?
current I wrote my test workflows so that the runtime parameters have default inputs. In deployment, it unlikely my users will know how to set these values. I noticed that when you "export task to workspace" you are asked to 'select a method configuration' What is the configuration? is this related to runtime configuration
Kind regards
Andy
Comments
3 comments
Hi Andrew Davidson,
Thank you for your question. We'll take a look and get back to you as soon as we can!
Best,
Samantha
Hi Andrew Davidson,
Terra runs on GCP VMs so this Google doc may be useful to explain the relationship between CPUs and threads.
We maintain ownership of billing projects created on Terra, which is why you do not have access to change any quotas. If you'd like to increase a quota in your project billing project, let us know and we will submit a request to Google on your behalf. The last section of this article has some guidance on determining how much CPU quota you will need for your analysis.
Lastly, the "method configuration" you are referring to is the default values you can set for the inputs in your workflow. It can also apply to your runtime attributes if you're calling a variable for those inputs. See this featured cram-to-bam WDL in the Methods Repository and it's method configuration for an example.
Best,
Samantha
thanks
Andy
Please sign in to leave a comment.