Terra notebook cluster size
I am seeking to setup a spark cluster to run a Terra notebook on for use with HAIL. It seems that the maximum number of workers/premptible nodes that can be requested is 20. Is there a reason for this upper limit and is it documented anywhere? When analyzing large genomic data with HAIL I frequently use spark clusters with 20-40 workers and 100-200 pre-emptible nodes.
Comments
1 comment
Hey Alex,
There should not be a limit - rather the UI is a tiny bit confusing! The “workers” field includes preemptibles, so you want 20 non-preemptible workers and 100 preemptibles you can add 120 in “workers” and 100 in “preemptibles”. This cluster would have 120 workers, 100 of which are preemptible. Below is a screenshot. Please let us know if this works!
Please sign in to leave a comment.