Tuning Apache Spark in Runtimes

June 02, 2020 17:43
3 comments

I am attempting to run PCA on a large dataset using Hail. I am using the Terra Hail environment. I have an "Increased Computing Power" master node, and 24-odd worker nodes with 15GB memory.

The error I am getting is:

Hail version: 0.2.39-ef87446bd1c7
Error summary: SparkException: Job aborted due to stage failure: Task 13 in stage 32.0 failed 4 times, most recent failure: Lost task 13.3 in stage 32.0 (TID 1081, saturn-62c1ecef-284b-4593-96ed-63bb90133cc2-w-4.c.ariel-research-and-development.internal, executor 70): ExecutorLostFailure (executor 70 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits.  6.0 GB of 6 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.
Driver stacktrace:

Per Hail's forums (https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285) I read the following advice:

"The YARN defaults aren’t super great for a lot of the memory-intensive linear algebra routines we use. One way to make things a bit better is to increase the number of cores per executor from 1 to 4, so that all the overhead per JVM drops by a factor of 4. You can do this by setting spark.executor.cores=4 in the spark configuration."

Is there a way to make changes to the spark configuration via the Notebook runtime setup? I am about to try upping my worker node memory to 265GB but I am not sure that will accomplish the same thing as upping the # of executor cores as per the advice above.

I also saw someone with a similar issue in the Terra forums, and it was suggested a custom container could be made in Terra. Is there any documentation that explains how to do so?

https://support.terra.bio/hc/en-us/community/posts/360055143311-Hail-notebook-error-Consider-boosting-spark-yarn-executor-memoryOverhead

Thanks for reading!

Comments

3 comments

Jason Cerrato
- June 02, 2020 20:59
Hi Dan Spagnolo,

Thank you for your inquiry. You can make changes to the spark default configuration by calling this api https://notebooks.firecloud.org/#/runtimes/createRuntime with something like:
```
 { "runtimeConfig": {
    "cloudService": "dataproc",
    "properties": {
    	"spark.executor.cores": "4"
    }
  },
"label: {
"saturnAutoCreated: true"
}
}
```
You would first need to authenticate your account, then click Try it out and add the custom properties before finally selecting Execute.

You can alternatively create a custom runtime by following the directions in this article: Docker tutorial: Custom runtime environments for Jupyter Notebooks

If you have questions about any of this, please let me know!

Kind regards,

Jason
0
Dan Spagnolo
- June 02, 2020 21:18
Hi Jason Cerrato . This is still a bit over my head. Not sure how to follow your instructions.

What do you mean by authenticate your account? I am clicking the "authorize" button but I'm not sure what to do next.

I understand what would go into the googleProject parameter but how do I figure out the runtimeName that goes into the name parameter? As far as I can see there is no name I can associate with the Notebook runtime.

Once I modify the request body and execute, will the Terra environment I am running immediatly have the changes I need in the Spark configuration or do I need to do anything else?

I was looking into creating my own Spark context and using that to initialize hail (rather than let hl.init() do it for me) but your suggestion seems more straightforward.

0
Jason Cerrato
- June 03, 2020 18:38
Hi Dan,

Happy to provide some additional details here.

The first thing you'll do is click Authorize on the page, check off the three boxes, then press the Authorize button at the bottom.

Next, you'll supply your Terra billing project name as your googleProject, and the name can be anything you want it to be so long as it meets the requirements.

After you add the runtime configuration you want, you will click Execute and you should see that you have an active runtime when you visit a workspace of the billing project you provided after approximately 2-5 minutes.

The example above is actually slightly in error, so you will want to use this instead:
```
{
  "runtimeConfig": {
    "cloudService": "dataproc",
    "properties": {
      "spark:spark.executor.cores": "4"
    }
  },
  "labels": {
    "saturnAutoCreated": "true"
  }
}
```
If you have any questions, please let me know.

Kind regards,

Jason
0

Please sign in to leave a comment.