I am trying to write a hail matrix to my workspace bucket and the job is not completing. I am getting this error:
Hail version: 0.2.11-daed180b84d8
Error summary: SparkException: Job aborted due to stage failure: Task 87 in stage 13.0 failed 4 times, most recent failure: Lost task 87.3 in stage 13.0 (TID 280, saturn-aac178b8-a115-47fb-9d73-96e89305ec9f-w-2.c.anvil-stage-demo.internal, executor 25): ExecutorLostFailure (executor 25 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.0 GB of 5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
I don't believe I have access to edit the container, correct? I found this thread discussing this issue, but I am unsure how to do this in Terra. https://stackoverflow.com/questions/38101857/boosting-spark-yarn-executor-memoryoverhead
Please sign in to leave a comment.