Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Hail notebook error: Consider boosting spark.yarn.executor.memoryOverhead

Comments

3 comments

  • Avatar
    Beri Shifaw

    Hi Elizabeth,

    What container are you using?
    You probably won't be able to edit the default docker containers but perhaps you can create a custom container based on the defaults and add the required changes. 

    1
    Comment actions Permalink
  • Avatar
    Dan Spagnolo

    Hi Beri Shifaw can you elaborate on creating a custom docker container with a custom Spark configuration? I am having the same error as Elizabeth, caused by a different function but the result is I need more Spark executor cores (default is 2 and I need to set to 4).

    To best do this, should I create my own docker container for this as you suggest, or is there a way to modify the Spark context after running hl.init()?

    0
    Comment actions Permalink
  • Avatar
    Beri Shifaw

    Hi Dan Spagnolo

    For compute resource error messages try increasing the notebook runtime. Follow along this article for details Understanding-and-Customizing-your-Notebook-Runtime.

    If you think the docker container being used doesn't have the correct version of tools that you need try creating your own. (being familiar with docker containers helps) Instructions are described here : https://github.com/databiosphere/terra-docker#terra-base-images

    Other helpful article : Docker-tutorial-Custom-runtime-environments-for-Jupyter-Notebooks

     

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk