Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, DataSTAGE. Learn more.

Errors starting Notebook Runtime Environments

Comments

9 comments

  • Avatar
    Adelaide Rhodes

    Hi Nick -

     

    When you say "parameters that others have on my team", can you please be more specific? 

    For example, do you use a custom script or pull a custom docker image?


    Thanks for any info.

     

    0
    Comment actions Permalink
  • Avatar
    Nick Moore

    Hi Adelaide,

    No custom scripts or docker images. I've tried making Spark clusters with the settings below (except with workers 20, not 2) as well as various permutations. None have worked.

    Of note I've been able to successfully make clusters and run notebooks using the default "profiles" when not setting as Spark cluster (normal compute, high compute). In that case I get a "Your new runtime is ready to use" message, when I click "Apply" I also get a red "cluster error" popup but thankfully the closter/notebook actually do work.

     

    0
    Comment actions Permalink
  • Avatar
    Nick Moore

    however for my application i need the Spark clusters

    0
    Comment actions Permalink
  • Avatar
    Adelaide Rhodes

    Nick - Have you tried changing the Kernel to PySpark 3?

    Also, when you see the red pop-up (which may be happening due to a large number of workshop users today), does the cluster finish creating?

    I will have someone from the notebook engineering team take a look.

    Is your notebook running the code as expected?

    Thanks for the additional info.

     

    0
    Comment actions Permalink
  • Avatar
    Nick Moore

    How do I change the kernel if i cannot start the cluster?

    With Spark cluster settings the cluster does not finish creating... I cannot run the notebook to test if things run as expected

    Without Spark cluster settings, the cluster does finish creating (despite the red "cluster error" popup) and I can run the notebook seemingly as expected

    Thank you

    0
    Comment actions Permalink
  • Avatar
    Adelaide Rhodes

    Nick - We were having some issues yesterday with cluster creation.  I was wondering if you have tried again today?

    Also, I was wondering if your permissions are set up correctly on the billing account associated with this workspace.  Are you able to start a notebook without spark clusters?


    Adelaide

    0
    Comment actions Permalink
  • Avatar
    Adelaide Rhodes

    Hi Nick -

    One of our notebook engineers looked into the log files, and it appears to be a quota limitation.

    You are requesting 20 worker nodes, each with 500GB of disk plus 100GB for the master node, which is a total of 10100GB.

    The error during creation is `Insufficient 'DISKS_TOTAL_GB' quota. Requested 10100.0, available 667.0`

    You will need to request a higher quota.

    The instructions on how to do that can be found in this article link.

    0
    Comment actions Permalink
  • Avatar
    Nick Moore

    Tried again today requesting disk below that quota, still getting errors like in the screenshots in my first post above.

    Yes I've been able to start a notebook without Spark clusters (even then I still get inaccurate error messages that as with the screenshot in my second post above). So permissions does not seem to be the issue.

    0
    Comment actions Permalink
  • Avatar
    Nick Moore

    Every time I click "SEE LOG INFO" in the red error popup, I get another error popup saying "Error loading cluster details" (see screenshots in first post)

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk