Errors starting Notebook Runtime Environments
I keep trying to launch clusters for a notebook using parameters that have worked for other people on my team. Every time after a few minutes I get a popup that says "Error Creating Cluster"
Then when i click on `See Log Info` I get a `Cluster Creation Failed` message and an `Error loading cluster details` message
When I click on “Details” in the red popup I get this popup
I’m very new to this aspect of Terra/FC and appreciate any places to start troubleshooting! I’ve already cleared my cache and that didn’t help
Comments
9 comments
Hi Nick -
When you say "parameters that others have on my team", can you please be more specific?
For example, do you use a custom script or pull a custom docker image?
Thanks for any info.
Hi Adelaide,
No custom scripts or docker images. I've tried making Spark clusters with the settings below (except with workers 20, not 2) as well as various permutations. None have worked.
Of note I've been able to successfully make clusters and run notebooks using the default "profiles" when not setting as Spark cluster (normal compute, high compute). In that case I get a "Your new runtime is ready to use" message, when I click "Apply" I also get a red "cluster error" popup but thankfully the closter/notebook actually do work.
however for my application i need the Spark clusters
Nick - Have you tried changing the Kernel to PySpark 3?
Also, when you see the red pop-up (which may be happening due to a large number of workshop users today), does the cluster finish creating?
I will have someone from the notebook engineering team take a look.
Is your notebook running the code as expected?
Thanks for the additional info.
How do I change the kernel if i cannot start the cluster?
With Spark cluster settings the cluster does not finish creating... I cannot run the notebook to test if things run as expected
Without Spark cluster settings, the cluster does finish creating (despite the red "cluster error" popup) and I can run the notebook seemingly as expected
Thank you
Nick - We were having some issues yesterday with cluster creation. I was wondering if you have tried again today?
Also, I was wondering if your permissions are set up correctly on the billing account associated with this workspace. Are you able to start a notebook without spark clusters?
Adelaide
Hi Nick -
One of our notebook engineers looked into the log files, and it appears to be a quota limitation.
You are requesting 20 worker nodes, each with 500GB of disk plus 100GB for the master node, which is a total of 10100GB.
The error during creation is `Insufficient 'DISKS_TOTAL_GB' quota. Requested 10100.0, available 667.0`
You will need to request a higher quota.
The instructions on how to do that can be found in this article link.
Tried again today requesting disk below that quota, still getting errors like in the screenshots in my first post above.
Yes I've been able to start a notebook without Spark clusters (even then I still get inaccurate error messages that as with the screenshot in my second post above). So permissions does not seem to be the issue.
Every time I click "SEE LOG INFO" in the red error popup, I get another error popup saying "Error loading cluster details" (see screenshots in first post)
Please sign in to leave a comment.