Troubleshooting: Notebook/Cloud Environment Issues

Samantha (she/her)
  • Updated

Issues with notebooks and/or cloud environments can happen for a variety of reasons. In this document, we’ll go over some common issues users experience on Terra and present some strategies to troubleshoot.

Cloud environment failed to create or is stuck loading

A common issue users run into with their Cloud Environment is when creation fails or is stuck loading when starting or pausing for long periods of time. Click below to see examples of what those errors could look like.

Example screenshots

mceclip1.png

mceclip2.png

mceclip3.png

mceclip5.png

If you run into this issue, you can try manually deleting your cloud environment by navigating to your list of cloud environments: app.terra.bio/#clusters. To delete a cloud environment, click on the trash bin icon on the right. 

mceclip0.png

In general, it's good practice to delete and recreate your cloud environment periodically since new software updates may impact compatibility with older cloud environments.

When deleting the cloud environment, you have the option to also delete the attached persistent disk. Try deleting just the cloud environment first then creating a new one. If you still have trouble creating a new cloud environment, it may mean that your persistent disk is in an error state. If this happens, you can try deleting the cloud environment and the persistent disk. 

icon-warning2.png Deleting the persistent disk will delete all data and files saved to the disk (notebooks saved to a workspace bucket will not be deleted). If you are unable to delete the persistent disk due to important data and/or files being lost, please reach out to Terra Support and we will assist you.

 

G0_read-more-icon.png

Galaxy users

  If you need to delete and create a new Galaxy app, you need to make sure that the previous app is deleted first before creating a new app in order to restore files from the same disk.

Errors installing packages

Sometimes `pip install --upgrade <pkg>` does not work successfully and people need to troubleshoot. 

Now that package installs are written to the Terra detachable persistent disk, one approach is to delete and recreate that disk to troubleshoot BUT it's easy to forget that you have some important files on that disk, then delete it during a troubleshooting session and regret the deletion.

An alternative is to troubleshoot by starting from an empty `packages/` directory. For example, open a terminal and then run the following commands to move all currently installed packages to another directory so that they are no longer visible to pip, Python, and Jupyter:

cd $HOME/notebooks

export PKG_STASH_DIR=packages-as-of-$(date +"%Y%m%d")

mkdir $PKG_STASH_DIR

# Move all the currently installed packages out of the existing
# destination directory for package installations.
mv packages/* $PKG_STASH_DIR

Now you can retry the `pip`  commands to install the packages again!

Other troubleshooting tips

  • Review error messages
  • Check package documentation for dependency information

Custom cloud environments

Terra maintains pre-configured environments that are available for you to use. If these environments don’t fit your needs, you can use a custom Docker Image or include a startup script. Or you can combine both - specify a custom Docker image and a startup script to modify your cloud environment.

mceclip1.png

Setting up a custom docker image

To learn more about developing and using custom Docker images in Terra, see these articles.

Terra currently supports the following container registries - Dockerhub, Github Container Registry, and Google Container Registry. Images hosted on Dockerhub and Github Container Registry must be public in order to use them in your cloud environment. Images hosted on Google Container Registry can be private as long as your Terra account can access them.

Using a startup script

A startup script is a bash shell script that is run once when a new cloud environment is created. It’s useful for one-time system configuration changes that must be performed as the root user, e.g.

  • Installing/updating commands/packages
  • Modifying system configuration files 
icon-warning2.png Startup scripts are a great option for users who don’t want to set up and maintain a custom Docker image. They are a good compromise of power and ease of use: it’s not difficult to make your own, and you can readily distribute the script for others who are also using the same kernel in Terra.

Startup script example

#!/usr/bin/env bash

pip install nbconvert
apt-get update
apt-get install -yq pandoc texlive-xetex
R -e "install.packages('devtools')"
pip install scanpy[louvain]==1.4.4.post1 anndata==0.6.22rc1 h5py==2.9.0
R -e "BiocManager::install('multtest')"

Steps to use startup script

  1. Create a startup script to perform the desired actions
  2. Upload/copy the startup script to a (workspace) bucket
  3. Identify the gs:// URI for the startup script
  4. Set the startup script location in the Cloud environment configuration, under “Cloud compute profile”
  5. Update cloud environment

Additional support

If you didn’t find an answer to your question above, feel free to reach out to support@terra.bio, keeping in mind our Terra Support Policy. In your email, please include a description of your issue and the following details, or as much as is applicable, so that we can help you more efficiently: 

  • Terra account email address
  • Terra billing project for notebook cloud environment
  • Cluster ID (if possible, found in terminal, looks like saturn-<long character string>)
  • Screenshot of your cloud environment configuration
  • Approximate time issue first occurred
  • Name of notebook
  • Screenshot(s) of any error(s) you are receiving 

If you are able, please also share your workspace with GROUP_FireCloud-Support@firecloud.org and let us know the workspace name in your inquiry. 

Additional (external) resources

While our goal is to help you with all your Terra platform questions, some technology or language questions may be out of scope. For help regarding bioinformatics questions, we recommend asking the expert community in forums such as Bioinformatics StackExchange. For questions specific to the functionality of RStudio, Bioconductor, or Galaxy, please go to each of the application’s support forums:

RStudio Community

Bioconductor

Galaxy Help

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.