Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, DataSTAGE. Learn more.

Copying notebook output to a Google bucket

Follow

Comments

10 comments

  • Avatar
    Matt Bookman

    Note that the above new functionality does not currently work for getting the WORKSPACE_NAME for workspaces with that have space in the name:

      https://github.com/DataBiosphere/leonardo/issues/864

    Also an alternative way to get the environment variables is through the Python `os` module:

    import os

    BILLING_PROJECT_ID = os.environ['WORKSPACE_NAMESPACE']
    WORKSPACE = os.environ['WORKSPACE_NAME']
    bucket = os.environ['WORKSPACE_BUCKET']

    This removes the need to pull out the 0th element from the SList returned from the "!command".

    0
    Comment actions Permalink
  • Avatar
    Robert Title

    Issue 864 should be released this week.

     

    I like the python approach better than shelling out `echo` commands. Here's an R version:

     

    project <- Sys.getenv('WORKSPACE_NAMESPACE')
    workspace <- Sys.getenv('WORKSPACE_NAME')
    bucket <- Sys.getenv('WORKSPACE_BUCKET')

     

    0
    Comment actions Permalink
  • Avatar
    Adelaide Rhodes

    It looks like this will be made permanent when the issue is closed this week. 

     

    @Matt Bookman --  Thanks for an update to the code, I will update that in the documentation.  

    0
    Comment actions Permalink
  • Avatar
    Matt Bookman

    I recreated my cluster today and confirmed that WORKSPACE_NAME variable was set correctly for a notebook whose workspace name has a space in it.

    So https://github.com/DataBiosphere/leonardo/issues/864 is fixed.

    0
    Comment actions Permalink
  • Avatar
    Adelaide Rhodes

    Okay, a little more information was provided in your email that I am sharing here.

    If the cluster has not been "recreated" recently, meaning completely replaced, this might not work.

    Network clusters can be on "pause" and "restarted" but that is not the same as "recreated".

    "Recreating" a cluster will cause any newly installed programs and notebook outputs to be cleared out, so please back up your work to the workspace bucket.  [I am working on a blog post on some simple commands that facilitate this.]

    0
    Comment actions Permalink
  • Avatar
    James Gatter

    bucket <- Sys.getenv('WORKSPACE_BUCKET')

    Returns an empty string for me. Hmm was the variable changed?

    Edit: print(Sys.getenv()) does not show any environmental variable that corresponds to it. 'CLUSTER_NAME' shows the ID for the notebook but not the bucket ID of the workspace.

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hi James,

    Were you perhaps working on an older Runtime Environment? Just made a fresh Runtime (with R kernel) and was able to run all the above commands successfully.

     

    I also ran print(Sys.getenv()) and while the Cluster name is listed at the top, if you scroll to the last few lines of the output form the command you can see the three variables:

    WORKSPACE_BUCKET        gs://fc-b8cc752e-77bf-481b-ba6a-315d2a0c0b78
    WORKSPACE_NAME          sushmac_sandbox
    WORKSPACE_NAMESPACE     help-gatk

    If you try again with a new runtime and the variables do not work please let us know!

    0
    Comment actions Permalink
  • Avatar
    James Gatter

    With a new runtime it still didn't work! `Sys.getenv()` prints only the text below. I think maybe I'm using one of the old custom images (us.gcr.io/broad-dsp-gcr-public/terra-jupyter-bioconductor:0.0.2). I'll see if there's anything newer and get back to you.

    _R_CHECK_COMPILATION_FLAGS_KNOWN_
                            -Wformat -Werror=format-security -Wdate-time
    CLUSTER_NAME            saturn-7ed3c13e-810b-4ac2-8233-72dc65985121
    ...
    GOOGLE_PROJECT          shalek-lab-firecloud
    ...
    USER                    jupyter-user
    WELDER_ENABLED          true
    WELDER_UID              1001
    WELDER_USER             welder-user
    WORKSPACE_NAME          dropseq_scCloud
    WORKSPACE_NAMESPACE     shalek-lab-firecloud
    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Awesome! Are you un-blocked?

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk