googleCloudStorageR auth on Terra RStudio

Post author
Dan Rozelle

I'm newer to using google buckets, so apologize if this is obvious. Most examples I've found just make a system call to gsutil. This does not seem to require any authentication to access as associated workspace bucket.

system(paste0("gsutil ls ",Sys.getenv('WORKSPACE_BUCKET')), 
intern = T)
[1] "gs://fc-secure-4317b24e-3c88-4a55-bf97-XXXXXXXXXXXXX/foo.txt"

 

When I attempt to use the googleCloudStorageR package on RStudio, it seems to explicitly require authentication. Is there any way to leverage the credentials/authorization already on this workspace? or alternatively, any suggestions for R-packages that work well on Terra R-Studio?

> gcs_list_buckets(Sys.getenv('WORKSPACE_NAME'))
 2021-06-01 17:07:28 > No authorization yet in this session!
 2021-06-01 17:07:28 > No  .httr-oauth  file exists in current working directory.  Do library authentication steps to provide credentials.
Error: Invalid token
> gcs_get_bucket() 2021-06-01 17:07:39 > No authorization yet in this session! 2021-06-01 17:07:39 > No .httr-oauth file exists in current working directory. Do library authentication steps to provide credentials. Error: Invalid token
> bkt <- Sys.getenv('WORKSPACE_BUCKET') > obj <- "foo.txt" > gcs_get_object(obj, bkt, meta = TRUE) 2021-06-01 17:09:04 > No authorization yet in this session! 2021-06-01 17:09:04 > No .httr-oauth file exists in current working directory. Do library authentication steps to provide credentials. Error: Invalid token

Comments

8 comments

  • Comment author
    Jason Cerrato

    Hey Dan,

    Thanks for writing in. Am I correct in my understanding that you are spinning up an RStudio cloud environment in a workspace, and you are having trouble accessing your workspace's bucket from that cloud environment using the googleCloudStorageR package? 

    Give me a bit of time to look into this package. I'll get back to you as soon as I can.

    Kind regards,

    Jason

    0
  • Comment author
    Dan Rozelle

    That is the gist of it. Thanks for your help.

    I'm also not tied to that package in particular, just trying to understand easiest way to fetch workspace bucket data from within R

    0
  • Comment author
    Jason Cerrato

    Hey Dan Rozelle

    Can you provide us with a few more details for what you are looking to ultimately do with the workspace bucket? Are you planning to access files from the bucket, copy data to the bucket, or something else?

    Kind regards,

    Jason

    0
  • Comment author
    Dan Rozelle

    Sure. Our group is testing out Terra as a tool to organize and enable scalable bioinformatics analysis. Workspace bucket storage will contain various input data (some pipeline output, other cleaned public data) for these analyses (e.g. count matrices, VCF files). Ideally we'd open R-Studio, pull down input data from the bucket, perform analysis on that instance and put back outputs from the analysis. Most of what we do is interactive analysis scripts.

    0
  • Comment author
    Jason Cerrato

    Thanks for that context. I'll do some digging to find our best solutions for your needs. I'll get back to you as soon as I can!

    0
  • Comment author
    Jason Cerrato

    Hi Dan,

    You may be interested in using the AnVIL R package, which allows you to interact with workspace buckets in a streamlined sort of way. You can see an example of how to use AnVIL in RStudio by viewing this video here: https://youtu.be/JAcCtTkkvJw?t=125

    The avdata() and avbucket() functions of the AnVIL package may be of interest to you.

    Here is the documentation for the package.

    Let me know if this would work for your purposes, or if there's anything missing in its functionality that googleCloudStorageR would be able to provide you.

    Kind regards,

    Jason

    0
  • Comment author
    Dan Rozelle

    Thank you for the suggestions Jason, AnVIL does looks like it does provide most of the functionality we are looking for. Thanks for all your help!

    0
  • Comment author
    Jason Cerrato

    Hey Dan,

    Glad to hear! If we can help with anything else, please let us know.

    Kind regards,

    Jason

    0

Please sign in to leave a comment.