gsutil cp from FC Buckets to Google VM Answered

Post author
Tim Wood

Hi,

I am in the process of spinning up my own Google VMs via the compute engine, and need to localize files onto them. When I try to run "gsutil cp" command, it complains of a permissions error:

 

AccessDeniedException: 403 173715694639-compute@developer.gserviceaccount.com does not have storage.objects.list access to fc-secure-d3d4bb1a-7835-45ba-959b-b7bc0042bda5.

 

How can I add each VM to have storage.objects.list access? I am an owner of the workspace, yet when I go to the "Permissions" tab of the bucket, it says I need to be added as a "IAM Storage Admin". I am looking to construct ~100 VM instances and would like a relatively easy way to add these instances to the permissions list.

 

Thanks

Comments

8 comments

  • Comment author
    Bhanu Gandham

    Hi Tim, 

     

    Sorry about the inconvenience. Our dev team is looking into it. Would you please share with us your billing project and workspace name?

    0
  • Comment author
    Tim Wood
    • Edited

    Billing project on Google cloud client: ld-shipp-dfci-4740a0

     

    Workspace name: shipp-dfci/Staudt_bams

     

    Now, unfortunately I'm not 100% sure about how FC interacts with gcloud, but it was my impression that the billing account listed above was also the billing account associated with the workspace. If that's not true, I can see how the permissions error would exist. But, otherwise, it seems intuitive that any VM instance created under a billing account X should have permissions to any FC workspace created under billing account X. This would inevitably have to be something on the FC backend (I think), because the VM created by Google should be just a complete fresh install of some Linux distro. I.e., there aren't any SSH keys on the system. Does that make sense? I guess stated in another way, the VM probably doesn't have any idea of the existence of the FC workspace, but the FC workspace should have an idea of the existence of such VMs.

     

    As it stands, my current solution was to manually type "gcloud auth login" and manually log in to every VM I created. So that being said, the problem is technically "solved", but it took roughly an hour to bang out.

     

    Thanks for the help!

    0
  • Comment author
    Beri

    Hi Tim,

    How do you plan on creating your 100 Google VMs? When creating your VMs you could select a specific account for the VM to use such as your Terra/Firecloud proxy group which should give you access to your workspace buckets. Setting the specific account is mentioned here.

    0
  • Comment author
    Tim Wood

    Via the Google compute engine, manually, with templates.

     

    Thanks, I think that second link is exactly what I needed.

    0
  • Comment author
    Brendan Reardon

    Hi Terra team,

    Can you provide more details about setting your Terra PROXY group to be the service account associated with a VM? We are also observing this issue and receiving the following error, based on the second link that Tim found helpful - 

    gcloud compute instances set-service-account {VM} --service-account PROXY_{value}@firecloud.org --scopes compute-rw,storage-ro

    ERROR: (gcloud.compute.instances.set-service-account) Could not fetch resource:
    - The user does not have access to service account 'PROXY_{value}@firecloud.org'. User: 'breardon@broadinstitute.org'. Ask a project owner to grant you the iam.serviceAccountUser role on the service account

    0
  • Comment author
    Sushma Chaluvadi

    Hello Brendan, 

    It is not possible to use your Terra proxy as the service account because it is a *group* not an actual service account.

    1. Get the service account email address associated with the billing-project and save the response body as a json file with this Swagger endpoint: https://sam.dsde-prod.broadinstitute.org/#!/Google/getPetServiceAccountKey

    2. After saving the response body as a .json file, you can authenticate yourself as the PET SA:

    gcloud auth activate-service-account [PET-SA@***.iam.gserviceaccount.com] --key-file=key.json

    3. Then you can set the service account:

    gcloud compute instances set-service-account [VM_instance_name] --service-account=[PET-SA@***.iam.gserviceaccount.com]

    Can you try these steps and let us know if you are able to successfully set the SA to your VM?

    0
  • Comment author
    Brendan Reardon

    Hi Sushma Chaluvadi, thank you for responding. Huh, was there something that I misinterpreted from Beri's suggestion and Tim's response?

    (1) When you write "Get the service account email address associated with the billing-project", I see that there is the compute engine default for our Terra billing project and then the pet service accounts for all users. Is this what you mean? Also, when using the endpoint, should we enter the terra billing project or the billing project that the VM is under (since we can't create non-terra VMs within the terra billing project)? 

    Thank you! have a good weekend!

    0
  • Comment author
    Sushma Chaluvadi

    Hi Brendan,

    I'm not positive what process the other users on the thread took to add their Proxy group but while attempting to follow the steps I was unable to add my Proxy group - as it is a group of service accounts rather than a single service account (or "user).

    For Step #1: You will need to enter in the Terra billing project that is in the "Billing" tab of your Terra account. 

     

    0

Please sign in to leave a comment.