gsutil cp from FC Buckets to Google VM
Answered
Hi,
I am in the process of spinning up my own Google VMs via the compute engine, and need to localize files onto them. When I try to run "gsutil cp" command, it complains of a permissions error:
AccessDeniedException: 403 173715694639-compute@developer.gserviceaccount.com does not have storage.objects.list access to fc-secure-d3d4bb1a-7835-45ba-959b-b7bc0042bda5.
How can I add each VM to have storage.objects.list access? I am an owner of the workspace, yet when I go to the "Permissions" tab of the bucket, it says I need to be added as a "IAM Storage Admin". I am looking to construct ~100 VM instances and would like a relatively easy way to add these instances to the permissions list.
Thanks
-
Billing project on Google cloud client: ld-shipp-dfci-4740a0
Workspace name: shipp-dfci/Staudt_bams
Now, unfortunately I'm not 100% sure about how FC interacts with gcloud, but it was my impression that the billing account listed above was also the billing account associated with the workspace. If that's not true, I can see how the permissions error would exist. But, otherwise, it seems intuitive that any VM instance created under a billing account X should have permissions to any FC workspace created under billing account X. This would inevitably have to be something on the FC backend (I think), because the VM created by Google should be just a complete fresh install of some Linux distro. I.e., there aren't any SSH keys on the system. Does that make sense? I guess stated in another way, the VM probably doesn't have any idea of the existence of the FC workspace, but the FC workspace should have an idea of the existence of such VMs.
As it stands, my current solution was to manually type "gcloud auth login" and manually log in to every VM I created. So that being said, the problem is technically "solved", but it took roughly an hour to bang out.
Thanks for the help!
-
Hi Tim,
How do you plan on creating your 100 Google VMs? When creating your VMs you could select a specific account for the VM to use such as your Terra/Firecloud proxy group which should give you access to your workspace buckets. Setting the specific account is mentioned here.
-
Hi Terra team,
Can you provide more details about setting your Terra PROXY group to be the service account associated with a VM? We are also observing this issue and receiving the following error, based on the second link that Tim found helpful -
gcloud compute instances set-service-account {VM} --service-account PROXY_{value}@firecloud.org --scopes compute-rw,storage-ro
ERROR: (gcloud.compute.instances.set-service-account) Could not fetch resource:
- The user does not have access to service account 'PROXY_{value}@firecloud.org'. User: 'breardon@broadinstitute.org'. Ask a project owner to grant you the iam.serviceAccountUser role on the service account -
Hello Brendan,
It is not possible to use your Terra proxy as the service account because it is a *group* not an actual service account.
1. Get the service account email address associated with the billing-project and save the response body as a json file with this Swagger endpoint: https://sam.dsde-prod.broadinstitute.org/#!/Google/getPetServiceAccountKey
2. After saving the response body as a .json file, you can authenticate yourself as the PET SA:
gcloud auth activate-service-account [PET-SA@***.iam.gserviceaccount.com] --key-file=key.json
3. Then you can set the service account:
gcloud compute instances set-service-account [VM_instance_name] --service-account=[PET-SA@***.iam.gserviceaccount.com]
Can you try these steps and let us know if you are able to successfully set the SA to your VM?
-
Hi Sushma Chaluvadi, thank you for responding. Huh, was there something that I misinterpreted from Beri's suggestion and Tim's response?
(1) When you write "Get the service account email address associated with the billing-project", I see that there is the compute engine default for our Terra billing project and then the pet service accounts for all users. Is this what you mean? Also, when using the endpoint, should we enter the terra billing project or the billing project that the VM is under (since we can't create non-terra VMs within the terra billing project)?Thank you! have a good weekend!
-
Hi Brendan,
I'm not positive what process the other users on the thread took to add their Proxy group but while attempting to follow the steps I was unable to add my Proxy group - as it is a group of service accounts rather than a single service account (or "user).
For Step #1: You will need to enter in the Terra billing project that is in the "Billing" tab of your Terra account.
Please sign in to leave a comment.
Comments
8 comments