gsutil cp from FC Buckets to Google VM Answered

• 

  Bhanu Gandham

  • June 27, 2019 14:20

  Hi Tim, 

   

  Sorry about the inconvenience. Our dev team is looking into it. Would you please share with us your billing project and workspace name?

  0
• 

  Tim Wood

  • Edited June 27, 2019 16:12

  Billing project on Google cloud client: ld-shipp-dfci-4740a0

   

  Workspace name: shipp-dfci/Staudt_bams

   

  Now, unfortunately I'm not 100% sure about how FC interacts with gcloud, but it was my impression that the billing account listed above was also the billing account associated with the workspace. If that's not true, I can see how the permissions error would exist. But, otherwise, it seems intuitive that any VM instance created under a billing account X should have permissions to any FC workspace created under billing account X. This would inevitably have to be something on the FC backend (I think), because the VM created by Google should be just a complete fresh install of some Linux distro. I.e., there aren't any SSH keys on the system. Does that make sense? I guess stated in another way, the VM probably doesn't have any idea of the existence of the FC workspace, but the FC workspace should have an idea of the existence of such VMs.

   

  As it stands, my current solution was to manually type "gcloud auth login" and manually log in to every VM I created. So that being said, the problem is technically "solved", but it took roughly an hour to bang out.

   

  Thanks for the help!

  0
• 

  Beri

  • June 28, 2019 12:04

  Hi Tim,

  How do you plan on creating your 100 Google VMs? When creating your VMs you could select a specific account for the VM to use such as your Terra/Firecloud proxy group which should give you access to your workspace buckets. Setting the specific account is mentioned here.

  0
• 

  Tim Wood

  • June 28, 2019 15:16

  Via the Google compute engine, manually, with templates.

   

  Thanks, I think that second link is exactly what I needed.

  0
• 

  Brendan Reardon

  • January 09, 2020 22:12

  Hi Terra team,

  Can you provide more details about setting your Terra PROXY group to be the service account associated with a VM? We are also observing this issue and receiving the following error, based on the second link that Tim found helpful - 

  gcloud compute instances set-service-account {VM} --service-account PROXY_{value}@firecloud.org --scopes compute-rw,storage-ro

  ERROR: (gcloud.compute.instances.set-service-account) Could not fetch resource:
  - The user does not have access to service account 'PROXY_{value}@firecloud.org'. User: 'breardon@broadinstitute.org'. Ask a project owner to grant you the iam.serviceAccountUser role on the service account

  0
• 

  Sushma Chaluvadi

  • January 10, 2020 20:37

  Hello Brendan, 

  It is not possible to use your Terra proxy as the service account because it is a *group* not an actual service account.

  1. Get the service account email address associated with the billing-project and save the response body as a json file with this Swagger endpoint: https://sam.dsde-prod.broadinstitute.org/#!/Google/getPetServiceAccountKey

  2. After saving the response body as a .json file, you can authenticate yourself as the PET SA:

  gcloud auth activate-service-account [PET-SA@***.iam.gserviceaccount.com] --key-file=key.json

  3. Then you can set the service account:

  gcloud compute instances set-service-account [VM_instance_name] --service-account=[PET-SA@***.iam.gserviceaccount.com]

  Can you try these steps and let us know if you are able to successfully set the SA to your VM?

  0
• 

  Brendan Reardon

  • January 10, 2020 22:09

  Hi Sushma Chaluvadi, thank you for responding. Huh, was there something that I misinterpreted from Beri's suggestion and Tim's response?
  
  (1) When you write "Get the service account email address associated with the billing-project", I see that there is the compute engine default for our Terra billing project and then the pet service accounts for all users. Is this what you mean? Also, when using the endpoint, should we enter the terra billing project or the billing project that the VM is under (since we can't create non-terra VMs within the terra billing project)? 

  Thank you! have a good weekend!

  0
• 

  Sushma Chaluvadi

  • January 13, 2020 20:06

  Hi Brendan,

  I'm not positive what process the other users on the thread took to add their Proxy group but while attempting to follow the steps I was unable to add my Proxy group - as it is a group of service accounts rather than a single service account (or "user).

  For Step #1: You will need to enter in the Terra billing project that is in the "Billing" tab of your Terra account. 

   

  0

Please sign in to leave a comment.