Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

gsutil cp from FC Buckets to Google VM

Answered

Comments

8 comments

  • Avatar
    Bhanu Gandham

    Hi Tim, 

     

    Sorry about the inconvenience. Our dev team is looking into it. Would you please share with us your billing project and workspace name?

    0
    Comment actions Permalink
  • Avatar
    Tim Wood

    Billing project on Google cloud client: ld-shipp-dfci-4740a0

     

    Workspace name: shipp-dfci/Staudt_bams

     

    Now, unfortunately I'm not 100% sure about how FC interacts with gcloud, but it was my impression that the billing account listed above was also the billing account associated with the workspace. If that's not true, I can see how the permissions error would exist. But, otherwise, it seems intuitive that any VM instance created under a billing account X should have permissions to any FC workspace created under billing account X. This would inevitably have to be something on the FC backend (I think), because the VM created by Google should be just a complete fresh install of some Linux distro. I.e., there aren't any SSH keys on the system. Does that make sense? I guess stated in another way, the VM probably doesn't have any idea of the existence of the FC workspace, but the FC workspace should have an idea of the existence of such VMs.

     

    As it stands, my current solution was to manually type "gcloud auth login" and manually log in to every VM I created. So that being said, the problem is technically "solved", but it took roughly an hour to bang out.

     

    Thanks for the help!

    0
    Comment actions Permalink
  • Avatar
    Beri Shifaw

    Hi Tim,

    How do you plan on creating your 100 Google VMs? When creating your VMs you could select a specific account for the VM to use such as your Terra/Firecloud proxy group which should give you access to your workspace buckets. Setting the specific account is mentioned here.

    0
    Comment actions Permalink
  • Avatar
    Tim Wood

    Via the Google compute engine, manually, with templates.

     

    Thanks, I think that second link is exactly what I needed.

    0
    Comment actions Permalink
  • Avatar
    breardon

    Hi Terra team,

    Can you provide more details about setting your Terra PROXY group to be the service account associated with a VM? We are also observing this issue and receiving the following error, based on the second link that Tim found helpful - 

    gcloud compute instances set-service-account {VM} --service-account PROXY_{value}@firecloud.org --scopes compute-rw,storage-ro

    ERROR: (gcloud.compute.instances.set-service-account) Could not fetch resource:
    - The user does not have access to service account 'PROXY_{value}@firecloud.org'. User: 'breardon@broadinstitute.org'. Ask a project owner to grant you the iam.serviceAccountUser role on the service account

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hello Brendan, 

    It is not possible to use your Terra proxy as the service account because it is a *group* not an actual service account.

    1. Get the service account email address associated with the billing-project and save the response body as a json file with this Swagger endpoint: https://sam.dsde-prod.broadinstitute.org/#!/Google/getPetServiceAccountKey

    2. After saving the response body as a .json file, you can authenticate yourself as the PET SA:

    gcloud auth activate-service-account [PET-SA@***.iam.gserviceaccount.com] --key-file=key.json

    3. Then you can set the service account:

    gcloud compute instances set-service-account [VM_instance_name] --service-account=[PET-SA@***.iam.gserviceaccount.com]

    Can you try these steps and let us know if you are able to successfully set the SA to your VM?

    0
    Comment actions Permalink
  • Avatar
    breardon

    Hi Sushma Chaluvadi, thank you for responding. Huh, was there something that I misinterpreted from Beri's suggestion and Tim's response?

    (1) When you write "Get the service account email address associated with the billing-project", I see that there is the compute engine default for our Terra billing project and then the pet service accounts for all users. Is this what you mean? Also, when using the endpoint, should we enter the terra billing project or the billing project that the VM is under (since we can't create non-terra VMs within the terra billing project)? 

    Thank you! have a good weekend!

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hi Brendan,

    I'm not positive what process the other users on the thread took to add their Proxy group but while attempting to follow the steps I was unable to add my Proxy group - as it is a group of service accounts rather than a single service account (or "user).

    For Step #1: You will need to enter in the Terra billing project that is in the "Billing" tab of your Terra account. 

     

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk