Error "Permission 'storage.buckets.list' denied on resource" when using gsutil
Following the instructions in the support articles here and here leads to errors with `gsutil ls`:
stephanie Applications $ ./google-cloud-sdk/bin/gsutil ls
AccessDeniedException: 403 sdmorris@uw.edu does not have storage.buckets.list access to the Google Cloud project. Permission 'storage.buckets.list' denied on resource (or it may not exist).
stephanie Applications $ ./google-cloud-sdk/bin/gsutil -u terra-3b9519a5 ls
AccessDeniedException: 403 sdmorris@uw.edu does not have storage.buckets.list access to the Google Cloud project. Permission 'storage.buckets.list' denied on resource (or it may not exist).
stephanie Applications $ ./google-cloud-sdk/bin/gsutil ls -p terra-3b9519a5
AccessDeniedException: 403 sdmorris@uw.edu does not have storage.buckets.list access to the Google Cloud project. Permission 'storage.buckets.list' denied on resource (or it may not exist).
Apparently, I can only list files in a bucket, not a project:
stephanie Applications $ ./google-cloud-sdk/bin/gsutil ls gs://fc-78a7b775-81f3-4c54-a9fb-da182178a827/
gs://fc-78a7b775-81f3-4c54-a9fb-da182178a827/HapMap_array_dataset_table.tsv
gs://fc-78a7b775-81f3-4c54-a9fb-da182178a827/HapMap_array_file_table.tsv
I get a similar error while trying to copy a controlled access file to a workspace. In workspace primed-data-topmed-1/PRIMED_JHS_TOPMED_DBGAP_PHS000964_V5_P1_HMB-IRB:
rstudio@e984c28f5740:~$ gsutil cp gs://nih-nhlbi-topmed-released-phs000964-c3/phs000964.v3.pht004839.v2.p1.TOPMed_WGS_JHS_Sample.MULTI.txt.gz .
AccessDeniedException: 403 pet-107443797655395020525@terra-c9b4d853.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
rstudio@e984c28f5740:~$ gsutil -u terra-c9b4d853 cp gs://nih-nhlbi-topmed-released-phs000964-c3/phs000964.v3.pht004839.v2.p1.TOPMed_WGS_JHS_Sample.MULTI.txt.gz .
AccessDeniedException: 403 pet-107443797655395020525@terra-c9b4d853.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
I can copy the same file using terra-notebook-utils and its DRS identifier, but another error has manifested in the last few days when trying to enable requester pays (though the copy is successful). Last week:
rstudio@f29b33636061:~$ /home/rstudio/.local/bin/tnu drs copy drs://dg.4503:dg.4503/288ff0aa-a426-11ea-82d1-8bda0857af94 .
2023-01-20 11:46:40::INFO Enabling requester pays for your workspace. This will only take a few seconds...
/home/rstudio/phs000964.v3.pht004839.v2. 100% [========================================] 56.9KiB 374.9KiB/s 0.15s
Today:
rstudio@e984c28f5740:~$ /home/rstudio/.local/bin/tnu drs copy drs://dg.4503:dg.4503/288ff0aa-a426-11ea-82d1-8bda0857af94 .
2023-01-25 06:40:04::INFO Enabling requester pays for your workspace. This will only take a few seconds...
2023-01-25 06:40:05::WARNING Failed to init requester pays for workspace primed-data-topmed-1/primed-data-topmed-1/PRIMED_JHS_TOPMED_DBGAP_PHS000964_V5_P1_HMB-IRB: Expected '204', got '405' for 'https://rawls.dsde-prod.broadinstitute.org/api/workspaces/primed-data-topmed-1/primed-data-topmed-1/PRIMED_JHS_TOPMED_DBGAP_PHS000964_V5_P1_HMB-IRB/enableRequesterPaysForLinkedServiceAccounts'. You will not be able to access DRS URIs that interact with requester pays buckets.
/home/rstudio/phs000964.v3.pht004839.v2. 100% [========================================] 56.9KiB 377.4KiB/s 0.15s
Comments
9 comments
Hi Stephanie,
Thanks for writing in! Can you double-check that you are added as a user on the Google Billing project and that sdmorris@uw.edu is the same email that is added to the project? Can you also confirm that you have an active link to your NIH account in your Terra profile page?
Kind regards,
Pamela
My account is not a user on the google billing project, so that must be the problem for the first part of this question. A subset of our team manages Google billing, and we create new workspaces under the billing project using a service account. Then the workspace is shared with me (rather, a group I am part of) as a writer with compute access, so I can upload data and run workflows. We confirmed that the Google billing account owner can use `gsutil ls` to list buckets in the project.
Given this; the documentation on how to use gsutil is misleading, since it's entirely possible to list the contents of buckets and copy files from local storage to a bucket and vice versa without being a user on the billing project. The documentation should be updated to reflect this, rather than instructing all users to run `gsutil ls` to confirm that gsutil works for them.
The second part of my question still applies, as the Google billing project owner got the same error "Permission 'storage.objects.list' denied" when trying to copy a controlled access file with gsutil.
Hi Stephanie,
I agree that the documentation is misleading so I will make a note to our education team about this document. For the second issue, can you confirm that you have authenticated your NIH account in your Terra profile? Are you able to successfully navigate to the bucket where this file is located?
Kind regards,
Pamela
I have linked my NIH account, as well as my account with NHLBI BioData Catalyst Framework Services, which is what gives me access to this particular file via its DRS and terra-notebook-utils. I just cannot access it with gsutil.
When I try to navigate to the bucket containing the file in a web browser, I see the warning "Additional permissions required to list objects in this bucket. Ask a bucket owner to grant you 'storage.objects.list' permission."
Hi Stephanie,
Okay, thank you for letting me know. Have you tried to copy the file again since you first wrote in? We are hearing a lot of reports of delays in Google propagating the appropriate IAM permissions which are leading to permission errors that eventually resolve. When navigating to the bucket in the web browser, can you confirm that you are logged into the same Google account you use for Terra?
Kind regards,
Pamela
I just tried again after several days and got the same error:
I am logged in with same google account.
Hi Stephanie,
Okay, thank you for letting me know. In the workspace you're trying to copy the data from, are you able to successfully view the file with an option to download it? (example below)
Kind regards,
Pamela
Interesting, when I click on the file (which has a DRS identifier), I get a curl command to download instead of gsutil. So maybe this is a known limitation of gsutil for controlled access data; it would be helpful to mention that in the documentation.
Hi Stephanie,
Yes, you are correct. DRS URIs contain both location and authentication, so when you click on the link in Terra, we authenticate your access to the DRS object and you are able to download the file via the "Download for < $0.27" button. However, when accessing the underlying GCS bucket to download the file, you receive a permissions error because GCS is unable to authenticate your access. This behavior is expected with DRS. This is explained in our DRS documentation here: https://support.terra.bio/hc/en-us/articles/6635247495579-How-to-access-data-with-DRS-URIs but I agree that it would be helpful to include something in our gsutil documentation.
Kind regards,
Pamela
Please sign in to leave a comment.