Randomly hitting permission access to files

Post author
Edyta Malolepsza

Dear Terra team,

Since yesterday most of my workflows fail with

Permission 'storage.objects.get' denied on resource (or it may not exist)

Relaunch occasionally helps, so I keep restarting the runs, but odds are against me and only from time to time workflow finishes successfully. The four workspaces I am actively using right now were created two days ago, input files were uploaded to buckets associated with these workspaces - everything was set up identically as earlier.

I am happy to share workspace with the support if useful. From internal discussion I know that others also experienced similar issues.

Thank you in advance for your help!

Edyta Malolepsza

Comments

7 comments

  • Comment author
    Jason Cerrato

    Hi Edyta,

    Thank you for reaching out about your issue. We're happy to take a closer look. 

    Can you share the workspaces where you are seeing this issue with Terra Support by clicking the Share button in your workspaces? The Share option is in the three-dots menu at the top-right.

    1. Toggle the "Share with support" button to "Yes"
    2. Click Save

    Please provide us with

    1. Links to your workspaces
    2. The relevant submission ID(s)
    3. The relevant workflow ID(s)

    We’ll be happy to take a closer look as soon as we can!

    Kind regards,

    Jason

    0
  • Comment author
    Edyta Malolepsza
    Hi Jason,
     
    Looks like I sent an answer to the noreply email, so here is copy and paste:
     
    Thank you for responding to my help request!
     
    Here is one of the workspaces I see the permission error:
     
    The whole workspace was created on purpose to run QC for a single RNA experiment on 96 well plate (6 wells with human material, 90 with mouse), so there are no other data and no older irrelevant workflows and you can examine the issues starting from the older workflow there (Feb 9, 2026, 6:56 PM). If you see workflow marked as aborted it was because I saw in job manager that the workflows failed and aborted the run.
     
    I need to do the following:
    • QC mouse wells (90 workflows)
    • gather the mouse results (1 workflow)
    • QC human wells (6 workflows)
    • gather the human results (1 workflow)
    As of 10 am I still have 31 mouse wells not processed. The expected input files are in place. 
     
    I have three other workspaces with similar issues and can share if helpful.
     
    Thank you in advance for your help!
     
    Best regards,
    Edyta

    Afternoon update: my coworker submitted the failing runs on my behalf and all run smoothly, my runs keep failing with the permission issue.
    0
  • Comment author
    Samantha (she/her)

    Hi Edyta,

    I believe the permission issues you're seeing are due to your account having reached the limit for indirect Google group membership. That would also explain why your coworker is able to run the workflows and you are not.

    Are you able to remove yourself from some groups or do you have another account you can use to run the workflows?

    Best,

    Samantha

    0
  • Comment author
    Edyta Malolepsza

    Hi Samantha,

    Thank you so much! I did not expect I would hit the group membership issue again after different way of propagating memberships was implemented.

    I will check my groups and see which I can withdraw from.

    Thank you once more!

    Best regards,

    Edyta

    0
  • Comment author
    Edyta Malolepsza

    Hi Jason,

    We are actively working on lowering my group membership count, but it is manual process requiring going through one snapshot / dataset / workspace I was added to at a time and trying to find the owner to request removal from the access. FYI, I am using https://data.terra.bio/swagger-ui.html#/snapshots/enumerateSnapshots and https://sam.dsde-prod.broadinstitute.org/#/Users/getSamUserCombinedState to find my groups.

    We went through this problem exactly year ago and at that time we got help from Christine Loreth, Douglas Voet, Adam Mullen and others who were able to run either scripts or APIs to help us programmatically. Do you know whom I need to get in touch with in case we need help? I am the first one who hit this issue this year and it is likely members of other groups will be there soon too, same as last year.

    Best regards,

    Edyta

    0
  • Comment author
    Jason Cerrato

    Hi Edyta,

    I'll follow up with you in a private ticket thread to make sure I don't share any sensitive details about your account access in further correspondence!

    Kind regards,

    Jason

    0
  • Comment author
    Edyta Malolepsza

    Hi Jason,

    Sounds great, thank you!

    Edyta

    0

Please sign in to leave a comment.