Allow Owners on a Billing Project to set BigQuery quotas
Google BigQuery comes with a set of very useful cost control mechanisms:
https://cloud.google.com/bigquery/docs/custom-quotas
These have been made available to users primarily to avoid accidental over-spending. The most useful are typically:
- Query usage per day
- Query usage per day per user
In Terra, when a Billing Project is set up, Owners on the project should be able to set these quotas to help researchers avoid accidental spending.
For further reading on controlling costs with BigQuery, see:
Comments
3 comments
Agreed!!! I worked with a team who are exploring terra and testing their data on a featured workflow. They ended up spending way to much accidently when they changed the configurations and used their own data.
Note on the implementation side:
- Cloud IAM currently has a role "Quota Administrator" (see the IAM documentation), but there is no more fine-grained "BigQuery Quota Administrator" role available. I have filed a feature request with Google Cloud for this.
That said, I think it would be overall very nice if Terra Project administrators had the ability to manage quotas. It is also useful to be able to manage Compute Engine quotas for running workflows and deploying notebook clusters.
I found the following really nice article around Compute Engine Quotas and FireCloud:
https://support.terra.bio/hc/en-us/articles/360029071251-CPUs-and-persistent-disk-quotas-What-are-they-and-how-do-you-request-more-
At the bottom, it indicates:
You can make quota requests by sending an email to fc-quota-requests @ googlegroups.com.
So that answers the question about Compute Engine Quotas. It does make sense for most users to work with the FireCloud/Terra team for quota management rather than directly with Google. There's a lot of context that the Terra team can provide.
I will reach out on that same address to see if we can get BigQuery quotas set as well.
Please sign in to leave a comment.