Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

How often does estimated monthly storage update?

Comments

10 comments

  • Avatar
    Jason Cerrato

    Hi Brendan,

    Thanks for writing in. Can you speak to how you know that the workspaces are taking several days or longer to update? Are you running workflows and using more storage but not seeing changes in the estimate?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    breardon

    Hi Jason,

    I deleted the underlying data in a couple of workspace's buckets with gsutil about a week ago. As of Tuesday they were not updated but they are now. Can you give some insight into how frequently Terra will update the cost estimate? Does it need to be triggered by something? 

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Brendan,

    Thanks for the details. I'll look to the engineers for some insight and get back to you.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Brendan,

    When you say you deleted the underlying data, are you saying that you deleted all of the bucket's data? Does the workspace bucket currently have data in it or is it empty?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    breardon

    I uploaded a text file to the bucket afterwards so the bucket is not presently empty but I used `cat "$file_containing_list_of_paths" | gsutil -m rm -I` to remove files from the bucket. The bucket was otherwise empty. 

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Brendan,

    To estimate storage costs we use Google's provided storage log file, which is generated daily. We discovered recently, however, that Google does not seem to generate a new file in the case that the bucket is emptied and has no contents. The "trigger" so to speak would be that the bucket has contents.

    Does this seem accurate with your experience? One of our engineering teams has an item on their backlog to handle cases where the bucket is empty and reflect that accurately in the estimate.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    breardon

    Thanks good to know, Jason. Thank you! Hm. I added a file afterwards and thus, since the bucket is not empty, that should trigger Google to update the storage estimate, no? 

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Brendan,

    That would be my assumption as well. I presume the file had content/size? I can add this information to the backlog ticket. Was the file added right after the deletion a week ago?

    Jason

    0
    Comment actions Permalink
  • Avatar
    breardon

    Correct to both of those items. 

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Great - I've added the details to the ticket for the team to investigate when they revisit the logic for the cost estimator. Thanks for letting us know!

    If you notice that the estimate is delaying again in the future despite the bucket not having been empty, please let us know.

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk