monitoring_script and monitoring_image options in Terra Completed
Hi Team,
We'd like to be able to monitor resource utilization (cpu/memory/disk) of tasks in our workflows run on Terra. Are there plans to enable support for monitoring_script and monitoring_image for this use case?
Thanks
Comments
16 comments
Any updates on this by chance? Thanks!
Hi Kenny,
The feature has not yet been built, but I would be happy to reach out and let you know if that changes.
Kind regards,
Jason
Another vote for support of monitoring_script. It's very difficult to troubleshoot workflows without this capability.
I don't know the best answer, but just wanted to throw a couple links up here for the next time I search for this:
https://github.com/broadinstitute/dsp-scripts/blob/master/cromwell/methods/cromwell_monitoring_script.sh
https://github.com/broadinstitute/cromwell-monitor
runtime {...
monitor_image: quay.io/broadinstitute/cromwell-monitor
}
I believe you can do the above in a WDL, but then I am not clear on how to get the results. Denis Loginov any tips on how to view results? I don't know what StackDriver is or how to use it.
Stephen Fleming unfortunately not, both `monitoring_script` and `monitoring_image` options need to be supplied through Cromwell API call as workflow-level options (they are not available through `runtime` section of the tasks).
Terra team may have some updates for you soon however ;)
Okay thanks!
Are there any updates on this? Thanks!
Hi William,
Unfortunately these scripts have not yet been prioritized, but I will let the appropriate team know of your interest in the feature!
Kind Regards,
Anika
Just want to add another vote for this request. Beyond the flexibility of monitoring_script and monitoring_image, building some sort of even basic resource use reporting into the Terra UI would be very valuable to me. Being able to see the memory, disk, cpu usage of all of my tasks in Terra, as opposed to needing to purposefully modify my WDL to gather these metrics myself, would be really great.
Hi Chris Kachulis
Thank you for your note, I will add your comments and upvote to the request for this feature!
Kind Regards,
Anika
Another up vote to have this feature available in the UI! Thank you.
Hi,
Thank you for writing in! I've already added your vote to the feature request and you'll be contacted if the feature gets built.
Best,
Josh
Thank you!
To anyone who visits this thread in the future, a quick fix may be
set -e
dstat -t --cpu --mem --disk --io > output.csv & <your bash commands>
#kill dstat pid once your command is executed
pkill dstat
This creates a CSV output file and will require more post-processing to get into a usable form.
Inspired from Firecloud's paper, published back in 2017 @ biorxiv.org/content/10.1101/209494v1.full.pdf http://dag.wiee.rs/home-made/dstat/ To install use !apt-get install dstat
Hi all,
Terra developer here. Pleased to report that this feature has shipped. Check it out!
Best,
Adam
Hello all! I am a product manager working on Terra and I'm excited to provide another update here. In addition to the monitoring_script, Google has recently improved their VM monitoring tools and we've added the ability for Terra GCP Workspace Owners and Workspace Writers with can_compute to access and create Monitoring Dashboards in the GCP Cloud Console.
These dashboards allow you to monitor interactive and batch VMs for performance and debugging purposes. You can also monitor quotas and setup alerts for any metric you might want to monitor in your workspaces. See https://cloud.google.com/monitoring/charts/dashboards for more information. Please check it out and let us know if you have any questions or feedback!
Please sign in to leave a comment.