monitoring_script and monitoring_image options in Terra
Hi Team,
We'd like to be able to monitor resource utilization (cpu/memory/disk) of tasks in our workflows run on Terra. Are there plans to enable support for monitoring_script and monitoring_image for this use case?
Thanks
Comments
11 comments
Any updates on this by chance? Thanks!
Hi Kenny,
The feature has not yet been built, but I would be happy to reach out and let you know if that changes.
Kind regards,
Jason
Another vote for support of monitoring_script. It's very difficult to troubleshoot workflows without this capability.
I don't know the best answer, but just wanted to throw a couple links up here for the next time I search for this:
https://github.com/broadinstitute/dsp-scripts/blob/master/cromwell/methods/cromwell_monitoring_script.sh
https://github.com/broadinstitute/cromwell-monitor
runtime {...
monitor_image: quay.io/broadinstitute/cromwell-monitor
}
I believe you can do the above in a WDL, but then I am not clear on how to get the results. Denis Loginov any tips on how to view results? I don't know what StackDriver is or how to use it.
Stephen Fleming unfortunately not, both `monitoring_script` and `monitoring_image` options need to be supplied through Cromwell API call as workflow-level options (they are not available through `runtime` section of the tasks).
Terra team may have some updates for you soon however ;)
Okay thanks!
Are there any updates on this? Thanks!
Hi William,
Unfortunately these scripts have not yet been prioritized, but I will let the appropriate team know of your interest in the feature!
Kind Regards,
Anika
Just want to add another vote for this request. Beyond the flexibility of monitoring_script and monitoring_image, building some sort of even basic resource use reporting into the Terra UI would be very valuable to me. Being able to see the memory, disk, cpu usage of all of my tasks in Terra, as opposed to needing to purposefully modify my WDL to gather these metrics myself, would be really great.
Hi Chris Kachulis
Thank you for your note, I will add your comments and upvote to the request for this feature!
Kind Regards,
Anika
Please sign in to leave a comment.