monitoring_script and monitoring_image options in Terra

Post author
Denis Loginov

Hi Team,

We'd like to be able to monitor resource utilization (cpu/memory/disk) of tasks in our workflows run on Terra. Are there plans to enable support for monitoring_script and monitoring_image for this use case?

Thanks

Comments

11 comments

  • Comment author
    Kenneth Westerman

    Any updates on this by chance? Thanks!

    0
  • Comment author
    Jason Cerrato

    Hi Kenny,

    The feature has not yet been built, but I would be happy to reach out and let you know if that changes.

    Kind regards,

    Jason

    0
  • Comment author
    Haley Abel

    Another vote for support of monitoring_script.  It's very difficult to troubleshoot workflows without this capability.

    0
  • Comment author
    Stephen Fleming

    I don't know the best answer, but just wanted to throw a couple links up here for the next time I search for this:

    https://github.com/broadinstitute/dsp-scripts/blob/master/cromwell/methods/cromwell_monitoring_script.sh

    https://github.com/broadinstitute/cromwell-monitor

    0
  • Comment author
    Stephen Fleming
    runtime {
    ...
        monitor_image: quay.io/broadinstitute/cromwell-monitor
    }

    I believe you can do the above in a WDL, but then I am not clear on how to get the results.  Denis Loginov any tips on how to view results?  I don't know what StackDriver is or how to use it.

    0
  • Comment author
    Denis Loginov

    Stephen Fleming unfortunately not, both `monitoring_script` and `monitoring_image` options need to be supplied through Cromwell API call as workflow-level options (they are not available through `runtime` section of the tasks).

    Terra team may have some updates for you soon however ;)

    0
  • Comment author
    Stephen Fleming

    Okay thanks!

    0
  • Comment author
    William Thistlethwaite

    Are there any updates on this? Thanks!

    0
  • Comment author
    Anika Das

    Hi William, 

    Unfortunately these scripts have not yet been prioritized, but I will let the appropriate team know of your interest in the feature!

    Kind Regards, 

    Anika

    0
  • Comment author
    Chris Kachulis

    Just want to add another vote for this request.  Beyond the flexibility of monitoring_script and monitoring_image, building some sort of even basic resource use reporting into the Terra UI would be very valuable to me.  Being able to see the memory, disk, cpu usage of all of my tasks in Terra, as opposed to needing to purposefully modify my WDL to gather these metrics myself, would be really great.  

    3
  • Comment author
    Anika Das

    Hi Chris Kachulis

    Thank you for your note, I will add your comments and upvote to the request for this feature!

    Kind Regards, 

    Anika

    0

Please sign in to leave a comment.