I ran the TOPMed Aligner WDL on 10 controlled-access CRAMs to benchmark costs as part of BioData Catalysts W3.1's benchmarking goal. I noticed the time that some steps took to complete seemed off, so I manually checked the logs alongside my team. In at least four instances the logs indicate that the time that the tasks actually took to execute was far above what is shown on Terra's UI. All of them (highlighted yellow) should have a reported time of about 24 hours or so, but they're being reported as either about 8 hours or as no time at all.
(The other zeros in the pre-align step are likely not directly related to this issue; according to the logs they were preempted within about a minute or two. They are still a bit concerning because it's unusual for Google to preempt disks so quickly and so often, especially when compared to the align step's preemptibles. But since that has little effect on cost or time it takes to execute a workflow I think that's less of priority.)
If these misreports are also what's going through FISS, then this is a likely culprit to the workflow cost estimator notebook under development by Brian Hannafious being about $50 off for this submission's total $80ish cost. That notebook is being used to generate estimates for Biodata Catalyst and AnVIL users, so there is a practical reason for these time estimates being correctly reported.
Please sign in to leave a comment.