As part of my 5000 submissions of BAM->gVCF workflow I run HaplotypeCaller with preemptible=3. looking at the operations (via firecloud) I see that I'm actually getting non-preemptible machines, and being charged accordingly (4x higher).
Concurrently, many of my workflows are stalling due to lack of resources (IP addresses, Disk, CPU). I'm wondering if these two elements are connected...could it be that cromwell is interpreting the lack of quota as a re-tryable failure, and then not giving my a preemptible node despite having asked for one?
For example, take a look at workflow id: 57f9be05-1d39-4cb0-9059-0b5726506c9b