GLnexus process killed before completion
Hello,
I've been trying for several days to run glnexus and the process keeps getting killed. It typically reports that it found "Killed" in the stderr and calls it as out of memory; that supposition becomes absurd as I trim the samples down from 3000 to 100 and each is roughly 1 Mb compressed (<sample>.vcf.gz). The workspace and workflow are here, I've shared it to support. It's a very simple workflow, I've made a few small revisions trying to get it to work, and I've tabulated the runtime attributes I've used. It does retry, though I didn't think it was configured to do so, using the preemptible count. The longest attempt I've seen is around 90 minutes for 3k.
| samples | mem (Gb) | cpus | disk (Gb) |
| 3k | 64 | 16 | 1000 |
| 3k | 64 | " | 1000 |
| 3k | 64 | " | 3000 |
| 5 hundred | 64 | " | 3000 |
| 5h | 256 | " | 3000 |
| 5h | 256 | " | 3000 |
| 5h | 16 | " | 200 |
| 1h | 64 | " | 400 |
| 1h | 64 | " | 400 |
| 1h | 64 | " | 400 |
| 1h | 64 | " | 400 |
| 5h | 64 | " | 400 |
| 5h | 256 | " | 2000 |
| 1h | 64 | " | 1000 |
| 1h | 256 | 32 | 1000 ssd |
Comments
3 comments
Hi Kyler Anderson,
It looks like the failure is indeed due to insufficient memory. The issue is that you are not specifying a unit of memory, so the default is just bytes, rather than GB.
As you can see in the compute details of the most recently failed workflow, the memory that was allocated was less than 1 GB, while the amount you requested was "256":
To resolve this, you'll need to update the memory runtime attribute to include the appropriate unit:
Best,
Samantha
So this documentation here, which says that for memory:
"Google Cloud interprets this as the minimum amount of memory to use. If you request more CPUs than makes sense for your memory request, Google automatically bumps up to the appropriate amount of memory. Example: You cannot request 8 CPUs and 1 GB memory, since each CPU needs between 1-6.5 GB of memory."
Is not true?
Hi Kyler Anderson,
That statement is still true. The issue is that you are not actually providing a CPU runtime attribute due to a typo, so it's using the default of 1 CPU. The correct runtime attribute is
cpu, rather thancpus, which is what you have in your WDL.Best,
Samantha
Please sign in to leave a comment.