Unending localization step
I am trying to run a genome-wide analysis using UK Biobank data, and during a recent run found that chromosomes 1 and 2 had not finished in five days (compared to <40hr for all other chromosomes). On looking at the logs, it appears that the workflow analysis step for these two chromosomes never got past the localization stage ("Localizing input: gs://.... -> /cromwell_root/....").
Might this have to do with hard disk provisioning? I wasn't sure based on the documentation and behavior whether the disk space is dynamically allocated as needed versus listening to the resource requested in the WDL (in this case, "local-disk 250GB HDD").
Comments
5 comments
Hi Kenneth,
We have opened a ticket with Google to investigate this as it seems to be a problem across many users. Can you please share the stdout, stderr, and *Task*.log files as well as the Operation ID so that we can pass it along to the team looking into this issue?
If you cannot post on the forum, feel free to email your information to Terra-support@broadinstitute.zendesk.com.
Jason
I will send an email with the info, thanks!
Are there any updates to this? I've had two jobs that are both stuck in localization steps for 30+ hours.
Hello Jett,
I just spoke and confirmed with the developer that is working with Google and they said that right now the investigation is in the hands of the Google team as we don't have much visibility into what is happening when the task is hanging.
What we know at this moment:
The log messages show that the localization transfer is starting but we are not able to confirm if it is hanging during the transfer or once the transfer is complete.
So far, this seems to be isolated to specific projects but we are not sure *why*. If you could provide the operations ID for any instances, we can also corroborate this hypothesis.
Sushma
Hello Kenneth and Jett,
You can follow updates on this issue here:
https://support.terra.bio/hc/en-us/community/posts/360056045911-Hanging-Localization-Step?page=1#community_comment_360009039632
Thank you,
Luda
Please sign in to leave a comment.