A couple of days ago I started a large job ($33.35) to process DNA microarray data from 27,778 samples. The job more or less completed in 40hrs:
The OUTPUTS tab has been populated and I can indeed access the output files just fine. However, it is still labeled as Running. My best guess is that it is stuck somewhere in the delete intermediate files step. The main bulk of this operation would certainly be the deletion of 27,778 files created in the first scattered task. This was a task scattered in 580 shards so that effectively those 27,778 files were localized in 580 directories with ~48 files each.
When I have looked a those directories about an hour ago, each directory contains now 30-35 files, for a total of 18,180 files. So something like a third of these intermediate files have indeed been deleted but then the deletion step must have gotten frozen or something as when I checked again later the number was still 18,180. So not sure what went wrong.
These are the LABELS for the job: