Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Intermittent StorageExceptions (unknown host www.googleapis.com)

Comments

2 comments

  • Avatar
    Tiffany Miller

    This has been resolved via another ticket. 

    0
    Comment actions Permalink
  • Avatar
    Bhandsaker

    It seems worth recording here the cause of this issue. After quite a bit of debugging and experimentation, we determined that the root cause was that the compute instance was experiencing memory pressure, which presumably was causing the DNS lookups to begin failing. One hypothesis, not confirmed to my knowledge, is that there is a process running on the compute instance that handles DNS lookups and if the compute instance is low on virtual memory this process can die causing all future DNS lookups to fail.

    The solution in this case was to increase the memory allocated to the compute instance, which eliminated the problem. We found that if we increased or decreased the allocated memory slightly, we could control the rate of failures due to these storage exceptions (less memory -> more memory pressure -> higher rate of DNS lookups leading to storage exceptions).

    It is not at all obvious from the symptom (StorageException caused by no-such-host) that the root cause is lack of memory on the compute instance.

     

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk