Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Failed to check notebook status: "Autosave Failed!"

Comments

22 comments

  • Avatar
    Stephen Fleming

    Sometimes this state is accompanied by an issue with opening the Terminal.  Here's an example where it hangs forever with the message "Almost ready..."

     

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

     

    We'll be happy to take a look! Can you provide the project name(s) and workspace(s) where you are seeing this happen? Is this happening on notebooks across different projects/workspaces, or in any one specific project/workspace?

    And please confirm that your user email is sfleming@broadinstitute.org.

     

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Jason,

     

    Thanks!  It's bayer-pcl-single-cell/bayer_pcl_single_cell

    I have only ever used notebooks in Terra from within that particular workspace.  And yes, that's my email.

     

    Thanks!

    Stephen

     

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

     

    To clarify, is the terminal open in a separate tab when this notebook error occurs?

     

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Jason,

     

    The terminal message is not really well correlated with the notebook error.  Often the notebook error occurs without the terminal error.  The notebook error is the real problem for me, and it occurs much more often.  But the only times I've seen the terminal error (open in a separate tab at the same time) was when this notebook error was occurring, so I thought I'd just mention it in case there's some link.  But there might be no link.

     

    Thanks,

    Stephen

    0
    Comment actions Permalink
  • Avatar
    Andrew Blair

    Hello,

    I am also frequently running into this error when I use my custom Docker environment. In addition to the error posted above, I will receive a prompt to reload, cancel, or overwrite my notebook. No matter what option I choose, within seconds, I am re-prompted with the same question. Afterwards, I will receive the same autosave error. I notice this error recently started occurring when I create folders and then notebooks (in the folders) using the Jupyter GUI.

    Please let me know if there is any more information I can provide.

    Thanks,

    Andrew

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

    We are still looking into this issue—thank you for your patience.

     

    Andrew, would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address.

     

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Andrew,

    Are you still experiencing this error? w=Would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address for Terra.

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Andrew Blair

    Hi Jason,

    Yes, I am still experiencing this error. My email is andrew.blair@gladstone.ucsf.edu and the workspace is bruneauHeartHub-BRM. Thanks for your help!

    Best,

    Andrew

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Andrew,

    Can you provide the full workspace path, including the billing project?

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

    Our developers have released a fix that should improve the experience of the notebook. They suspect that memory issues caused these original errors, and recommend deleting and recreating the cluster in order to get the fix. While the fix doesn't prevent someone from using all the available memory in the kernel, it has the notebook failing more gracefully when this does happen so that you can retry your notebook commands.

    Please recreate the cluster and let us know if you experience any improvement in the notebook experience.

    Many thanks,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Jason,

    Just as a note, sometimes when I get that "Autosave Failed!" message, if I refresh the page, I get this:

     

     

    Don't know if that's any kind of a clue...

    I have not deleted and re-created my cluster, but I will try that eventually.  (Been putting it off because it's always hard for me to get a cluster set up the way I want it.)

     

    Thanks!

    Stephen

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

    Do let us know when you have a chance to re-create it, and let us know how it goes! Many thanks for your help in reporting details of the issue.

    Jason

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Jason,

    One more note that may be helpful to others who read this...

    I have noticed very recently that some of the time when I get this error, my laptop has jumped onto a different wifi network (unexpectedly)!  If I go back to my main wifi network, the problem sometimes resolves.

    It may have been my own wifi connection issues the whole time... I'm still not 100% certain, but this is looking like a possibility.

    Thanks again for your help,
    Stephen

     

    0
    Comment actions Permalink
  • Avatar
    Jason Cerrato

    Hi Stephen,

    Thanks for letting us know! This type of thing can often happen if you have an unprotected network in your network history, especially if it's of higher priority than the protected network. It may be worth removing or lowering priority of the unprotected network to keep this from happening in the future—I'd recommend removing it completely if you don't really need to use it.

    Have you gotten a chance to recreate the cluster since the fix?

    Kind regards,

    Jason

    0
    Comment actions Permalink
  • Avatar
    Verča Niederlová

    Hi Terra team,

    Thanks for making Terra as brilliant as it is and for providing great user support! I wanted to ask whether you have any solution for the aforementioned error. Unfortunately, it happens very often to me and my colleagues and it makes our work a little bit complicated. From my experience, this behavior is not related to a wifi connection, but rather to a particular notebook. I suspect that this issue arises once a notebook gets too large (over ~60Mb). 

    Any info will be appreciated! Best,

    Veronika

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Jason,

    I wanted to weigh back in since I realized I never responded to your last question two years ago :)

    When I re-created my Cloud Environment, things did improve to a large extent.  However, I still see the error come up from time to time.  But it is not as frequent as it used to be.  I do think the issue was related to notebook size.

    Since I saw Veronika post, I figured I'd just mention... I used to have problems once notebooks reached ~15MB.  Seems like the fix you mentioned before helped with that.  But it sounds like there is still some kind of upper limit that Veronika is now running into ~60MB.  I wonder why large notebooks should be any more trouble than small ones?

    -Stephen

    0
    Comment actions Permalink
  • Avatar
    Samantha (she/her)

    Hi Verča Niederlová,

    Does the "Autosave Failed!" error happen randomly when you're in your notebook, or do you notice it happen after running certain commands? Can you share a screenshot of your current cloud environment configuration (gear icon at the top-right of workspace)? And if possible, can you also share your workspace with GROUP_FireCloud-Support@firecloud.org so we can try to reproduce the error? Let us know the workspace name and which notebook is giving you issues.

    Best,
    Samantha

    0
    Comment actions Permalink
  • Avatar
    Verča Niederlová

    Hi Samantha,

    Thanks for your reply and for trying to solve this issue! I haven't found out when exactly the error happens, but I think it might be after the notebook exceeds some size. Importantly, I've never been able to get rid of the error in a notebook once it happens for the first time. No matter what I try (changing the environment options, copying the notebook, renaming), after it happens once, the notebook is doomed forever. The workspace is shalek-lab-firecloud/PREDICT-Study-5prime-Workflows, notebook for example vn-R-kraken-5p-v2.ipynb.
    Environment configuration:

    I've shared the workspace as requested. Also thanks to Stephen for sharing updated info. Hope we'll be able to come up with a solution!

    Thank you,

    Veronika

    0
    Comment actions Permalink
  • Avatar
    Samantha (she/her)

    Hi Verča Niederlová,

    Sorry for the delayed response. There's not really a set limit to the notebook size, but you may be seeing the error because the notebook size is larger than the memory you allocated to your cloud environment. Since you mentioned your notebook is ~60GB, can you try increasing the CPU/GB for your cloud environment next time?

    Best,
    Samantha

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Hi Samantha,

    If I understood right, I think the notebook size was 60MB, rather than 60GB.  While this is (extremely) large for a jupyter notebook, it is still orders of magnitude smaller than the available memory.

    -Stephen

    0
    Comment actions Permalink
  • Avatar
    Verča Niederlová

    Hi Samantha and Stephen,

    Yes, Stephen is right, the size of the notebook was 60MB, not GB. The issue can't be resolved by increasing the environment memory, unfortunately. I am wondering whether you were able to replicate the error in the environment that I've shared with you? Let me know if I can add any more information that would be helpful in resolving the issue!

    Best,

    Veronika

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk