Failed to check notebook status: "Autosave Failed!"

Post author
Stephen Fleming

Hello!

I do a lot of work in Terra notebooks these days.  Fantastic feature, thanks for creating it!

 

One thing that is a huge bummer is that if I'm working on a notebook, I sometimes get a message (red box in the Jupyter status bar) that says "Failed to check notebook status, changes may not be saved to workspace.  Retrying..."  It's accompanied by the notification "Autosave Failed!"

 

This happens more often than I'd like, and it is quite problematic, since I cannot continue to work in that notebook for fear of losing my changes.  Often I have computations I've carried out in the notebook that took some time, and so having to restart the kernel and get back to the point I was at is a big pain.  But I cannot figure out any way to get this error to go away once it appears.  It seems like the state does not recover, and I'm unable to save the notebook from that point on, unless I restart the kernel and refresh the page completely.

 

It would be great if this didn't happen!  It would really help me to trust and use notebooks in Terra even more.

 

Thanks very much,

Stephen

 

 

 

Comments

22 comments

  • Comment author
    Stephen Fleming

    Sometimes this state is accompanied by an issue with opening the Terminal.  Here's an example where it hangs forever with the message "Almost ready..."

     

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

     

    We'll be happy to take a look! Can you provide the project name(s) and workspace(s) where you are seeing this happen? Is this happening on notebooks across different projects/workspaces, or in any one specific project/workspace?

    And please confirm that your user email is sfleming@broadinstitute.org.

     

    Many thanks,

    Jason

    0
  • Comment author
    Stephen Fleming

    Hi Jason,

     

    Thanks!  It's bayer-pcl-single-cell/bayer_pcl_single_cell

    I have only ever used notebooks in Terra from within that particular workspace.  And yes, that's my email.

     

    Thanks!

    Stephen

     

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

     

    To clarify, is the terminal open in a separate tab when this notebook error occurs?

     

    Kind regards,

    Jason

    0
  • Comment author
    Stephen Fleming
    • Edited

    Hi Jason,

     

    The terminal message is not really well correlated with the notebook error.  Often the notebook error occurs without the terminal error.  The notebook error is the real problem for me, and it occurs much more often.  But the only times I've seen the terminal error (open in a separate tab at the same time) was when this notebook error was occurring, so I thought I'd just mention it in case there's some link.  But there might be no link.

     

    Thanks,

    Stephen

    0
  • Comment author
    Andrew Blair

    Hello,

    I am also frequently running into this error when I use my custom Docker environment. In addition to the error posted above, I will receive a prompt to reload, cancel, or overwrite my notebook. No matter what option I choose, within seconds, I am re-prompted with the same question. Afterwards, I will receive the same autosave error. I notice this error recently started occurring when I create folders and then notebooks (in the folders) using the Jupyter GUI.

    Please let me know if there is any more information I can provide.

    Thanks,

    Andrew

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

    We are still looking into this issue—thank you for your patience.

     

    Andrew, would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address.

     

    Many thanks,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Andrew,

    Are you still experiencing this error? w=Would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address for Terra.

    Kind regards,

    Jason

    0
  • Comment author
    Andrew Blair

    Hi Jason,

    Yes, I am still experiencing this error. My email is andrew.blair@gladstone.ucsf.edu and the workspace is bruneauHeartHub-BRM. Thanks for your help!

    Best,

    Andrew

    0
  • Comment author
    Jason Cerrato

    Hi Andrew,

    Can you provide the full workspace path, including the billing project?

    Many thanks,

    Jason

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

    Our developers have released a fix that should improve the experience of the notebook. They suspect that memory issues caused these original errors, and recommend deleting and recreating the cluster in order to get the fix. While the fix doesn't prevent someone from using all the available memory in the kernel, it has the notebook failing more gracefully when this does happen so that you can retry your notebook commands.

    Please recreate the cluster and let us know if you experience any improvement in the notebook experience.

    Many thanks,

    Jason

    0
  • Comment author
    Stephen Fleming

    Hi Jason,

    Just as a note, sometimes when I get that "Autosave Failed!" message, if I refresh the page, I get this:

     

     

    Don't know if that's any kind of a clue...

    I have not deleted and re-created my cluster, but I will try that eventually.  (Been putting it off because it's always hard for me to get a cluster set up the way I want it.)

     

    Thanks!

    Stephen

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

    Do let us know when you have a chance to re-create it, and let us know how it goes! Many thanks for your help in reporting details of the issue.

    Jason

    0
  • Comment author
    Stephen Fleming

    Hi Jason,

    One more note that may be helpful to others who read this...

    I have noticed very recently that some of the time when I get this error, my laptop has jumped onto a different wifi network (unexpectedly)!  If I go back to my main wifi network, the problem sometimes resolves.

    It may have been my own wifi connection issues the whole time... I'm still not 100% certain, but this is looking like a possibility.

    Thanks again for your help,
    Stephen

     

    0
  • Comment author
    Jason Cerrato

    Hi Stephen,

    Thanks for letting us know! This type of thing can often happen if you have an unprotected network in your network history, especially if it's of higher priority than the protected network. It may be worth removing or lowering priority of the unprotected network to keep this from happening in the future—I'd recommend removing it completely if you don't really need to use it.

    Have you gotten a chance to recreate the cluster since the fix?

    Kind regards,

    Jason

    0
  • Comment author
    Verča Niederlová

    Hi Terra team,

    Thanks for making Terra as brilliant as it is and for providing great user support! I wanted to ask whether you have any solution for the aforementioned error. Unfortunately, it happens very often to me and my colleagues and it makes our work a little bit complicated. From my experience, this behavior is not related to a wifi connection, but rather to a particular notebook. I suspect that this issue arises once a notebook gets too large (over ~60Mb). 

    Any info will be appreciated! Best,

    Veronika

    0
  • Comment author
    Stephen Fleming

    Hi Jason,

    I wanted to weigh back in since I realized I never responded to your last question two years ago :)

    When I re-created my Cloud Environment, things did improve to a large extent.  However, I still see the error come up from time to time.  But it is not as frequent as it used to be.  I do think the issue was related to notebook size.

    Since I saw Veronika post, I figured I'd just mention... I used to have problems once notebooks reached ~15MB.  Seems like the fix you mentioned before helped with that.  But it sounds like there is still some kind of upper limit that Veronika is now running into ~60MB.  I wonder why large notebooks should be any more trouble than small ones?

    -Stephen

    0
  • Comment author
    Samantha (she/her)

    Hi Verča Niederlová,

    Does the "Autosave Failed!" error happen randomly when you're in your notebook, or do you notice it happen after running certain commands? Can you share a screenshot of your current cloud environment configuration (gear icon at the top-right of workspace)? And if possible, can you also share your workspace with GROUP_FireCloud-Support@firecloud.org so we can try to reproduce the error? Let us know the workspace name and which notebook is giving you issues.

    Best,
    Samantha

    0
  • Comment author
    Verča Niederlová
    • Edited

    Hi Samantha,

    Thanks for your reply and for trying to solve this issue! I haven't found out when exactly the error happens, but I think it might be after the notebook exceeds some size. Importantly, I've never been able to get rid of the error in a notebook once it happens for the first time. No matter what I try (changing the environment options, copying the notebook, renaming), after it happens once, the notebook is doomed forever. The workspace is shalek-lab-firecloud/PREDICT-Study-5prime-Workflows, notebook for example vn-R-kraken-5p-v2.ipynb.
    Environment configuration:

    I've shared the workspace as requested. Also thanks to Stephen for sharing updated info. Hope we'll be able to come up with a solution!

    Thank you,

    Veronika

    0
  • Comment author
    Samantha (she/her)

    Hi Verča Niederlová,

    Sorry for the delayed response. There's not really a set limit to the notebook size, but you may be seeing the error because the notebook size is larger than the memory you allocated to your cloud environment. Since you mentioned your notebook is ~60GB, can you try increasing the CPU/GB for your cloud environment next time?

    Best,
    Samantha

    0
  • Comment author
    Stephen Fleming

    Hi Samantha,

    If I understood right, I think the notebook size was 60MB, rather than 60GB.  While this is (extremely) large for a jupyter notebook, it is still orders of magnitude smaller than the available memory.

    -Stephen

    0
  • Comment author
    Verča Niederlová

    Hi Samantha and Stephen,

    Yes, Stephen is right, the size of the notebook was 60MB, not GB. The issue can't be resolved by increasing the environment memory, unfortunately. I am wondering whether you were able to replicate the error in the environment that I've shared with you? Let me know if I can add any more information that would be helpful in resolving the issue!

    Best,

    Veronika

    0

Please sign in to leave a comment.