Failed to check notebook status: "Autosave Failed!"
Hello!
I do a lot of work in Terra notebooks these days. Fantastic feature, thanks for creating it!
One thing that is a huge bummer is that if I'm working on a notebook, I sometimes get a message (red box in the Jupyter status bar) that says "Failed to check notebook status, changes may not be saved to workspace. Retrying..." It's accompanied by the notification "Autosave Failed!"
This happens more often than I'd like, and it is quite problematic, since I cannot continue to work in that notebook for fear of losing my changes. Often I have computations I've carried out in the notebook that took some time, and so having to restart the kernel and get back to the point I was at is a big pain. But I cannot figure out any way to get this error to go away once it appears. It seems like the state does not recover, and I'm unable to save the notebook from that point on, unless I restart the kernel and refresh the page completely.
It would be great if this didn't happen! It would really help me to trust and use notebooks in Terra even more.
Thanks very much,
Stephen
Comments
34 comments
Sometimes this state is accompanied by an issue with opening the Terminal. Here's an example where it hangs forever with the message "Almost ready..."
Hi Stephen,
We'll be happy to take a look! Can you provide the project name(s) and workspace(s) where you are seeing this happen? Is this happening on notebooks across different projects/workspaces, or in any one specific project/workspace?
And please confirm that your user email is sfleming@broadinstitute.org.
Many thanks,
Jason
Hi Jason,
Thanks! It's bayer-pcl-single-cell/bayer_pcl_single_cell
I have only ever used notebooks in Terra from within that particular workspace. And yes, that's my email.
Thanks!
Stephen
Hi Stephen,
To clarify, is the terminal open in a separate tab when this notebook error occurs?
Kind regards,
Jason
Hi Jason,
The terminal message is not really well correlated with the notebook error. Often the notebook error occurs without the terminal error. The notebook error is the real problem for me, and it occurs much more often. But the only times I've seen the terminal error (open in a separate tab at the same time) was when this notebook error was occurring, so I thought I'd just mention it in case there's some link. But there might be no link.
Thanks,
Stephen
Hello,
I am also frequently running into this error when I use my custom Docker environment. In addition to the error posted above, I will receive a prompt to reload, cancel, or overwrite my notebook. No matter what option I choose, within seconds, I am re-prompted with the same question. Afterwards, I will receive the same autosave error. I notice this error recently started occurring when I create folders and then notebooks (in the folders) using the Jupyter GUI.
Please let me know if there is any more information I can provide.
Thanks,
Andrew
Hi Stephen,
We are still looking into this issue—thank you for your patience.
Andrew, would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address.
Many thanks,
Jason
Hi Andrew,
Are you still experiencing this error? w=Would you be able to share the project name(s) and workspace(s) where you are seeing this happen? And please confirm your email address for Terra.
Kind regards,
Jason
Hi Jason,
Yes, I am still experiencing this error. My email is andrew.blair@gladstone.ucsf.edu and the workspace is bruneauHeartHub-BRM. Thanks for your help!
Best,
Andrew
Hi Andrew,
Can you provide the full workspace path, including the billing project?
Many thanks,
Jason
Hi Stephen,
Our developers have released a fix that should improve the experience of the notebook. They suspect that memory issues caused these original errors, and recommend deleting and recreating the cluster in order to get the fix. While the fix doesn't prevent someone from using all the available memory in the kernel, it has the notebook failing more gracefully when this does happen so that you can retry your notebook commands.
Please recreate the cluster and let us know if you experience any improvement in the notebook experience.
Many thanks,
Jason
Hi Jason,
Just as a note, sometimes when I get that "Autosave Failed!" message, if I refresh the page, I get this:
Don't know if that's any kind of a clue...
I have not deleted and re-created my cluster, but I will try that eventually. (Been putting it off because it's always hard for me to get a cluster set up the way I want it.)
Thanks!
Stephen
Hi Stephen,
Do let us know when you have a chance to re-create it, and let us know how it goes! Many thanks for your help in reporting details of the issue.
Jason
Hi Jason,
One more note that may be helpful to others who read this...
I have noticed very recently that some of the time when I get this error, my laptop has jumped onto a different wifi network (unexpectedly)! If I go back to my main wifi network, the problem sometimes resolves.
It may have been my own wifi connection issues the whole time... I'm still not 100% certain, but this is looking like a possibility.
Thanks again for your help,
Stephen
Hi Stephen,
Thanks for letting us know! This type of thing can often happen if you have an unprotected network in your network history, especially if it's of higher priority than the protected network. It may be worth removing or lowering priority of the unprotected network to keep this from happening in the future—I'd recommend removing it completely if you don't really need to use it.
Have you gotten a chance to recreate the cluster since the fix?
Kind regards,
Jason
Hi Terra team,
Thanks for making Terra as brilliant as it is and for providing great user support! I wanted to ask whether you have any solution for the aforementioned error. Unfortunately, it happens very often to me and my colleagues and it makes our work a little bit complicated. From my experience, this behavior is not related to a wifi connection, but rather to a particular notebook. I suspect that this issue arises once a notebook gets too large (over ~60Mb).
Any info will be appreciated! Best,
Veronika
Hi Jason,
I wanted to weigh back in since I realized I never responded to your last question two years ago :)
When I re-created my Cloud Environment, things did improve to a large extent. However, I still see the error come up from time to time. But it is not as frequent as it used to be. I do think the issue was related to notebook size.
Since I saw Veronika post, I figured I'd just mention... I used to have problems once notebooks reached ~15MB. Seems like the fix you mentioned before helped with that. But it sounds like there is still some kind of upper limit that Veronika is now running into ~60MB. I wonder why large notebooks should be any more trouble than small ones?
-Stephen
Hi Verča Niederlová,
Does the "Autosave Failed!" error happen randomly when you're in your notebook, or do you notice it happen after running certain commands? Can you share a screenshot of your current cloud environment configuration (gear icon at the top-right of workspace)? And if possible, can you also share your workspace with GROUP_FireCloud-Support@firecloud.org so we can try to reproduce the error? Let us know the workspace name and which notebook is giving you issues.
Best,
Samantha
Hi Samantha,
Thanks for your reply and for trying to solve this issue! I haven't found out when exactly the error happens, but I think it might be after the notebook exceeds some size. Importantly, I've never been able to get rid of the error in a notebook once it happens for the first time. No matter what I try (changing the environment options, copying the notebook, renaming), after it happens once, the notebook is doomed forever. The workspace is shalek-lab-firecloud/PREDICT-Study-5prime-Workflows, notebook for example vn-R-kraken-5p-v2.ipynb.
Environment configuration:
I've shared the workspace as requested. Also thanks to Stephen for sharing updated info. Hope we'll be able to come up with a solution!
Thank you,
Veronika
Hi Verča Niederlová,
Sorry for the delayed response. There's not really a set limit to the notebook size, but you may be seeing the error because the notebook size is larger than the memory you allocated to your cloud environment. Since you mentioned your notebook is ~60GB, can you try increasing the CPU/GB for your cloud environment next time?
Best,
Samantha
Hi Samantha,
If I understood right, I think the notebook size was 60MB, rather than 60GB. While this is (extremely) large for a jupyter notebook, it is still orders of magnitude smaller than the available memory.
-Stephen
Hi Samantha and Stephen,
Yes, Stephen is right, the size of the notebook was 60MB, not GB. The issue can't be resolved by increasing the environment memory, unfortunately. I am wondering whether you were able to replicate the error in the environment that I've shared with you? Let me know if I can add any more information that would be helpful in resolving the issue!
Best,
Veronika
Hi Samantha and Jason,
Just wanted to re-post here because I still do see this issue from time to time. If you can, it'd be great to suggest to someone on the Terra interactive analysis engineering team that it would be fantastically useful if they would track metrics about how often users see these kinds of errors, and maybe then they could track down where they come from. I suspect that users are still seeing this issue a whole lot.
Thanks!
Hi Stephen,
Thanks for writing in! I am contacting the IA team about the possibility of tracking this error. Is it possible for you to give us some recent runtimes during which you experienced this error? This way the team may be able to look into the logs and see if they notice anything unusual.
Kind regards,
Shoaib
Thanks Shoaib! Yes, I recently saw the error several times (today) on the bayer-pcl-single-cell/bayer_pcl_human_PV workspace. The runtime details are
Hi Stephen,
Thanks for the info! Also just to confirm, are you still experiencing the same "Autosave failed" error? Is it possible to get a screenshot of the banner you are seeing?
Kind regards,
Shoaib
Hi Stephen,
We haven't heard from you in a while so we're going to close out this ticket. If you still require assistance, simply respond to this post and we'll be happy to pick up where we left off!
Kind regards,
Shoaib
Hi Shoaib,
Sorry, I didn't see your previous post. Yes I am still experiencing the error intermittently. The banner is the same as the very first screenshot I posted in this thread. Identical.
Thanks,
Stephen
Hi Stephen,
No worries, I will reopen the ticket. I reached out to the engineers and we're going to take a look into the issue.
Kind regards,
Shoaib
I still see it, even today. Sometimes it does resolve on its own, so it is less of a problem than it used to be.
I wonder if it has something to do with authentication, or with very costly compute operations that tie up all the compute / memory.
Please sign in to leave a comment.