Terra workspace deleted itself?!
I was working on in a workspace's notebook section, using FISS to attempt to upload a TSV file to Terra. While testing, but without changing any of my FISS commands (ie, I didn't use accidentally use an API command related to deleting a workspace, if such a command even exists), I suddenly started getting FISS errors because the workspace couldn't be found. Got confused and checked my workspace list... it's not there. Went to the URL of the workspace and it says...
Could not display workspace
You are trying to access a workspace that either does not exist, or you do not have access to it.
So now I can't get to anything I had in there -- not the dashboard, the workflows, or the notebooks. This is the URL: https://app.terra.bio/#workspaces/fccredits-xenon-denim-7890/TOPMed%20Alignment%20and%20Freeze8%20Variant%20Calling
There is no auth domain (doesn't contain controlled access data so I didn't set one) and I'm the owner and creator of this workspace. I'm still logged in and can access other workspaces. It's on a free credits account, but since there's no way to check how many free credits you have, surely a workspace wouldn't delete itself when you run out, right?
Comments
5 comments
Hi Aisling,
That seems very unexpected, though I'm not so familiar with how FISS works (it isn't built by or maintained by the Terra team). I will flag this up with our engineers for investigation and get back to you as soon as possible.
Kind regards,
Jason
Hi Aisling,
One of our engineers was able to identify the moment the workspace was deleted.
Mar 26 18:04:14 gce-rawls-prod803 rawls-proxy[1604]: 35.191.3.232 - 117475951917658785644 "SNIP" "Saturn" [2020-03-26T18:04:12.677-0400] 2161114 + "DELETE /api/workspaces/fccredits-xenon-denim-7890/TOPMed%20Alignment%20and%20Freeze8%20Variant%20Calling HTTP/1.1" 202 86 "https://app.terra.bio/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
The log indicates that the workspace was deleted using the Terra UI at Mar 26 18:04:14 Eastern time. The log specifically shows it was deleted through the UI, rather than through FISS or API call.
Kind regards,
Jason
At the risk of sounding like that one patient on Doctor House who insisted she knows how her inhaler works and then proceeded to spray it on her neck, it's doesn't seem possible for that to be the case. I have deleted workspaces via the UI before; it's very clear when you're deleting a workspace. It's not something that one can accidentally do. Is it possible to see if someone else deleted it by mistake? The workspace was shared with multiple people all at different IP addresses.
Hi Aisling,
I deeply sympathize with what you must be feeling, hearing this report—it's certainly unbelievable that one would be capable of deleting a workspace in the UI by accident. I can confirm that the email address tied to the action is the same one you are using to contact us here.
One of our engineers investigated whether there was a way to produce the very same error log using the command line, and found that you can create an identical log if you copied the network request from the UI (which copies all of the headers) from another deletion, and changed the workspace name. Using this, it's feasible that the workspace was deleted using the command line if you had copy-pasted any previously used commands that were used to delete a workspace. Based on your initial report, this doesn't sound likely, but it's just about the only way this could show up in the logs looking like this, where it shows the deletion having come from https://app.terra.bio on a browser without actually having been done in a browser directly.
The only other way this could have happened was if you were somehow logged in (on the UI or in gcloud) somewhere else and someone else deleted it with your account.
If you have any idea at all of how this may have happened, please let us know.
Jason
I managed to grab the Jupyter notebook I was working on when the workspace was deleted (it was still able to execute commands even as the workspace itself was deleted, strangely enough) plus another one from about two hours prior. Would that be of any use to your engineers? If so, where should I forward that? I'm on the BioData Catalyst slack; that might be easiest if your engineers are on there.
Please sign in to leave a comment.