I’m starting to write this post while trying to recover the sources for my website from the unnecessary and irrecoverable damage Syncthing has done to me about a week ago.
I used Syncthing to sync my documents, photos and passwords between my Android phone and my computer. The phone (mistakenly) errored that there was no more space in the device, and apps wouldn’t work, frequent random crashes from background apps would render the phone unusable. So I decided I stop Syncthing on my computer and delete synced stuff on the phone, just to be able to fix the problem. My folders were configured so that the copy on the computer was read only, i.e. modifications made on the phone would not propagate. Depending on that, I just stopped the daemon on my system and deleted the synced files from my phone. Finally became able to try and fix the issue, turns out it thought the space Spotify took on the external SD card was occupied not only there but also on the system storage space; a reboot fixed it, and because I had plenty of space left, I started the Syncthing daemon on my computer, and got busy doing other things while waiting it to sync in the background.
When I realised it was not syncing, I checked the Syncthing app on the phone. I saw that it complained that the
.stfolders did not exist. So I went ahead and created them, but by mistake, I created empty text files instead of empty folders. Did not realise it and left it to work in the background. Maybe half an hour past by, and I realised it was not syncing. So I went and looked at my computer and somehow realised that all the data that was meant to be synced was deleted for good, with no chance for recovery. After inital shock, I immediately remounted the filesystem in readonly mode (
mount -o remount,ro), and after inspecting the damage and rebooting into a live Ubuntu environment, tried a variety of tools to recover files, online and offline (e.g. extundelete, testdisk), to no avail.
I lost my documents tree, which included prose and notes I’ve written, including the source to this website, and all my reading and research notes. I also lost my pass database, which contained all my paswords. Luckily my Org mode files tree were intact, which was the most frequently edited subtree under the documents tree. My notes and prose work was saved with minimal (but still non negligible) loss. The documents tree also contained all sorts of documents that pertain to legal stuff, documents from institutions, etc.
I had, by chance, a backup from November 19th in a USB stick. After spending a couple days w/o any tangible recovery, I decided to use that backup. I was, up until this disaster, postponing setting up a proper back up system on my computer. So I ended up only losing a month’s work, documents and passwords. I still don’t know the actual impact on documents, but hopefully nothing important was permanently lost.
Right after this incident I made a backup tarball of my files and repartitioned my hard drive so that half of it (~200GB) is spared for backups only. I also started making weekly full backups, and copying some of those to my brother’s computer.
I have to transcribe some prose work again, that’s luckily still availabe on paper. I am also in the process of recovering this website, which suffered permanent loss, but luckily I was staging in a different tree before publishing. I have lost the new shell scripts that pertain to the move to GitLab Pages, and my Let’s Encrypt configuration. I may have lost some important data permanently and may be unaware of it. I am able to continue using my computer now, though.
The major takeaway from this is what the post description says: Syncthing is not a safe file synchronisation utility. Avoid using it. It has costed me more than a week’s productivity and a month’s worth of work. Syncthing should not have done the destructive operation of mass deleting my files under no circumstances without asking for confirmation first. Furthermore, and actually more importantly, under no circumstances should it have ignored my read only setting for the synced trees on my computer. It provided the option, and I relied on it, and I was screwed.
The other takeaway is that, backup, and backup often. If I was already backing up every week, I wouldn’t have lost what I lost. I kept postponing setting up backups, and this is what happened in the end. Now, every weekend, I go into recovery mode, mount the filesystem that contains my files (which is a separate file system mounted under
/igk and accessed via symlinks from
$HOME), and run my backup script which makes a XZ’d tarball of
/backups. Takes about an hour and a half, produced an ~8Gb tarball.
Lastly, never trust software to prioritise your data’s integrity. I’ve experienced it with Firefox before, multiple times, in different ways. Sadly, most software out there are produced by incompetent developers. I was lucky that my files were organised into rather granular directory trees, which contained the effect of the loss to certain, luckily less actively modified parts of the file system. If I synced more broadly, I could’ve lost way more than I could manually recover.