Made a btrfs snapshot mess, destroyed fabric of the universe

[Schievel]

Hi,

on my system I use one fat partition for efi, mounted on /boot and one big btrfs partition for everything else, which is devided into the subvolumes @ (for /) @home (for /home) and @cache for /var/cache.

Now I use timeshift to make snapshots of that system and roll them back, which worked well in the past.
Anyway, so yesterday evening something strange happened. I wasnt even doing something system wide, just surfing the internet a bit and working on stuff in my homefolder, when I noticed my zsh-autosuggestion that are usually /usr/share went missing. I tried cd ing to /usr/share, but could not. When I did a 'ls /', it said command not found!
It was as if I, or something else, ran a rm -rf / on my system.

I rebooted, and gdm failed to start. I have had that recently, and it was because of some mismatch with versions of the gentoo-bin kernel, nvidia driver and gentoo-sources when gentoo-kernel-bin-5.14.15 was released, but gentoo-sources-5.14.15 wasnt yet. So this was a few days ago and to fix this, I rolled back my system with timeshift to a point before the update with gentoo-kernel-bin-5.14.14 and gentoo-sources-5.14.14 and I masked the packages of gentoo-sources and gentoo-kernel-bin bigger than 5.14.14 in my package.mask.
Anyway, I did some fiddling with the nvidia driver and it worked, I looked up package.mask and my masks I put in a few days ago were gone!
Apparently I booted into a snapshot that was taken right when the update to gentoo-kernel-bin-5.14.15 happened, and before I put in the package masks.

Well, maybe I am a bit tired, rebooted, made sure I am 100% that I boot a snapshot from a few hours ago (remember the update to 5.14.15 was a few days ago, so this snapshot must have 5.14.14 in them and the packages masked) but I ended up in the same snapshot from after the update to 5.14.15! I made a few other snapshots to test, and I always end up in the same one again regardless of which one I am booting. Even though I deleted it from timeshift.
It seems like something went wrong when I booted that snapshot before the update a few days ago.

It also happened again a few times, that my whole system is gone and all commands arent recognized.

So no matter what I do, I am always back to that system state from a few days ago. I do not really know where to start and what to look for.
I feel like I am Bill Murray in Grundhog Day, send help!

[alamahant]

Is your system existent in some form and functional?
If yes, rsync it to another non-btrfs partition, reinstall grub,update your fstab grub and initrd.
And maybe avoid using timeshift or even btrfs.
Good old plain ext4 and rsync is much better.
IMHO.
This is what i would do.
_________________

[Schievel]

[Schievel]

I took me a while but after all I am back to a system that is not set back after the next reboot. I don't know if this strange wiping of root comes back.

Heres what I have figured out:
In order to fix the root partition being set back to a snapshot, I mounted the actual hardware partition (so no subvolume, I mounted /dev/sda3) to /mnt.
So I can see all of my subvolumes in /mnt now: @, @home and @cache and timeshift-btrfs, this is where timeshift keeps the snapshots.
Now btrfs subvolumes behave like folders in a certain way, so I just did

[figueroa]

Spooky; just in time for the Halloween lights to go out. Who wants a spooky file system, anyway?
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

[Schievel]

I think the problem here is not the filesystem in itself, but rather timeshift and how it handles snapshots that the system is currently bootet in or possible the person in front of the computer.

When I bootet into that snapshot, no programs does shift the snapshot into @. It just changes the subvol= in grub to that snapshot and it stays there, so I kept booting to that snapshot.
Timeshift also deletes old snapshots, if there are enough new ones. So maybe it tried to delete the snapshot I had currently booted.

Anyway, I will do some research here and maybe make a bug report upstream

[Zucca]

My wild guess is that your root subvolume is a read-only snapshot or a rw snapshot that is created from some earlier snapshot on boot.

[Schievel]

Well I found out about it. Its just timeshift rolling back to a wrong snapshot.

I just made a snapshot at 15:00. Then i created some testfiles and then chose to restore to that snapshot at 15:00. I rebooted and the kernel command line in grub says subvol=/timeshift-btrfs/blabla/2021-11-03_15:16.
That snapshot from 15:16 is the one timeshift creates automatically before doing a restore. So of course the testfiles where still in there.

The spooky thing is, timeshift doesnt move that snapshot over to the @ subvolume. Instead it leaves you booted into that snapshot.
But it TAKES new snapshots from that @ subvolume. That means when I change something in the currently bootet snapshot from 15:16 and make a new snapshot in timeshift, it wont have what I changed in there.

Now if thats not an upstream bug, I dont know what is.

To sum up what happened here yesterday:
I messed up the kernel and nvidia drivers a few days ago, and booted into snapshot A.
I did not check the kernel command line in grub and let timeshift do its thing instead. Timeshift kept me booted into snapshot A.
Timeshift happilfy took its snapshots B, C and D at boot time, hourly, daily, etc. but it took all of them from the @ snapshot, not from snapshot A and the @ snapshot was from a few days ago. So snapshots B, C and D are all not from the currently booted system, but from a few days ago, because @ was kept in that state.
When I tried to boot any of them yesterday, they all where the same, regardless of what I was booting. (except the snapshot A)

now, that explains it. What I cant explain yet is why my whole system was deleted while it was running. But I got an itch this has something to do with timeshift cleaning up old snapshots and therefore deleting snapshot A.

[alamahant]

So maybe do NOT use timeshift.
Prefer rsync or tar instead to backup your WHOLE system rather than individual btrfs sub volumes.
_________________

[Zucca]