online backup f2fs?

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

The title basically says it, anyone has a good way to get a somewhat consistent backup a mounted f2fs?
There is no lvm underneath, so can't snapshot it this way, and it doesn't seem like it supports snapshotting at fs level either.

I found one thing I might be able to abuse for a block level backup, but honestly I hope someone can suggest a better option.
Also, I'd prefer file-level backup, so I can easily drop caches worth like 50GB of transient data I could just re-download if I really needed to.
_________________
Make Computing Fun Again

figueroa · Posted: Thu Apr 11, 2024 3:11 am Post subject:

I suggest making a compressed tar archive. Look for my contribution for making a stage4 backup in the following thread. The thread is worth looking looking through for it's on sake since the subject was backing up Gentoo.
https://forums.gentoo.org/viewtopic-t-1162924-highlight-stage4.html
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

Pushing it _somewhere_ in whatever format is easy and I can handle it just fine; I still need to get the data off of the original device though, in a consistent state, while the device is in use. That is the tricky part, and I probably need something specific to F2FS that I'm currently not aware of.
Like, can I somehow double mount it to get a frozen view of the data? As stated earlier, unfortunately I don't have LVM underneath.
_________________
Make Computing Fun Again

tholin · Apprentice Joined: 04 Oct 2008 Posts: 203

I am aware of two possible solutions but neither is ideal.

1. You could use the blksnap module. This tool enables the creation of writable snapshots of any Linux block devices even without LVM. The code is not part of the mainline kernel. The author of blksnap is working on getting it merged into the upstream kernel but that is still a work in progress. I've never used blksnap myself and I am only familiar with it from an older lwn.net article.

https://github.com/veeam/blksnap
https://lore.kernel.org/all/20240209160204.1471421-1-sergei.shtepa@linux.dev/

2. Another option is to use /sbin/fsfreeze to freeze the file system while the backup is running. The fsfreeze manpage states that f2fs is supported. Any program attempting to write to a frozen file system will halt until the file system is unfrozen. Some programs do not respond well when write() calls stall for several minutes during a backup. Fsfreeze is primarily designed to be used by LVM or blksnap to force the on-disk layout into a consistent state for the brief period of time it takes to create a snapshot. Even so I use fsfreeze in my backup scripts with an ext4 rootfs and have never encountered any issues.

If you decide to use fsfreeze make sure that you enable sysrq so you can execute "sysrq + j" to unfreeze the file system if your entire system becomes unresponsive. You may also want to include "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" in your backup script because the kernel will begin spamming warnings in the system log about hung tasks otherwise.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

Yes, not ideal, but fsfreeze might actually be acceptable. I didn't want to remount=ro, because it would make writes fail, but delaying them instead doesn't seem that bad. Even if some applications become unresponsive for a few minutes, it's still OK-ish as long as they resume afterwards.
I like this much more than remounting with checkpoints disabled (should keep inconsistencies caused by writes during backup to free blocks)
Definitely worth testing. Thanks
_________________
Make Computing Fun Again

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

So I tried to use fsfreeze and run borg backup... And it didn't start. Strace showed its trying to create a file in /tmp, so I converted it to tmpfs. Launched borg again, and it didn't start. This time trying to create a file in ~/.cache/something...
Seriously... When I read docs, it looked like locks would be created inside repositories.

Damn. Looking for more ideas. (I suppose it is possible to do a workaround for those locks, but since I already know about 2 locks at the beginning of a single job, I'm reluctant to try and discover 3rd, 4th and 5th too)
_________________
Make Computing Fun Again

figueroa · Posted: Thu Apr 11, 2024 10:51 pm Post subject:

What's wrong with the compressed tar archive (stage4)?
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

It doesn't come with deduplication across multiple backups of the same machine (and multiple machines), so I'd have to store much more data than I really need. Also, I'd have to manage pruning myself. (Which I _can_ do and have in the past, but I hoped to move on to something better than rsync rather than worse)
So nothing wrong with it by itself, but there are better tools for the job.
_________________
Make Computing Fun Again

tholin · Apprentice Joined: 04 Oct 2008 Posts: 203

figueroa · Posted: Sat Apr 13, 2024 4:13 am Post subject:

wjb · Posted: Sat Apr 13, 2024 3:28 pm Post subject:

The type of backup you do depends on what you expect to do with it. In my case, it's file/directory recovery or comparison, so borg's ability to mount a backup set is ideal. If the PC/disk blows up, I expect to install new Gentoo and recover my files/config to the new installation eventually while being able access them in the interim using any (linux) pc capable of running borg. I have done this once for real (three weeks before retiring, which was very inconvenient but not a disaster) so I know it works.

There is currently about 4TB of backup data for 4 PCs taking up <300GB on a 2T drive.

There is one database I care about, and I just have my backup script pause the server while the backup is in progress - takes about 5min which is fine for my usage. I generally try to avoid doing installs while the backup is running, but if it happens it happens - it'll sort itself out the following day, and although its nice to have the built/installed files safe it's not strictly necessary.

I am careful to keep up-to-date & step-by-step recovery instructions on the backup media, because googling stuff on a phone while in a tizz never seems to end well.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

Guys, you're really focusing on the wrong end of the stick now. The point was specifically to get the data off of f2fs, while its in use, without corrupting it.
So far fsfreeze is the winner, and I'll most likely use. I think it fits the bill well enough to be the missing piece from my first post.

Backup target wasn't really a part of that question. Worst case, I'll just dust my old rsync script, which is not perfect, but worked without issues for a number of years.
I wanted to upgrade to borg to make things easier for myself, but I'm second-guessing that now. Dropping lockfiles into the backup target is perfectly fine; using the FS being backed up for processing said backup is not. I mentioned it because it caught me off-guard and I consider this part a broken design.
With additional information about borg's variables and so on, I'll weight my options and priorities and pick a suitable solution. This is still not really a part of getting the data off off that source device though :roll:

Anyway... Thanks for tips, everyone!
_________________
Make Computing Fun Again

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

Just to wrap it up in case anyone was interested in the final result:
I ended up going with rdiff-backup instead of borg, and fsfreeze remains an important part of the whole setup.

Rdiff-backup is not the fastest thing in the world (io-wait on spinning plates hits it hard), can't deduplicate data across multiple nodes, and overall is only a minor upgrade compared to rsync, but it is available on both Gentoo and Debian, seems to have decent backwards compatibility, doesn't clash with fsfreeze, and doesn't require hacks to pull data from remote clients (borg only works in push mode and I don't want to leave ssh keys to my home network on a VPS).
So, while it's not perfect, it's probably the best ready-to-use option for me right now. I might improve its compression ratio by converting underlying storage to btrfs one day, but it won't be a priority for some time.
_________________
Make Computing Fun Again

wjb · Posted: Mon Apr 22, 2024 7:09 pm Post subject:

(the ssh key can be restricted so a client is only able to access its own repository on the borg server, though sadly that does not provide any way to prevent a hijacked client from wiping its archives)

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140

Well, to be fair borg can be launched in append-only mode which according to its docs would prevent the client from removing _anything_ from its repo, but forcing command on a key is waaay more hackish than just running rdiff-backup user@remote::/path /local/mirror without having any key to local machine on the remote node (no sharing via ssh_agent either, so no window of opportunity for re-using the exposed key for other purposes), and adds more places where I can make mistakes setting things up.
_________________
Make Computing Fun Again

Zucca · Posted: Mon Apr 22, 2024 9:19 pm Post subject:

wjb · Posted: Mon Apr 22, 2024 11:15 pm Post subject:

Ok I'm interested, two things - why do you consider setting a command on a key 'hackish' (I'd call it annoyingly whitespace-sensitive), and it's not exactly expensive to set up a separate ssh key just for the purpose of backing up (other than maybe having to add it to /etc/ssh/ssh_config).

szatox · Advocate Joined: 27 Aug 2013 Posts: 3140