Backups... so many options!

[leyvi]

I've been using Gentoo for almost 2 years now, and I'd like to back up my system before something goes catastrophically wrong. The thing is, there's an absolutely insane number of different ways I could do this, and I'm having difficulty picking one. I just got a 512 GB USB drive, and formatted it. I'd like to use it for storing backups somehow, but I don't know whether to use rsync, tar, or something else?

[zen_desu]

I've been using Gentoo for almost 2 years now, and I'd like to back up my system before something goes catastrophically wrong. The thing is, there's an absolutely insane number of different ways I could do this, and I'm having difficulty picking one. I just got a 512 GB USB drive, and formatted it. I'd like to use it for storing backups somehow, but I don't know whether to use rsync, tar, or something else?

I like btrbk, (for btrfs), it lets me make snapshots on my main storage device, and send them to an external one with a cron job. It's great to have multiple points in time I can go back to, instead of a few backups.

[blu3bird]

My recommendation is app-backup/borgbackup.

It is a full-fledged backup software that supports a vast number of features such as deduplication and encryption while still remaining simple and easy to use.

PS: You rarely ever restore a full backup. Most of the time you just restore a few files. You definitely want something that can be used as a sort of revision system for your files.

[Zucca]

Few to consider also:

• rdiff-backup
• restic

[pietinger]

I've been using Gentoo for almost 2 years now, and I'd like to back up my system before something goes catastrophically wrong. The thing is, there's an absolutely insane number of different ways I could do this, and I'm having difficulty picking one. I just got a 512 GB USB drive, and formatted it. I'd like to use it for storing backups somehow, but I don't know whether to use rsync, tar, or something else?

Maybe read number 2 of: https://wiki.gentoo.org/wiki/User:Pieti ... _recommend

Yes, you can use it as often as you want and it transfers only changed files after ther first initial backup ... I use it with two different USB-drives (one week this drive, the next week the other drive, so I have a father and a grandfather backup). Before this solution I used "luckybackup" ... unfortunately deprecated ... but it was "only" a "front-end" to rsync. Dont forget you have with "rsync" the great parameter "--dry-run, -n -> perform a trial run with no changes made" ...

[sublogic]

Like @blu3bird I have been happy with app-backup/borgbackup . You get to keep multiple archives, each a snapshot in time. I've never had to do a full restore, but I did pull back individual files; that is easily to do by mounting the appropriate archive (or all of them!) as a fuse filesystem.

One drawback, if you use filesystems like xfs: they don't seem to be aware of reflinks. I think a reflink would be restored as an independent copy but I have no way to test, I'm on ext4.

512 GB is not very much but it might work, depending on how much data you have. I've been backing up about 50 GB weekly since Feb. 2023, 138 cycles, and my deduplicated archives add up to 333 GB. I use a 2 TB USB HDD so I haven't bothered to prune archives yet.

[Banana]

Also try to think about your problem without thinking about a solution at first.

Like:
- What do I need to backup?
- What is the goal if the data which is in the backup?
- Wow to do a recovery?
- How often do I need a backup?
- How fast do I need to be able to do a restore?

Solituons depend on the problem, not the other way around.

[C5ace]

I use since many years "app-backup/rsnapshot" (wiki.gentoo.org/wiki/Rsnapshot and wiki.gentoo.org/wiki/Advanced_backup_using_rsnaphot) to backup my server to a USB drive.
My setup backups the mail every hour, the full server once a week using these batch files and corresponding rsnapshot configuration and rsync exclude files.

/usr/local/bin/snapshot-mail

Code: Select all

#!/bin/sh

date | tee /var/log/snapshot-mail.log
# echo "-------- /bin/mount /home/backup -------" | tee -a /var/log/snapshot-mail.log
/bin/mount -L USB_HOME /home/backup | tee -a /var/log/snapshot-mail.log
ls -l /home/backup/Server-1 | tee -a /var/log/snapshot-mail.log
echo "-------- rsnapshot -c /etc/rsnapshot-mail.conf $1 -------" | tee -a /var/log/snapshot-mail.log
rsnapshot -c /etc/rsnapshot-mail.conf $1 | tee -a /var/log/snapshot-mail.log
ls -l /home/backup/Server-1 | tee -a /var/log/snapshot-mail.log
# echo "-------- /bin/umount /home/backup -------" | tee -a /var/log/snapshot-mail.log
/bin/umount /home/backup | tee -a /var/log/snapshot-mail.log
date | tee -a /var/log/snapshot-mail.log

/usr/local/bin/snapshot-mail

Code: Select all

#!/bin/sh

date | tee /var/log/snapshot-server.log
echo "-------- /bin/mount /home/backup -------" | tee -a /var/log/snapshot-server.log
/bin/mount -L USB_HOME /home/backup | tee -a /var/log/snapshot-server.log
ls -l /home/backup/Server-1 | tee -a /var/log/snapshot-server.log
echo "-------- rsnapshot -c /etc/rsnapshot-server.conf $1 -------" | tee -a /var/log/snapshot-server.log
rsnapshot -c /etc/rsnapshot-server.conf $1 | tee -a /var/log/snapshot-server.log
ls -l /home/backup/Server-1 | tee -a /var/log/snapshot-server.log
echo "-------- /bin/umount /home/backup -------" | tee -a /var/log/snapshot-server.log
/bin/umount /home/backup | tee -a /var/log/snapshot-server.log
date | tee -a /var/log/snapshot-server.log

/var/spool/cron/contabs/root

Code: Select all

#Mins  Hours  Days   Months  Day of the week

01     *      *      *       *       /opt/bin/snapshot-mail hourly
16     3      *      *       *       /opt/bin/snapshot-mail daily
31     4      *      *       7       /opt/bin/snapshot-mail weekly
46     6      1      *       *       /opt/bin/snapshot-mail monthly

31     7      *      *       7       /opt/bin/snapshot-server weekly
46     1      1      *       *       /opt/bin/snapshot-server monthly

Restore is done using rsync or copying individual files from the relevant backup.

[NeddySeagoon]

leyvi,

Before choosing a backup solution you need to ask yourself what will you use the backups for, if you ever need them.

Worst case, you need to be back working on new hardware as soon as possible.
You need a complete, validated backup.

You don't mind rebuilding the install?
Keep /home, /etc/portage, and your world file.

Something less?
You decide.

You only have a backup when you have two validated copies.
With only one copy, when your primary fails, you no longer have a backup.
Validated matters. You need to know that you can read the copies if you ever need them

[pa4wdh]

It all depends on what you need.

To give you some inspiration, this is what i'm doing:
For me backups cover two use-cases:
1) Me doing stupid things, like deleting the wrong file
2) Hardware failures that something destroy my data

Daily off-site backup for important files
This is for files that change on a regular basis or files that i think are important. For example my mailbox, but also /etc and some specific locations in /var of all my systems. The goal is to have important information available that allows me to either restore the important data or rebuild it, trade-off is that i try to avoid high volume data. This runs from my server which encrypts the data and sends the encrypted result to a remote server. Just to be sure i have a printed version of the keys offline, and yes, i did test a manual restore . The backup is made with tar, gpg for encryption and scp for the transfer. On the remote site i keep 2 weeks of daily backups, keep the backups for mondays for 3 months, and always keep the first monday per quarter. Currently this is about 4GB a day.

Quarterly full-backups
Every 3 months i update all my systems and after that i make a full system backup of everything. This is done on a (luks protected) external harddrive and a script which basically tar's everything. When doing backups i connect the drive to a single system and run the backup script, and then the next, etc .... On the drive i keep as much backups as capacity allows me, which differs per system, but i try to keep at least 4.
When i'm not running backups it's disconnected.

So far i've had a few times where i needed the full backups to restore single files, i've never needed to restore from the off-site backups. But whatever you do, do test restores. If you're not sure if you can restore, you technically don't have a backup.

One more thing:

My recommendation is app-backup/borgbackup.

Feature wire this looks good, but it's manual states:

EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER CHANGES (like when going from 0.x.y to 1.0.0 or from 1.x.y to 2.0.0).

This is a huge red flag for me. The last thing you want is "I can't restore my important data because i just upgraded borgbackup".

[sublogic]

My recommendation is app-backup/borgbackup.

Feature wire this looks good, but it's manual states:

EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER CHANGES (like when going from 0.x.y to 1.0.0 or from 1.x.y to 2.0.0).

This is a huge red flag for me. The last thing you want is "I can't restore my important data because i just upgraded borgbackup".

I think they may break *forward* compatibility. Your new borg will read your old backups (perhaps with some precaution, https://borgbackup.readthedocs.io/en/st ... rade-notes , but your rescue media with old borg may not restore subsequent backups. So yes, some caution required for full restores after a hardware failure.

[Spanik]

A bit mixed. First line of defence is a raid-5 to gain some time to backup the last files. Next a NAS for important files and photos that is updated when a new load comes from the camera or is scanned. Next 2 sets of 3 usb disks, one set for the files and photos, another for the music. In each set they have the same label and are mounted with the label so the command for rsync stays the same. (rsync -avh --no-group --no-owner --delete)Those are rotated and the last one is always taken off-site, the one that comes back from off-site goes to the bottom of the pile. For mail I have set the application so to leave the last 3 months on the server of the provider which is another "backup".

I don't backup the system. If that happens then a clean install isn't that bad. After more then 5 years it tens to accumulate quite a bit that isn't used anymore or just making it more complicated to keep up to date.

[Zucca]

I tend to only use backup solutions which have the possibility to restore backups just by using common filesytem cli utils. That means I should be able just pretty much cp -a /backup /destination.
This is why I like filesystem snapshots and programs like rdiff-backup (only the latest backup is "human-readable").
However restic has been on my radar because of its features. I'm not sure if it offers restoration via those basic cli utils.

[pietinger]

I tend to only use backup solutions which have the possibility to restore backups just by using common filesytem cli utils. That means I should be able just pretty much cp -a /backup /destination. [...]

+ 1 !!!

[leyvi]

Wow, lots of good options!
Not sure if this helps, or if it just made the problem worse though (I've just heard of ~5 new options, all of them promising).

[NeddySeagoon]

leyvi,

Start with a list of requirements for the use of your backups.
Try a few and test the restore in a Virtual Machine.

That is, audition a few through the backup and restore cycle.
Don't forget the validation step. You don't want to discover that you don't really have any backups when you need them most.

[wjb]

@Zucca

One of the nice things about borg is that you can mount a backup repository - so its very easy to browse, compare things and extract exactly what you want using all your normal tools.

Code: Select all

# borg mount borgs@borgserver:backup /mnt/backup
# ls /mnt/backup
panther-2025-05-02T10:59:21  panther-2025-07-01T19:00:00
panther-2025-05-31T19:00:00  panther-2025-07-02T19:00:00
panther-2025-06-08T19:00:00  panther-2025-07-03T19:00:00
panther-2025-06-15T19:00:00  panther-2025-07-04T19:00:00
panther-2025-06-22T19:00:00  panther-2025-07-05T19:00:00
panther-2025-06-29T19:00:00  panther-2025-07-06T19:00:00
panther-2025-06-30T19:00:00
# ls '/mnt/backup/panther-2025-07-06T19:00:00'
bin   etc    home  lib64  opt              root  setup-chroot  var
boot  files  lib   mnt    readme-uefi.txt  sbin  usr