Backup strategy - parity or other redundancy

[number_nine]

I'm thinking about building a machine to act as a backup server. I have a bunch of static files I want to backup.

Of course, making a second copy is good... but what if the media starts to fail? E.g., a sector goes bad, I get a case of "bit rot", the media is scratched, etc?

So my thought was, short of making multiple redundant copies (which is too expensive), why not go "half way" and use something like parchive/par/par2 to add parity information to the data?

Just wondering if anyone else does anything like this, or has any other suggestions.

My goal is to protect against partial failures in the backup media. Say I have a collection of 10 files. And part of the disk (or whatever media) goes bad. I'd like to store some extra information with those 10 files to be able to recover all 10 files, even if the degraded media effectively deletes a couple files.

Thoughts?

Thanks!

[poly_poly-man]

two big hard drives in a RAID 1 array.

And fortnightly tape backups.

poly-p man

[halfgaar]

I do it like this:

- RAID1 array in my machine.
- daily backups of frequently changing dirs (/home, /etc, /var) with rsync to external drive.
- randomly scheduled backups with dar+parchive to external drive.

(external drive is encrypted with cryptsetup+LUKS. This not a backup feature, but merely protection against the user forgetting the backup disk in the train or something...)

Using parchive is actually a better idea than using multiple copies. Because if you have 100 backup disks, and 2 bytes are damaged on all of them, none of them are useful. Whereas when you use parity, let's say 5% on 1000MB, you can loose any 50 MB of data, scattered throughout either the backup files or the parity files. It's very robust. One wonders why they don't send parity information in communications in Star-Trek, to avoid the "part of the message has been lost captain"

I have automated the entire procedure. I simply have to run a script, and everything is taken care of.

I think you may be interested in my backup article. It considers a whole bunch of issues you need to watch out for.

[Carnildo]

Hard drives use error-correcting codes, so it's reasonably safe to assume that if you can access the data, the data's correct. In order for the data to be corrupted undetectably, you would need at least three bit flips in a single error-correction unit -- an event so unlikely that, across all the hard drives in current use, it has probably never happened.

[halfgaar]

But that doesn't guarantee that you can retrieve all of your data of any broken disk. That's the whole point of adding parity information to the dar archive. When copying, and certain area's of the disk are unreadable, it simply cannot retrieve that data. If the amount of data lost falls with the percentage of parity data you created, you can re-create it.

[poly_poly-man]

Hmm... how about we just agree that no backup strategy is perfect

I'm still liking the idea of tape backups on top, tho.

poly-p man

[halfgaar]

Hmm... how about we just agree that no backup strategy is perfect

You get no argument from me there

I'm still liking the idea of tape backups on top, tho.
poly-p man

I consider tapes somewhat clumsy. Hard drives are cheap and big nowadays. I just have to turn on the disk, run a script, wait a couple of minutes, and the backup is up-to-date. The full-system backup I do with dar+parchive takes longer; I run it when I go away for the better part of a day, and it'll be done when I'm home.

[twstd3bc]

I'm thinking about building a machine to act as a backup server. I have a bunch of static files I want to backup.

Of course, making a second copy is good... but what if the media starts to fail? E.g., a sector goes bad, I get a case of "bit rot", the media is scratched, etc?

So my thought was, short of making multiple redundant copies (which is too expensive), why not go "half way" and use something like parchive/par/par2 to add parity information to the data?

Just wondering if anyone else does anything like this, or has any other suggestions.

My goal is to protect against partial failures in the backup media. Say I have a collection of 10 files. And part of the disk (or whatever media) goes bad. I'd like to store some extra information with those 10 files to be able to recover all 10 files, even if the degraded media effectively deletes a couple files.

Thoughts?

Thanks!

I use Par2. More specifically, I tar up certain directories into, say, archive.tar, gpg encrypt it to a file archive.tar.gpg, split it up with the split command, make some pars from this, and mail them to one of my gmail accounts. For weekly backups I use a bunch of mini (and non-mini) DVD-RWs and keep them on a shelf at home. I also have a linode to back things up as well.

[halfgaar]

I use Par2. More specifically, I tar up certain directories into, say, archive.tar, gpg encrypt it to a file archive.tar.gpg, split it up with the split command, make some pars from this, and mail them to one of my gmail accounts. For weekly backups I use a bunch of mini (and non-mini) DVD-RWs and keep them on a shelf at home. I also have a linode to back things up as well.

Why don't you use dar? It has the ability to split automatically.

[twstd3bc]

I use Par2. More specifically, I tar up certain directories into, say, archive.tar, gpg encrypt it to a file archive.tar.gpg, split it up with the split command, make some pars from this, and mail them to one of my gmail accounts. For weekly backups I use a bunch of mini (and non-mini) DVD-RWs and keep them on a shelf at home. I also have a linode to back things up as well.

Why don't you use dar? It has the ability to split automatically.

DAR looks nice, especially the split option and par support. I'll have to look into how it implements encryption, though I like that it uses blowfish, which is fast. Thanks!