Advice on backup software

[hika]

I would like to ask some advise on backup software.
I recently replaced an old samba etc. server with a new one. The old one used rdiff-backup to daily backup the data internally and about weekly to an external disk. It used rdiffweb, so the users could retrieve data.
While rdiff-backup is well suited, I found out that rdiffweb is no longer actively developed and relies on Python 2.5. An alternative, rdiff-backup-web is even older with pre php 5 code.

Can anybody advise on alternatives. My wishes are:
- automated incremental backup, which doesn't need a graphical frontend. There is no X on the server
- a webbased frontend for retrieval of backed-up files
- Databases (ldap, mysql) are internaly backed up separately, which backups will be added to the weekly external backup (or routed to my own server over vpn)
- maybe at a later date I want to expand to remote backups from Windows 7 workstations

Hika

[gcasillo]

Add me to the list of folks looking for a backup solution particularly the automated incremental backup part. Apparently rdiff-backup, which I have used for the last few years, has been masked as it is not being actively maintained upstream.

[hika]

Looking around, I so far have come up with backuppc as the best solution. I haven't yet really looked into it, but will the upcoming weeks.

Hika

[dahoste]

Currently also on the hunt for a replacement to rdiff-backup, having just been informed by emerge that it's masked due to abandonment and that I'm apparently not 'sane' for continuing to use it.

I swear... sometimes the portage maintainers are just snotty jerks for no good reason.

Anyway, I use rdiff-backup six ways to Sunday, on tons of systems, so am very anxious to get something else suitable chosen and get on with life. If/when I settle on something, I'll check back in here and make a note of it.

[beandog]

I think backup-manager does incremental tarball updates.
_________________
If it ain't broke, tweak it. dvds | blurays | blog | wiki

[1clue]

1. Add a large removable drive, preferably with esata interface so it's fast.
   
2. Develop a scheme for backup location, such as /mnt/backups/2014-04-10/
   
3. Make a script that copies all your precious files from your "hot" drive(s) to the backup, such as cp -prd /home /mnt/backups/2014-04-10/home
   
4. Every so often, attach the backup disk, create a new folder and run your script.
   
5. Unmount and detach the backup drive
   
6. Put the backup drive in some other building.
   
7. When the drive is full, delete the oldest copy, or prune such that you keep one a month rather than one a week, or one a quarter or whatever. Or retire one backup set and start with new disks.
   
8. Cycle through more than one physical drive, round-robin style.
   

I've been doing backups for decades. At one point I used magnetic tape, until such time as I actually had to use the tape. I found out then that, as was more common than you might imagine, the backups were not restorable.

I've gone through almost every conceivable solution, from network rsync style stuff to tarballs to CDROM, RAID etc.

rsync has several disadvantages:

1. You only get a copy of what you have right now; it has no history to undo stupid changes that happened two weeks ago but you just found now.
   
2. If you're going over a network, you have a significant speed problem for large backups, and you still have to move the backup device out of the building.
   
3. If you're going across the Internet, you have a huge speed problem.
   

Tarballs, if the tarball is corrupted you lost the entire backup.

Other proprietary file formats like tapes, you lose your device you lost your backups.

CDROM, well I've found that the ones you burn yourself aren't always that reliable, and anymore you have to spend a lot of money to get a blank that will hold enough data to be worthwhile.

RAID is not a backup solution. Period. It's a way to stay running when a drive fails, or a way to speed up your throughput or expand the size of the drive. Or some combination of the above. RAID protects you from drive failure but not fire, robbery, electrical problems or pretty much every other reason you might want a backup.

So my most reliable solution is the one at the top of the post. You use the most reliable technology you have, a tech that is profoundly available from lots of manufacturers. You have more than one backup device, and it's as fast (or almost as fast) as your normal drives, so you can back up everything you want. It's removable so you can take it with you when you go home.

You have a native hard disk, you store in a native format on a native filesystem, one bad spot on your drive doesn't ruin the entire backup. If you have dated backups of just your data, you can reinstall a system with a new drive or a completely different computer, pull your data back and be done with it. If you need something from a month or a year ago, you have it, and you can do a find without decompression or worrying if your backup device is good for it. More importantly, you can look for it at native hard disk speeds instead of some third-rate storage device speed.
[/b]

[1clue]

Sorry for the double-post.

Whatever solution you use, you need to test a restore. If you can't restore it, it's not a backup. It's a waste of time.

Another thing, you need to have a means to externally recognize your backup media. Something more than A B or C.

Good luck and have fun.

Edit: Rather than triple-post I figured I'd edit. The system you put in place needs to be convenient but not completely automatic. You need to be aware of backups, or you forget about them and how your needs change as time goes on. You need it to be convenient enough to be mostly automatic, but the time you REALLY need convenience is when you need to restore data.

When you back up, your system is probably running normally. When you restore, you have a problem. When you have a problem it means you were trying to access data that is either gone or corrupted. When your data is gone or corrupted and you need it, you need an easy and fast way to get the backup and get your system up with the least headache possible, and the least chance of corruption possible.

[gcasillo]

I've tinkered with rsnapshot this week, and it looks suitable for me. This post in another thread helped explain the configuration file.

[John R. Graham]

[lexflex]

[1clue]

[1clue]

[John R. Graham]

[1clue]

OK John,

So how much does it cost for your office to be down? For me, the last time I checked, there were 80 people, about 14 of which were on salary So guess at $3000/hour for the non-salary people, no idea for the ones with salaries. If the server that goes down is something needed by everyone in the office (which there are several) then every minute of downtime costs the pay you need to keep paying all your employees, PLUS the money they're not making while you monkey with tapes (A LOT more than the employee cost), PLUS the cost of the replacement hardware.

I can reproduce a point at any time for databases, but the rest really doesn't matter much in my case.

It's up to you really, to me the decision to spend a few bucks to get a hard disk with sata2 or sata3 speeds depending on the server, and random access is pretty easy.

[Splink]

cpio ?
_________________
It said "Requires Windows XP or better"
So I installed gentoo.

[hika]

For some reason I've been thrown out of the maillist. MAybe I misclicked the wrong line in the mail.

Anyhow I have been looking into backuppc and for those who joined my question I will here give my experiences. I hope others will join with their further experiences.

First the pros and cons:
Pro:
- It has a very nice and user-friendly web-interface. As admin you can manage everything, the users can only see the backups you give them rights to and themselves restore what they have lost. They can only change the settings you allow them to.
- It is very efficient at storing the data, making consecutive full backups in a way incremental. Identical files are only stored ones and then hard-linked. So running an immediate second full backup will take no extra space, save for administrative overhead.

Con:
- It is designed to backup workstations to a central server. You can backup from localhost but creating securityrisks in the process. I do advice against using this product to backup the system areas from the server.
- It is for security designed to run as a separate user, not as root. It will have to login to do a backup and you have to provide the credentials. For backup of the local system I think a separate process running as root under chron will be more secure.
- Because of this design apache also needs to run as this user. Starting with apache 2.3 there is a directive 'VHostUser/VHostGroup' to set the user for a vhost to run as. In portage, 2.3 is not present and when I tried 2.4.9-r2 it gave incompatibility issues with perl. So I didn't investigate that. Instead I run this apache server as a separate process on port 8080/8043. I created a modular configuration for this I will add further down. You can also use the apache2-backup script that comes with it. If you use other apache sites, you will have to change the port of either one.
- The application basic design is a bit weird, placing pif and socket files not in /var/run but in the log directory. I adapted the code as shown below for this.

There is an archive function for storing a backup on other media. I haven't yet looked into that.

Minor bug without consequence except filling up your log:

[platojones]

I'm in the same boat as you folks. Thank you hika for that review of backuppc...based on your experiences, just setting it up, it already looks significantly more complex and fiddly for what I want or need. For now I've copied the rdiff-backup ebuilds to my local overlay, but I know that isn't a permanent solution. rsnapshot may be the way to go...but it's still not rdiff-backup. Hell, maybe I should fork it and start maintaining it. It's hard to walk away from a piece of software that does exactly what you need it to do and does it so well.

[hika]

[Navar]

@jrg, wow, DLTs have really went down in cost in 15 years. I'm kind of surprised they still exist.

[1clue]

[Navar]

[1clue]

Wow again.

You totally missed what I was wowing about above. I suppose that's my bad, considering how terse my post was.

I'm not an opponent of insurance, it's a good idea if your business is above a certain size, or rather if your risk is above a certain point.

What surprises me is that you would rely on that insurance so you can tolerate a slow, possibly kludgy backup system when something far better exists for not much more money, and needs zero special software.

Insurance aside (not my part of the business) the cost of downtime is always significant, insurance or not. Even with the best possible insurance, you have a loss: They won't pay you the full value of your time and anticipated income, and they also generally don't pay full replacement cost of your equipment.

Most importantly, they don't pay you for lost customers.

From my perspective, the ability to get data back rapidly outweighs anything a tape can give you. Random-access backups with no compression give you the ability to reduce your downtime hugely. Insurance or not, that's a big money saver when you need it.

Insurance might pay 60% of your total loss, but if your backup system reduces that gross loss from $10,000 to $2,000 then you just paid for a lot of backup storage.

If you read far enough back in this thread, I also used tapes of all sorts. I had a need to restore from a tape -- which failed -- and then on testing realized my tapes were essentially worthless.

I strongly support disk-based backups, no compression and a simple directory structure for organization of data. It's faster to write, faster to restore and you can (and should) frequently search the data both in order to ensure that your data is good, and to make sure you know what you're doing when crunch time comes.

[Navar]

[1clue]

[hika]

Navar and Iclue, can you two move your fundamental discussion else! My question was and is about good alternatives for rdiff-backup!
Not about fundamental discussions, that in essence come to nothing. Especially within aforementioned context!

Hika