Complete server backup without downtime. Possible?

[oKtosiTe]

Here's a somewhat open question.
For about a year or so, I've been running a Gentoo server, and very happily so, despite the fact that it's a Pentium II system.
Last week, I have received a very nice new box, and I am planning to transplant my carefully crafted setup to it relatively soon. It got me pondering, however, because when I originally started working on the old server, I never implemented a means of creating a complete backup without having to take the box down. Before I get started on the transplant, I very much wish to resolve this.

Here are the options as I see them:

1. Software RAID
   mirroring with evms/lvm2?
   Not quite sure how to migrate to these; hardware RAID is not an option here.
2. Virtualization
   xen/chroot?
   I've never done this before, but it sounds cool. I'd rather resort to a non-free product like VMware (it is non-free, right?).
3. A combination of both
   It seems like a fairly easy way out: build a mirrored host and run the existing environment as a guest.
   On the other hand, I don't believe it to to be the best option performance-wise.

Perhaps I've missed something...

The server runs the usual muck (GLAMP, exim, dovecot), but that's probably not too relevant.

Anyway, the way I'm going about it now is backing the whole thing up whenever something else requires a reboot (which isn't very often).
Any alternatives, suggestions, best practices, personal experiences, etc. are very welcome!

P.S.: I hope I'm not being a clueless idiot here...

[nevynxxx]

What exactly can you not back-up live?

The only thing I have ever heard off is database apps, MySQL etc. Every one of those I have seen has their own online backup mechanism, that will leave you with a file you can back up.

The RAID is *not* a backup solution, it reduces downtime, but anything that will delete/courrupt data (filesyetm problems etc) will do that to both parts of the raid.

The only exception is that a mirror can be split, and the live system left running on one disk, while the second disk is backed-up, and then the mirror re-created. You would need to be very very careful though!

As for the virtualisation, the only one I have any experiance with is VMware.

VMware Server, is free as in beer, and you can make "snapshots" of the guest that can be backed up, that gives you a compleate image of the system, with everything from memory, to hard disk. I have paused a VM half way through a download, restarted it a few days later, and it continued the download with no interruption!

ESX Server is non-free and costs a minimum of $1,000. It is a very very nice way to virtualise though if you have a need that justifies the expense.

[asiobob]

I have a bunch of critical servers (Linux but not Gentoo, although that doesn't matter).

Basic setup is

LVM2 on top of RAID5

That is, I have the flexibility of LVM with the redundancy of RAID 5. For backups, live back ups what I do is take a LVM snapshot of the logical volume, mount the snap shot and do a backup of the snap shot, all live in time, then destory the snap shot. The backup is a simply a tar job

[CloseYetFar]

Why not RAID 5 for physical disk redundancy and extra hard drive performance. Then once a week or more, rsync the whole RAID 5 container to an external usb harddrive. You may want to use exclude options for directory's like /tmp /dev and so on. I have never rsynced a RAID container tho, so you may want to read up on it first.

If one of the three hard drives fails, then replace it and RAID 5 will use the parity data on the other two hard drives to update the new one.

If two hard drives fail or the whole RAID container gets corrupted, then use the external hard drive to update a new RAID container. All you would have to do to restore the rsync image to the server is:

1. Get three hard drives of the same size.
2. Create a RAID 5 container
3. Partition and format the RAID container just like the external hard drive is.
4. Mount the usb external hard drive read only and rsync it to the RAID container.
5. Use grub or lilo to copy the boot loader to the MBR. I think its number 10 in the Gentoo handbook.
6. Reboot

[tgh]

(yeah I know... late to the party)

For server administration... Xen and running servers inside of DomUs is very powerful. What you can do, is put the DomU areas on top of LVM2 and use LVM2 snapshots to get a disk-level snapshot of the DomU's file system. Then you can mount the snapshot as read-only in the hypervisor Dom0 and create a tarball of it.

OTOH, most linux boxes can easily be backed up using the stage4 script that is floating around on these forums. Application data should be backed up separately from the OS anyway. The snapshot method is more of a poor-man's disk imaging trick (useful for getting a new DomU up and running quickly).

Then, the hard part is backing up your Dom0 without taking the system down. Which can be done by running the Dom0 on top of a RAID1 array where you can pull the drive on-the-fly. Pull the drive, put a new one in, let the RAID1 rebuild. Of course, this requires hot-plug capable hardware...

[oKtosiTe]

Thanks for your replies, people!
You've given me enough to think about, and I'll investigate some more before I'll actually move the server system.

[halfgaar]

Perhaps this article will help you.