One server to do it all?

[morphal]

Trying to come up with a solution for what I want has been difficult. I've been admining my Gentoo boxes for about five years but have never really moved on into anything beyond a simple web server. I've hosted a phpBB forum, helped with some basic scripts for a work project, swapped some files around and, I think, been very good at keeping it all secure. Long ago I figured out a nice little fix for all the fancy buttons on my Logitech mouse and I managed to keep abreast of things when the first (in my experience) big change to Gentoo's config file layout for Apache came along and helped some users through that. I've received (at least) my share of help from the forums along the way but I do try to Google and RTFM as often as I can.

Rambled enough? Great. I'd like to do something more with my machines.

I'd like to get something useful running, for once. Specifically, I want a server that hosts multiple domains/sites with message boards, mail (webmail/IMAP/POP/SMTP and all securely!), Jabber, single sign-in, preferably in a chrooted environment. I'd also like to support mail aliasing, but that seems trivial after the other requests. Blocking spammers is a huge, huge problem on my current forum and I'd like to be as secure against that as can be realistically expected. I know this is a fairly tall order all by itself, but I'm probably looking at no more than 50 registered users with 10 concurrent users being positively hectic. Yes, I know it's overkill. Yes, it gets worse.

If I were to get really specific, I'd like it to run off PGSql and have their login/mail account be independent of the name that actually displays when they post in the forums. However, finding native support for PGSql has been uncommon with a lot of the software I run across and it's not a crucial enough issue to insist upon. I'm also willing to sacrifice the single sign-on feature, if I have to.

Additionally, I'd like to set up some hocus pocus to backup the entire rig on a regular basis (also over a secure connection) to some ridiculously complex RAID storage. That can be part two of this project though. =)

The usual answer in the open source world is to tell a person like me to get coding. Realistically, that's not an option. I have an 8-5 (or 6, or 7, or ... ) job and some degree of a life away from my computer. To learn to code well enough to do something like this would take years. To learn how to do it right, do it securely, and make it stable would require a team of people and degree of obsession I just don't have the time for. What tools are available to me that are able to do what I'm after?

Eventually, I'd like to take it further: redundant boxes with configs that replicate to both/all of them, redundant routers, multiple Internet connections, failover DNS, and some kind of Linux-equivalent to a Windows domain controller (Is it sad I don't know a better term for that? ). However, if a RAID backup is secondary, these concerns are definitely tertiary. It all starts with one server to make the rest of the work desirable.

[Hu]

You mention spammers, so I assume that the forum will be accessible to the Internet at large. Are you planning on exposing all of these services to the Internet, or are some of them restricted to a smaller set of users? If you expect that the service can be reached by malicious users, you should consider whether a hardened server would be appropriate.

For spammers and the forum: at that volume, I think it would be more practical to disallow user registration and do the registration out of band, via phone call or e-mail to an administrator. The administrator creates the account, then the user changes the password. If that does not work for you, some forums can be configured to require administrator approval before an account has posting rights. This can serve as a barrier to spambots, but an attacker could still flood you with account creation requests. If you are concerned about that, choose a forum that has native support for a CAPTCHA.

Regular backups are fairly reasonable with a cronjob that uses dump or tar to archive the system to remote storage.

[morphal]

Manual activation is how I handle my current forum. It's a little awkward and has led to some confusion, but that's to be expected when you make a change to a system that most people are familiar with. CAPTCHA might slow the registrations but I have still deleted hundreds, if not thousands, of spam attempts. Keeping an eye on registrations, deleting the spam requests ad nauseum, and verifying with the right users has been effective but, again, a little awkward.

Cron + dump/tar is probably fine, although, I admit I've never used any combination of them. I'd like to be able to restore a system from scratch in a pretty decent hurry. I know there are all sorts of files in use, links, virtual filesystems, and the like. Are any of these likely to cause a problem with the backup and restore process, especially if I'm restoring to bare metal?

Backups are only a concern after I have something to back up. I still don't know where to start with building a system that covers these needs comprehensively. I know there are systems out there that accomplish all of this. Have I just put too many limitations on myself by not becoming a basement-dwelling coder? I wish I could devote that kind of obsession to it because I've always enjoyed my coding classes and the feeling of accomplishment when dealing with something technical is great.

Is there really no shortcut for pseudo-noobs, like me?

[Hu]

Restoring from a tarball is typically limited mainly by the speed at which the source medium can be read and the destination medium can be written. If you expect to restore onto bare metal, you will also need to preserve the partition table, since neither dump nor tar save those. A restoration would then consist of:

Boot a LiveCD or other working system with the restoration drive in the same tower.
Lay down the partition table.
Lay down the MBR.
Make the filesystems on the partitions.
Mount the filesystems in the same layout as they have in the live system. For example, your /usr directory on the live system should be mounted at /mnt/gentoo/usr in the restore system, assuming that / on the live system is mounted at /mnt/gentoo in the restore system.
Restore the files from backup.
Reboot into restored system.

One possible sequence of commands to do this (untested):
# Lay down the partitions, which were saved by sfdisk -d /dev/hda during the backup process
sfdisk /dev/hda < /root/hda.partitions
# Restore the MBR
dd if=/root/system-mbr of=/dev/hda
# Make filesystems
mkfs.ext2 /dev/hda1
mkfs.ext3 /dev/hda2
mkswap /dev/hda3
mkfs.ext3 /dev/hda5
mkfs.ext3 /dev/hda6
mkfs.ext3 /dev/hda7
mkfs.ext3 /dev/hda8
# Mount the filesystems
mount /dev/hda2 /mnt/gentoo
# Make mountpoints for the other filesystems. Throw in directory structure for the rest of / while we are here.
mkdir /mnt/gentoo/{bin,boot,dev,etc,home,lib,mnt,opt,proc,root,sbin,sys,tmp,usr,var}
mount /dev/hda1 /mnt/gentoo/boot
mount /dev/hda5 /mnt/gentoo/usr
mount /dev/hda6 /mnt/gentoo/var
mount /dev/hda7 /mnt/gentoo/tmp
mount /dev/hda8 /mnt/gentoo/home
cd /mnt/gentoo
# Restore the files
tar -x -j -f /root/system-backup.tar.bz2
reboot

This sequence assumes your drive manifests as hda. Depending on your hardware, it may have another name. It also assumes that the backup files are located in /root/ on the restoration system. I also make certain assumptions about your partition layout, based on patterns I see used. The exact commands will vary depending on which directories receive private partitions versus which are left as part of a parent partition. If you have the hardware, I suggest trying a restore at least once early on, so you can get a feel for the time requirements and identify any holes in your preparations.

As far as potential problems: dump and tar should both be able to handle special files. You will need to avoid archiving virtual filesystems, such as /proc, /sys, and /dev. Both can do a decent job with a live system, but you may not get a completely consistent archive if the files are changing as they are archived. For a perfect snapshot, it is best to archive the filesystem while it is not changing.

I think you will not be restricted by your lack of programming ability, but you may encounter some problems if you are not comfortable administering a Linux system. I think most of your goals can be satisfied or almost satisfied. However, I am not familiar with the relevant tools to answer your requirements further than I have done above.

[morphal]

Well then! Thank you very much for taking the time to offer me that much info. I may attempt to use some of that on my existing systems, just to see if I can get it right.

My problem is that I can probably get each of those services to run from one machine but I have no idea how to integrate them. phpBB and e107 use one database for registrations and content. A Jabber server would use another. A mail server would use yet another. Trying to get all of it setup, setup securely, and working together is my real challenge.

[Hu]

Generally, Linux programs do not conflict with completely unrelated programs, so there should not be a problem with having multiple services on one box, even if they all use the same type of database for authentication. At this point, I think your best bet would be to pick out some candidate packages and give them a try. If you run into problems, start a thread in the appropriate forum about the troublesome package. Someone should be able to help you, usually within a day or two.

Validating that you have secured the package correctly is harder. If you are concerned that you may have done it wrong, post your current configuration and ask for advice on potential holes.

[djprotoss]

I'm currently playing around with something similar,
Heres a couple of thoughts:

I'm using openvz (although vserver is another option*) to provide the chrooting (I view containers as a heavyweight chroot rather than lightweight virtualisation). Xen could also be an option, although at a slight performance hit I guess...

for backups, put the whole thing on top of LVM and use its snapshot ability to enable easy backing up (you can effectively 'freeze' the state of a filesystem in a snapshot and then get a synchonised backup at your leisure. Of course you have to ensure your apps cache's are flushed at this point, but thats a potential issue with any backup strategy)

*vserver takes up less space because of the handy hardlinks and vunify, but openvz nicely seperates the network interfaces so each chrooted instance just appears as another seperate machine and you don't have to worry about servers that bind to ::/*

[morphal]

I'm not worried about them conflicting, exactly. I know they can all run on the same database. My question is how to get them all to integrate. I've got a small group of users, sure, but I'd like to give them a tool that's actually useful for communicating with each other. For the overall system to be useful, it has to be easy to use. If they have to manage separate logins for each of 3-5 (or more, as I add things) services, they're just not going to bother using them.

I'd be interested to hear more about how you're doing that, dj. My experience with chrooting (or VMs) has been fairly limited but I hate the idea of a single exploit in Apache (or whatever) ruining the rest of my machine. I know it's a good idea to learn more and use them so I feel obliged to.