DIY virtualized router with real hardware

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

@NeddySeagoon,

I'm currently running a setup very much like what you're describing in that doc. I have a /boot and then the rest of it is LVM2. The host_root volume is in the same pool as the guests.

So I guess I need to ask, what can I do in a VM to set this up and have it actually work for a real install? I have VMs running on this thing that would be difficult to move.

I'm using Ubuntu 12.04 as the host OS right now because I don't want to have to keep tinkering with it and it's the lowest maintenance thing I've used recently. But that's not what I had on there at first, I've reinstalled the host OS several times now without touching any of the VMs. Back up the VM non-disk files, restore them and you're done.

I know the hardware drivers will change, but can I emulate the actual CPU and build a real kernel at least? That would be the real clincher for me, it takes a long time to get through that. And I've never even attempted a hardened anything. I would think I could add support for the real hardware AND the virtio drivers, and maybe get it working that way. Am I correct?

Here's the deal: I'm using an ssd as the boot drive, I have /boot as a regular partition and everything else is LVM2.

I have the SSD, a plain hard disk and a raid1 array.

NeddySeagoon · Posted: Fri Oct 18, 2013 5:07 pm Post subject:

1clue,

It should be fine. KVM performace is best if you tell the KVM to use the same CPU as the real hardware.
I did it the other way round to you, I did the real bare metal hardware install. When that worked, I tarred it as a stage4 to make a master KVM, which isn't used, it just gets cloned before I customise the cline for the application at hand.

That reminds me. My master KVM is about 18 months out of date.

Going the other way should work too but your KVM starter kernel may need three sets of drivers.
1. Virtio Drivers (Only used in the KVMs)
2. Drivers for the emulated hardware in the KVM (may not be used ever - but see below)
3. Drivers for your real bare metal hardware (so you can move the KVM to the bare metal).

What will you use for a boot loader in the KVMs ?
If you use grub and a kernel with the virtio support, you will need to pass grub a device.map file as the virtio partitions will have no BIOS equivilents.
If you install grub using the KVM hardware drivers, you will have /dev/sda and friends in the KVM and grub will install normally
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

NeddySeagoon,

The box has a real /boot partition. It's using grub2 I think right now.

I guess I'm not sure what makes most sense for me. I'm guessing that a few VMs will be based on Gentoo, and then the bare metal box. So maybe virtio for speed, and then real hardware drivers in case I use one on the bare metal. Does that sound reasonable?

For VMs, I typically install a minimal VM of a distro and then clone it for whatever purpose. Actually the one I started is my "base" for new Gentoo, if it all works out right. What you just said about kernels reinforces this, I think once I get it right I won't need to have hardly any down time, I can compile the kernel on a base vm and try it as I need.

I made the partition map pretty much as the wiki page suggests, and added a volume for /boot as well. I was going to set up the boot volume for use when it's a VM, and then copy stuff over when I want to boot the bare metal from it.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

I just ordered one of these: http://www.newegg.com/Product/Product.aspx?Item=33-114-037

I'm not sure why they're so cheap, but it says they work on Linux.

If it works well I'll probably order another one.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

OK I have another question. Maybe I'm just nervous about this being a VM that will become the core OS.

When I map these logical volumes to the VM which will become the Gentoo host, I figure I'll define them as a virtio filesystem in the VM, but then I have a question.

Is it best to leave the device paths exactly as they are? Or make them appear to be /dev/vg/host_whatever?

In VMs I typically use a minimal partition strategy, just a / and maybe a swap, and maybe tmpfs here and there. It makes it easier to move things.

But with Gentoo I don't really like that idea, not really sure why that makes a difference though. Except that with this installation I intend to have a copy of it as the real host.

Is it very difficult to make a single-partition VM and then split it out when I know it's working? Never really had luck with that when actually running the machine, but here I could use another OS (maybe the installer) to deal with it.

NeddySeagoon · Posted: Sat Oct 19, 2013 8:43 pm Post subject:

1clue,

That startech 4 post NIC looks to only have a single lane PCIe bus, so you get at most, 1.5Gbit/sec over the PCIe interface.
That means it won't support all four LAN ports going flat out.
My Intel 4 port NIC needs a PCIeX4 slot.

Not being able to drive your network ports flat out may not be a problem. It wouldn't be for me as I have mt internet connection in one, at 80/20 Mbit, 54 Mbit wireless on another.
A raspberry pi on a 3rd. and my protected wired network, that is all 1G everywhere on the fourth.

The virtio drivers are in two parts. The host part, which you need in the bare metal kernel and the VM part, which you need in the VMs.
You must have the right real hardware drivers in the bare metal kernel since there is nothing to provide the other half of the virtio drivers.

I can post kernel configs if you want.

Splitting up a single partition install when you move it is trivial. Tar it up. Make the new filesystem tree. untar it in its new home. Fix /etc/fstab, grub.conf hostname and the ssh host key.
Install grub to the new MBR and it may boot first time. You may need a new kernel if the hardware has changed - it will have when you move from a VM to real hardware but you can make a kernel that works in both places.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

What a drag. I knew there was something wrong with that card. I will need to have some high speed ports, in fact when I get my next box up I might need to bond a couple ethernet channels between the existing i7 and the other. Or maybe I can figure a way to segment the network by high traffic connections between VMs.

Well, I guess I'll use that one for the low-speed stuff when I get another card. Live and learn. I can use one of them for the cable anyway, give it priority over everything and let it go at that.

I will be interested in your kernel configs.

There's got to be a better way of looking at this. Regular hardware I'm fine with, VMs I'm fine with. This VM-as-real-router-with-real-devices thing makes my head hurt. Once I get it up I'll undoubtedly think it was easy, but it's one step to the left of black magic right now.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

It seems it's pcix*4. I'm in luck. They're just on sale for $100 off or so. The reviews say they're getting full speed out of them. Maybe it's not a bad deal after all.

http://www.startech.com/Networking-IO/Adapter-Cards/4-Port-PCI-Express-Gigabit-Ethernet-NIC-Network-Adapter-Card~ST1000SPEX4

NeddySeagoon · Posted: Sun Oct 20, 2013 3:07 pm Post subject:

1clue,

I couidn't find any data, so I was judging by the PCIe connector size.

For real hardware in a VM the bare metal hardware kernel needs PCI pass through support.
Like I said, my hardware has a bug so I never made it work.

VM kernel
Bare metal kernel
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

You're using multilib on your host kernel. Is that necessary? Can't I just do that in the guests?

I'm going to want multilib on guests, I just thought it was purely the OS, in which case the guest would be fine.

This is taking a lot longer than I'd anticipated. I'm having to do real work, and trying to wedge an install of Gentoo in between the spaces.

I've finally gotten around to kernel config.

By the time I actually install it my @#%@ network card will probably be here.

NeddySeagoon · Posted: Mon Oct 21, 2013 8:04 pm Post subject:

1clue,

I was not aware I was using multilib at all. On that box or anywhere else.
I'll need to look into that.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

Your kernel config has it as one of the first settings.

I will definitely want one or more of the guests on there to be multilib, but I don't think I want the host to be multilib unless it actually needs to be.

OK thanks. I'll get through it sooner or later, hopefully before the @#$% nic gets here.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

I lost the race. I just got my network card, and I haven't had time to get Gentoo working.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

The good news is the existing xubuntu install recognized it with no problems, so there's no driver issue.

I still need to do real work before I can play with this more.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

I sort of abandoned the link to your system and went the normal handbook route.

Is there a reason to use hardened+selinux on the host image? More importantly, if I have hardened can I then add selinux later?

I've never done either of these before, so I don't know.

Thanks.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

emerge-webrsync doesn't seem to work, so I downloaded a portage and extracted it myself. I hope that's all emerge-webrsync was supposed to do.

Mad Merlin · Veteran Joined: 09 May 2005 Posts: 1155

It looks like you already got past the hardware selection process, but I've been very pleased with the EdgeRouter Lite. It meets all of your requirements for $100, ships with software based on Vyatta, but can run Gentoo too. I have several of them running in a variety of roles, including as a router for my home network. Highly recommended.
_________________
Game! - Where the stick is mightier than the sword!

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

@Mad Merlin,

It looks interesting for normal circumstances, but given NeddySeagoon's virtual router I think that's the way for me. This network I'm making is a few pieces of hardware and a bunch of VMs anyway.

Well, let me amend that: I have a boatload of home devices with whatever random junk we buy or my wife downloads from the net. That's going to be partitioned off by this device, over to the router appliance with wireless. Aside of that, it's going to be a few virtualization servers and a workstation. Trying to figure out a way to do this with only one 4-port NIC on the host, but so far I think I'm going to need another one.

It's odd, lspci shows EXACTLY the same hardware type for the 4-way NIC as what's on my motherboard. Realtek RTL8111/8168. Go figure.

It's been a _LONG_ time since I've started a Gentoo install. I've never done a hardened anything on any distro, and so this has a lot of newness to me, and I detect a lot of learning on the virtualized router end too.

I was just thinking, this box I'm installing on now is the first Gentoo install I ever made. I bought the hardware all new and installed Gentoo as the first OS. So this is homecoming in a way, even though it's been wiped and reinstalled with a BUNCH of different distros since.

I got lost in the infinity of kernel options, then reloaded Neddy's bare metal kernel and fixed it for my hardware, ignoring all else.

This thing is dragging on way too long, I have to get it done.

Question to anyone: When I'm making a KVM host, do I need filesystem support for all the guest operating systems as well? I'm using LVM mapping to manage filesystems directly on the host.

I have:

/boot is a real partition.
ssd volume group
hdd volume group (just a disk, no RAID or anything, for things like downloads.
raid 1 volume group
A removable slot-load sata drive for backups.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

Renamed from "Need hardware recommendations for DIY router" since that no longer defines the project.

I realized the title no longer matches the project. Rather than start a new thread I renamed the existing one.

Thanks to everyone so far for your input. It's been educational and interesting.

NeddySeagoon · Posted: Mon Oct 28, 2013 9:37 pm Post subject:

1clue,

I'm only using hardened, not SELinux

I pass logical volumes directly to the KVMs To the bare metal, they are just block devices. The bare metal has no knowledge of the filesystems im use by the KVMs.
Thats true if you use file systems in a file too.

My 4 port NIC is used for

1. The internet my public /29
2. The protected wired network 192.168.100.0/24
3. The protected wireless network 192.168.54.0/24
4. Internet facing services 192.168.10.0/24
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

NeddySeagoon,

I compiled in some extra filesystem support then.

I have the kernel for the "bare metal" compiled. I ran out of free time and it's sitting there, I'll get started again later.

Thanks.