Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
DIY virtualized router with real hardware
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Fri Oct 18, 2013 4:40 am    Post subject: Reply with quote

@NeddySeagoon,

I'm currently running a setup very much like what you're describing in that doc. I have a /boot and then the rest of it is LVM2. The host_root volume is in the same pool as the guests.

So I guess I need to ask, what can I do in a VM to set this up and have it actually work for a real install? I have VMs running on this thing that would be difficult to move.

I'm using Ubuntu 12.04 as the host OS right now because I don't want to have to keep tinkering with it and it's the lowest maintenance thing I've used recently. But that's not what I had on there at first, I've reinstalled the host OS several times now without touching any of the VMs. Back up the VM non-disk files, restore them and you're done.

I know the hardware drivers will change, but can I emulate the actual CPU and build a real kernel at least? That would be the real clincher for me, it takes a long time to get through that. And I've never even attempted a hardened anything. I would think I could add support for the real hardware AND the virtio drivers, and maybe get it working that way. Am I correct?

Here's the deal: I'm using an ssd as the boot drive, I have /boot as a regular partition and everything else is LVM2.

I have the SSD, a plain hard disk and a raid1 array.

Code:

root@chronos:~# vgs
  VG     #PV #LV #SN Attr   VSize   VFree 
  hddvg1   1   5   0 wz--n- 698.63g 435.15g
  rd1vg1   1   1   0 wz--n-   2.73t   2.44t
  ssdvg1   1   5   0 wz--n- 223.07g 159.07g
root@chronos:~# lvs
  LV                VG     Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  host_spinner      hddvg1 -wi-ao-- 153.13g                                           
  host_swap         hddvg1 -wi-ao--  22.35g                                           
  popeye            hddvg1 -wi-a---  60.00g                                           
  y_root            hddvg1 -wi-ao--  20.00g                                           
  z_root            hddvg1 -wi-ao--   8.00g                                           
  freenas_files     rd1vg1 -wi-a--- 300.00g                                           
  host_root         ssdvg1 -wi-ao--  30.00g                                           
  ...
and about 10 more.                                     
root@chronos:~#


I was originally just going to do a test run on your instructions so I know what to expect. Then I started wondering, why not just use it if it works?

Edit:
I guess I'm thinking something along these lines:

  1. Install Gentoo using a throw-away /boot on lvm and a "keeper" /.
  2. Build the kernel for the real hardware, which I guess I can do using lspci and all that from the real box.
  3. Probably have to support virtio devices too?
  4. Boot it and test it for actual functionality.
  5. Merge the throwaway /boot into the real /boot, update grub to support dual boot and reboot.


I don't see why this wouldn't work, but I'd like to have some sort of reinforcement that I'm not half baked.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31342
Location: 56N 3W

PostPosted: Fri Oct 18, 2013 5:07 pm    Post subject: Reply with quote

1clue,

It should be fine. KVM performace is best if you tell the KVM to use the same CPU as the real hardware.
I did it the other way round to you, I did the real bare metal hardware install. When that worked, I tarred it as a stage4 to make a master KVM, which isn't used, it just gets cloned before I customise the cline for the application at hand.

That reminds me. My master KVM is about 18 months out of date.

Going the other way should work too but your KVM starter kernel may need three sets of drivers.
1. Virtio Drivers (Only used in the KVMs)
2. Drivers for the emulated hardware in the KVM (may not be used ever - but see below)
3. Drivers for your real bare metal hardware (so you can move the KVM to the bare metal).

What will you use for a boot loader in the KVMs ?
If you use grub and a kernel with the virtio support, you will need to pass grub a device.map file as the virtio partitions will have no BIOS equivilents.
If you install grub using the KVM hardware drivers, you will have /dev/sda and friends in the KVM and grub will install normally
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Sat Oct 19, 2013 12:51 am    Post subject: Reply with quote

NeddySeagoon,

The box has a real /boot partition. It's using grub2 I think right now.

I guess I'm not sure what makes most sense for me. I'm guessing that a few VMs will be based on Gentoo, and then the bare metal box. So maybe virtio for speed, and then real hardware drivers in case I use one on the bare metal. Does that sound reasonable?

For VMs, I typically install a minimal VM of a distro and then clone it for whatever purpose. Actually the one I started is my "base" for new Gentoo, if it all works out right. What you just said about kernels reinforces this, I think once I get it right I won't need to have hardly any down time, I can compile the kernel on a base vm and try it as I need.

I made the partition map pretty much as the wiki page suggests, and added a volume for /boot as well. I was going to set up the boot volume for use when it's a VM, and then copy stuff over when I want to boot the bare metal from it.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Sat Oct 19, 2013 1:49 am    Post subject: Reply with quote

I just ordered one of these: http://www.newegg.com/Product/Product.aspx?Item=33-114-037

I'm not sure why they're so cheap, but it says they work on Linux.

If it works well I'll probably order another one.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Sat Oct 19, 2013 8:08 pm    Post subject: Reply with quote

OK I have another question. Maybe I'm just nervous about this being a VM that will become the core OS.

When I map these logical volumes to the VM which will become the Gentoo host, I figure I'll define them as a virtio filesystem in the VM, but then I have a question.

Is it best to leave the device paths exactly as they are? Or make them appear to be /dev/vg/host_whatever?

In VMs I typically use a minimal partition strategy, just a / and maybe a swap, and maybe tmpfs here and there. It makes it easier to move things.

But with Gentoo I don't really like that idea, not really sure why that makes a difference though. Except that with this installation I intend to have a copy of it as the real host.

Is it very difficult to make a single-partition VM and then split it out when I know it's working? Never really had luck with that when actually running the machine, but here I could use another OS (maybe the installer) to deal with it.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31342
Location: 56N 3W

PostPosted: Sat Oct 19, 2013 8:43 pm    Post subject: Reply with quote

1clue,

That startech 4 post NIC looks to only have a single lane PCIe bus, so you get at most, 1.5Gbit/sec over the PCIe interface.
That means it won't support all four LAN ports going flat out.
My Intel 4 port NIC needs a PCIeX4 slot.

Not being able to drive your network ports flat out may not be a problem. It wouldn't be for me as I have mt internet connection in one, at 80/20 Mbit, 54 Mbit wireless on another.
A raspberry pi on a 3rd. and my protected wired network, that is all 1G everywhere on the fourth.

The virtio drivers are in two parts. The host part, which you need in the bare metal kernel and the VM part, which you need in the VMs.
You must have the right real hardware drivers in the bare metal kernel since there is nothing to provide the other half of the virtio drivers.

I can post kernel configs if you want.

Splitting up a single partition install when you move it is trivial. Tar it up. Make the new filesystem tree. untar it in its new home. Fix /etc/fstab, grub.conf hostname and the ssh host key.
Install grub to the new MBR and it may boot first time. You may need a new kernel if the hardware has changed - it will have when you move from a VM to real hardware but you can make a kernel that works in both places.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Sun Oct 20, 2013 12:31 am    Post subject: Reply with quote

What a drag. I knew there was something wrong with that card. I will need to have some high speed ports, in fact when I get my next box up I might need to bond a couple ethernet channels between the existing i7 and the other. Or maybe I can figure a way to segment the network by high traffic connections between VMs.

Well, I guess I'll use that one for the low-speed stuff when I get another card. Live and learn. I can use one of them for the cable anyway, give it priority over everything and let it go at that.

I will be interested in your kernel configs.

There's got to be a better way of looking at this. Regular hardware I'm fine with, VMs I'm fine with. This VM-as-real-router-with-real-devices thing makes my head hurt. Once I get it up I'll undoubtedly think it was easy, but it's one step to the left of black magic right now.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Sun Oct 20, 2013 1:32 am    Post subject: Reply with quote

It seems it's pcix*4. I'm in luck. They're just on sale for $100 off or so. The reviews say they're getting full speed out of them. Maybe it's not a bad deal after all.

http://www.startech.com/Networking-IO/Adapter-Cards/4-Port-PCI-Express-Gigabit-Ethernet-NIC-Network-Adapter-Card~ST1000SPEX4
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31342
Location: 56N 3W

PostPosted: Sun Oct 20, 2013 3:07 pm    Post subject: Reply with quote

1clue,

I couidn't find any data, so I was judging by the PCIe connector size.

For real hardware in a VM the bare metal hardware kernel needs PCI pass through support.
Like I said, my hardware has a bug so I never made it work.

VM kernel
Bare metal kernel
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Mon Oct 21, 2013 2:01 am    Post subject: Reply with quote

You're using multilib on your host kernel. Is that necessary? Can't I just do that in the guests?

I'm going to want multilib on guests, I just thought it was purely the OS, in which case the guest would be fine.

This is taking a lot longer than I'd anticipated. I'm having to do real work, and trying to wedge an install of Gentoo in between the spaces.

I've finally gotten around to kernel config.

By the time I actually install it my @#%@ network card will probably be here. :(
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31342
Location: 56N 3W

PostPosted: Mon Oct 21, 2013 8:04 pm    Post subject: Reply with quote

1clue,

I was not aware I was using multilib at all. On that box or anywhere else.
I'll need to look into that.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Mon Oct 21, 2013 9:12 pm    Post subject: Reply with quote

Your kernel config has it as one of the first settings.

I will definitely want one or more of the guests on there to be multilib, but I don't think I want the host to be multilib unless it actually needs to be.

OK thanks. I'll get through it sooner or later, hopefully before the @#$% nic gets here.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Wed Oct 23, 2013 6:12 pm    Post subject: Reply with quote

I lost the race. I just got my network card, and I haven't had time to get Gentoo working. :(
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Wed Oct 23, 2013 10:54 pm    Post subject: Reply with quote

The good news is the existing xubuntu install recognized it with no problems, so there's no driver issue.

I still need to do real work before I can play with this more. :(
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Thu Oct 24, 2013 6:19 am    Post subject: Reply with quote

I sort of abandoned the link to your system and went the normal handbook route.

Is there a reason to use hardened+selinux on the host image? More importantly, if I have hardened can I then add selinux later?

I've never done either of these before, so I don't know.

Thanks.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Thu Oct 24, 2013 6:29 am    Post subject: Reply with quote

emerge-webrsync doesn't seem to work, so I downloaded a portage and extracted it myself. I hope that's all emerge-webrsync was supposed to do.
Back to top
View user's profile Send private message
Mad Merlin
Veteran
Veteran


Joined: 09 May 2005
Posts: 1141

PostPosted: Sun Oct 27, 2013 5:28 am    Post subject: Reply with quote

It looks like you already got past the hardware selection process, but I've been very pleased with the EdgeRouter Lite. It meets all of your requirements for $100, ships with software based on Vyatta, but can run Gentoo too. I have several of them running in a variety of roles, including as a router for my home network. Highly recommended.
_________________
Game! - Where the stick is mightier than the sword!
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Mon Oct 28, 2013 4:59 am    Post subject: Reply with quote

@Mad Merlin,

It looks interesting for normal circumstances, but given NeddySeagoon's virtual router I think that's the way for me. This network I'm making is a few pieces of hardware and a bunch of VMs anyway.

Well, let me amend that: I have a boatload of home devices with whatever random junk we buy or my wife downloads from the net. That's going to be partitioned off by this device, over to the router appliance with wireless. Aside of that, it's going to be a few virtualization servers and a workstation. Trying to figure out a way to do this with only one 4-port NIC on the host, but so far I think I'm going to need another one.

It's odd, lspci shows EXACTLY the same hardware type for the 4-way NIC as what's on my motherboard. Realtek RTL8111/8168. Go figure.

It's been a _LONG_ time since I've started a Gentoo install. I've never done a hardened anything on any distro, and so this has a lot of newness to me, and I detect a lot of learning on the virtualized router end too.

I was just thinking, this box I'm installing on now is the first Gentoo install I ever made. I bought the hardware all new and installed Gentoo as the first OS. So this is homecoming in a way, even though it's been wiped and reinstalled with a BUNCH of different distros since.

I got lost in the infinity of kernel options, then reloaded Neddy's bare metal kernel and fixed it for my hardware, ignoring all else.

This thing is dragging on way too long, I have to get it done.



Question to anyone: When I'm making a KVM host, do I need filesystem support for all the guest operating systems as well? I'm using LVM mapping to manage filesystems directly on the host.

I have:

  1. /boot is a real partition.
  2. ssd volume group
  3. hdd volume group (just a disk, no RAID or anything, for things like downloads.
  4. raid 1 volume group
  5. A removable slot-load sata drive for backups.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Mon Oct 28, 2013 5:08 am    Post subject: Reply with quote

Renamed from "Need hardware recommendations for DIY router" since that no longer defines the project.

I realized the title no longer matches the project. Rather than start a new thread I renamed the existing one.

Thanks to everyone so far for your input. It's been educational and interesting.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 31342
Location: 56N 3W

PostPosted: Mon Oct 28, 2013 9:37 pm    Post subject: Reply with quote

1clue,

I'm only using hardened, not SELinux

I pass logical volumes directly to the KVMs To the bare metal, they are just block devices. The bare metal has no knowledge of the filesystems im use by the KVMs.
Thats true if you use file systems in a file too.

My 4 port NIC is used for

1. The internet my public /29
2. The protected wired network 192.168.100.0/24
3. The protected wireless network 192.168.54.0/24
4. Internet facing services 192.168.10.0/24
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
1clue
Veteran
Veteran


Joined: 05 Feb 2006
Posts: 1192

PostPosted: Mon Oct 28, 2013 10:41 pm    Post subject: Reply with quote

NeddySeagoon,

I compiled in some extra filesystem support then. :)

I have the kernel for the "bare metal" compiled. I ran out of free time and it's sitting there, I'll get started again later.

Thanks.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum