Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Bridged eth0, KVM, access to guest from network [solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
dkasak
n00b
n00b


Joined: 10 Feb 2012
Posts: 61

PostPosted: Tue Jun 10, 2014 6:43 am    Post subject: Bridged eth0, KVM, access to guest from network [solved] Reply with quote

Hi all. I'm having an interesting problem. I have to run a bunch of VMs, that should sit on the same network as other PCs on my local network. I've set up my bridged networking ( bridging an ethernet device, not wireless ) as per the instructions for systemd users, at: http://wiki.gentoo.org/wiki/Network_bridge#Host_Configuration_.28using_systemd.29. ie I have:

Code:
# bridge ports defined empty to avoid DHCP being run for their configuration
config_eth0="null"

# Bridge-config

config_br0="192.168.1.128"
routes_br0="192.168.1.254"
brctl_br0="setfd 0 sethello 10 stp on"
bridge_br0="eth0"


... and the /etc/systemd/system/bridge@.service file as per the instructions.

I've set up a couple of VMs now. I can ssh into them from the host, and also ssh to the host from the guest. But I can't ssh to guests from other PCs on the network, or visa-versa. I can, however, ping things. eg:

Pinging a VM from another PC on network:

Code:
ultrabook ~ # ping 192.168.1.130
PING 192.168.1.130 (192.168.1.130) 56(84) bytes of data.
64 bytes from 192.168.1.130: icmp_seq=1 ttl=64 time=2.21 ms
64 bytes from 192.168.1.130: icmp_seq=2 ttl=64 time=1.94 ms
^C
--- 192.168.1.130 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.941/2.078/2.215/0.137 ms
ultrabook ~ #


Attempting to ssh into it:
Code:
ultrabook ~ # ssh 192.168.1.130
ssh: connect to host 192.168.1.130 port 22: No route to host
ultrabook ~ #


Why would that be? I'm admittedly not so hot on networking :?

Oh - on the VM host:

Code:
tesla ~ # brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.d050991071d2       no              eth0
                                                        vnet0
virbr0          8000.000000000000       yes
virbr1          8000.525400af33dd       yes             virbr1-nic
                                                        vnet1
                                                        vnet2
tesla ~ #


Any help greatly appreciated :)


Last edited by dkasak on Fri Jun 13, 2014 12:09 am; edited 1 time in total
Back to top
View user's profile Send private message
schorsch_76
Guru
Guru


Joined: 19 Jun 2012
Posts: 450

PostPosted: Tue Jun 10, 2014 6:57 am    Post subject: Reply with quote

You need to add tap devices to the bridge. These tap devices are used by the qemu/kvm VMs to access the network.

I do it the same. Here is my config:

Code:

#################################################
# the bridge br0 contains eth0 and tap0
# tap0: virtual machine
# tap1: VPN
# eth0: Network
#################################################
config_eth0="null"

tuntap_tap0="tap"
config_tap0="null"
tunctl_tap0="-t tap0 -u me"

config_br0="192.168.178.19/24"
routes_br0="default via 192.168.178.1"
brctl_br0="setfd 0 sethello 10 stp off"
bridge_br0="eth0 tap0"

depend_br0() {
    need net.eth0
    need net.tap0
}
Back to top
View user's profile Send private message
dkasak
n00b
n00b


Joined: 10 Feb 2012
Posts: 61

PostPosted: Tue Jun 10, 2014 7:40 am    Post subject: Reply with quote

Thanks for the response. Alas, this config doesn't work for me, and in fact it breaks connectivity completely. Maybe it's because I'm using systemd? I end up with BOTH eth0 and br0 'active' and with IP 192.168.1.128. I've never seen eth0 get an IP address when using bridging. I guess for now, I can try to revert to 'vanilla' networking, and use KVM's "macvtap" thing. It's a bit annoying that you can access the host from the guest, but I'm looking at workarounds, and they're not TOO bad.
Back to top
View user's profile Send private message
schorsch_76
Guru
Guru


Joined: 19 Jun 2012
Posts: 450

PostPosted: Tue Jun 10, 2014 8:02 am    Post subject: Reply with quote

systemd should not have effect on the network. I am not sure how systemd manages "depends_br0()". I use plain openrc without networkmanager/wicd/whatever.

In bridge setup only the bridge interface has an ip address. Maybe you need to shut the network complete down and start it again. The important thing is, that the tap device and eth0 are in the same bridge and only the br0 has the ip adress. The tap device get assigned to the QEMU VM's. If you use systemd or openrc to setup this config doesn't matter.
Back to top
View user's profile Send private message
dkasak
n00b
n00b


Joined: 10 Feb 2012
Posts: 61

PostPosted: Tue Jun 10, 2014 9:45 am    Post subject: Reply with quote

Yeah I rebooted multiple times while attempting to figure things out.

I'm pretty sure systemd is involved in all this, as it's the init scripts that parse the config files, bring up the interfaces, etc. The WIKI ( http://wiki.gentoo.org/wiki/Network_bridge#Host_Configuration_.28using_systemd.29 ) includes a new service file /etc/systemd/system/bridge@.service that has 'simple' support for bridging, but doesn't appear to bring up the tap0 interface. So I guess this isn't "strictly" a systemd issue, but an issue with the much simpler systemd service file that I'm using, compared to the more complex networking scripts that Gentoo has had up until systemd arrived.
Back to top
View user's profile Send private message
schorsch_76
Guru
Guru


Joined: 19 Jun 2012
Posts: 450

PostPosted: Tue Jun 10, 2014 9:52 am    Post subject: Reply with quote

Ok, then it is most probably a issue with the systemd service file.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jun 10, 2014 2:55 pm    Post subject: Reply with quote

Lovely tip schorsch. I've saved it off in my tips thread.

Would be cool to see the setup for the other VPN interface as well.

Thanks,
steveL.
Back to top
View user's profile Send private message
Aiken
Apprentice
Apprentice


Joined: 22 Jan 2003
Posts: 239
Location: Toowoomba/Australia

PostPosted: Tue Jun 10, 2014 10:23 pm    Post subject: Reply with quote

schorsch_76 wrote:
You need to add tap devices to the bridge. These tap devices are used by the qemu/kvm VMs to access the network.


I do it differently. Instead of setting up tapX in /etc/conf.d/net I let quemu configure the tap device at runtime. At boot time my bridge consists of eth0, eth1 and wlan0. The tap devices are created by qemu at runtime and it has options to name scripts to be run when it brings up and takes down the tap device. Qemu passes the name of the tap device to the script. I use simple scripts for when I wanted to start qemu and for qemu-ifup. This adds the tapX devices to br0 on demand instead of preallocating them at boot time.

Code:

qemu-system-x86_64 -enable-kvm -cpu host \
   -drive file=${1},if=virtio \
   -net nic -net tap,script=qemu-ifup,downscript=qemu-ifdown \
   -m 2000M \
   -daemonize \
   -vnc :$2 -smp 2 \
   -pidfile pid-$2 \
   -usbdevice tablet


Code:

#!/bin/bash

#echo Interface $1
ifconfig $1 0.0.0.0 promisc up
brctl addif br0 $1

_________________
Beware the grue.
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3919
Location: Hamburg

PostPosted: Tue Jun 10, 2014 10:50 pm    Post subject: Reply with quote

I'm just curious about qemu-ifdown script ...
Back to top
View user's profile Send private message
Aiken
Apprentice
Apprentice


Joined: 22 Jan 2003
Posts: 239
Location: Toowoomba/Australia

PostPosted: Tue Jun 10, 2014 11:15 pm    Post subject: Reply with quote

My qemu-ifdown script only had an echo in it to see what arguments were passed to it. Thinking I forgot to do anything with it as when that qemu instance stops the tap device is destroyed and removed from the bridge.

Code:

#!/bin/bash

#echo Interface ${1}
brctl delif br0 ${1}


While writing this post I changed my qemu-ifdown to that. That removes the tapX device from the bridge instead of hoping it is automatically removed when qemu stops.
_________________
Beware the grue.
Back to top
View user's profile Send private message
dkasak
n00b
n00b


Joined: 10 Feb 2012
Posts: 61

PostPosted: Wed Jun 11, 2014 9:37 am    Post subject: Reply with quote

OK I've changed my approach. virt-manager, and quite possibly libvirt and other related things, are flakey shit. I had a massive number of iptables rules, most of them totally invalid, which were responsible for not only 75% of my VM issues, but also prevented the system from booting properly, and eventually Gnome from starting at all. This seems to be related to me passing br0 to virt-manager and telling it to set up VMs using this interface.

I ended up deleting /etc/libvirt and starting again, passing in my disk images, and avoiding using gentoo's "kinda half-way-there" bridged support with systemd. I now have my normal eth0 active, and I've set up macvtap bridging. I then created a host-only VPN and attached a 2nd network interface to it. So now:

- PCs on the LAN can access the guest via the macvtap interface
- the host can access the guest via the host-only VPN

<phew> :P

Incidentally, it would be great if this wasn't necessary - I don't remember this being necessary with other VMs, but then I guess I can't complain - it's open-source.

If I abandoned virt-manager and used the command-line, I might have a better experience, but I unfortunately don't have time for that now - this is for work, required yesterday. If I have to abandon virt-manager, I might as well go back to virtualbox / vmware, and run with an 'evaluation' license or whatever ;)
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Jun 12, 2014 5:49 pm    Post subject: Reply with quote

dkasak wrote:
I now have my normal eth0 active, and I've set up macvtap bridging. I then created a host-only VPN and attached a 2nd network interface to it. So now:

- PCs on the LAN can access the guest via the macvtap interface
- the host can access the guest via the host-only VPN

Great :) Can you post (redacted-for-work) configs to show the setup, and how the various things tie together?

Even if it's only two files, or you just refer to above, it helps to have the specific solution to the problem asked, as a guide to others in the future.
Back to top
View user's profile Send private message
dkasak
n00b
n00b


Joined: 10 Feb 2012
Posts: 61

PostPosted: Fri Jun 13, 2014 12:08 am    Post subject: Reply with quote

OK sure. It's pretty simple. I'd taken some screenshots and gone to upload them, but ... the forums don't let you post images. WTF? Whatever. I use virt-manager to configure my VMs. Those who want to use the command-line can post the command-line alternatives ...

First I go into virt-manager, and double-click on 'localhost (QEMU). This opens a 'Connection Details' window. Click the 'Virtual Networks' page / tab / whatever. Click on the 'add' button in the bottom-left of the window. Fill in the wizard. It doesn't really matter what network you define, I guess as long as it's not the same network as your LAN.

Typically people would use the built-in DHCP server, but for my purpose I disabled this, as one of the VMs that I'm using actually needed a VPN for guest-to-guest communication, and it provides it's own DHCP server. If you have something similar going on in your guests, note that your host will create a network interface that sits inside this VPN, and you should be careful that you don't have multiple DHCP servers, or IP address clashes between host and guests.

Next I go to the guest configuration, and define ...

1st network interface:

Code:
Source device: Host device eth0: macvtap
Device model: virtio
Source mode: VEPA


2nd network interface:

Code:
Virtual network 'nznet' : Isolated network, internal and host routing only
Device model: virtio


The 'Device model' you choose must be supported in the guest(s) you're running. 'virtio' is recommended if your guests have kernel support for this device. Otherwise use something else. The 'Source mode' in the device running in 'macvtap' mode doesn't appear to make much difference for what I'm using it for.

After you've configured this way, everything on the LAN except for the host itself ( and possibly other guests ) can communicate with the guest via the 1st network interface, which will have an IP address in your LAN. Your host can communicate with the guest via the 2nd network interface, which sits in the VPN.

---

It's not quite perfect, but it's an acceptable hack, and much less hassle than doing bridging 'properly'. It also doesn't freak out network-manager, which can have flow-on effects, with other network-manager-dependent services not starting up. I've actually had bridging working properly before with both VirtualBox ( before it supported doing it via it's own kernel module ... and also before everything depended on network-manager ) and KVM / QEMU ... but as noted above, it doesn't seem to work so well these days :/ Such is life.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3095

PostPosted: Fri Jun 13, 2014 3:43 pm    Post subject: Reply with quote

My way to make it work with command line. Just in case someone might find it useful.
Code:
cat ./qemu_launcher.sh
#! /bin/bash

# HELP: This script is intended to be ran by user with limited priviliges.
# HELP: User must be allowed to run 'sudo tunctl', 'sudo brctl' and 'sudo ifconfig'  without password in order to work.
# HELP: Intended invocation of this script is:
# HELP: "cmd=<params you want to append to qemu command line> <THIS FILE'S NAME> <virtual machines name> [name of the bridge to bind]"
# HELP: Multiple bridges are allowed. Bridges must already exist and be configured before this script is started (e.g. configured by init scripts).
# HELP: Feel free to edit contents of "static_params" near the end of this file to define set of options you want always active.


add_ifaces ()
{
i=1
USER=$(whoami)
while [ ${#1} -gt 0 ]
        do sudo /usr/bin/tunctl -u $USER -t "${name}-${i}" > /dev/null
        sudo /sbin/brctl addif $1 "${name}-${i}" > /dev/null
        sudo /bin/ifconfig "${name}-${i}" up promisc > /dev/null
        # HINT: VIRTIO NETWORK IS DISABLED BY DEFAULT, ADJUST THE FOLLOWING LINE IF YOU NEED IT
        params="${params} -net nic,macaddr=$( printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256)) ),vlan=$i -net tap,ifname=$name-$i,script=no,downscript=no,vlan=$i"
        shift
        i=$(($i+1))
done
echo $params
}

del_ifaces ()
{
i=1
while [ ${#1} -gt 0 ]
        do sudo /bin/ifconfig "${name}-${i}" down > /dev/null
        sudo /sbin/brctl delif $1 "${name}-${i}" > /dev/null
        sudo /usr/bin/tunctl -d "${name}-${i}" > /dev/null
        shift
        i=$(($i+1))
done
}

main ()
{
name=$1
shift
bridges="$@"
networking="$(add_ifaces ${bridges})"
RUN="qemu-system-x86_64 -name ${name} ${static_params} ${networking} ${cmd}"
echo starting machine $name with command
echo $RUN
$RUN
del_ifaces ${bridges}
}

        # HINT: PUT YOUR FAVOURITE OPTIONS HERE
static_params="-enable-kvm -m 1024"
main $@
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum