| View previous topic :: View next topic |
| Author |
Message |
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
 Posted: Fri Jun 01, 2012 3:49 pm Post subject: [SOLVED] KVM and network bridge |
 |
|
I need help with the setup of network configuration of my Host system.
Goal is to set up a virtual hub so that the guests are on the same subnet as the host and are visible to other hosts on the net as if they were physical machines.
I tried to follow this instruction. To my opinion the instruction is somewhat ambiguous as there are two concepts (virtual hub and NAT/masquerading) mixed up. May be I didn't understand it all. So I need a little help here.
Up till now I have one FreeBSD guest (elrond) running, to which I have access from the host (dragon) via ssh. From the guest I can see the host but can not see any other machine on the net (featuring celeborn as an example below) as well as I can not get into the guest from any other machine except from the virtualization host.
My configuration so far looks like this:
| Code: |
dragon ~ # ifconfig eth0
eth0 Protokoll:Ethernet Hardware Adresse 00:19:99:b9:8d:2d
inet6 Adresse: fe80::219:99ff:feb9:8d2d/64 Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:45111 errors:0 dropped:0 overruns:0 frame:0
TX packets:4828 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:6556353 (6.2 MiB) TX bytes:472331 (461.2 KiB)
Interrupt:18 Speicher:cd420000-cd440000
dragon ~ # ifconfig br0
br0 Protokoll:Ethernet Hardware Adresse 00:00:00:00:00:02
inet Adresse:172.16.1.4 Bcast:172.16.31.255 Maske:255.255.224.0
inet6 Adresse: fe80::200:ff:fe00:2/64 Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23760 errors:0 dropped:0 overruns:0 frame:0
TX packets:3716 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:0
RX bytes:3601507 (3.4 MiB) TX bytes:368598 (359.9 KiB)
dragon ~ # ifconfig tap0
tap0 Protokoll:Ethernet Hardware Adresse 00:00:00:00:00:02
UP BROADCAST PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:1 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:500
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
dragon ~ # ifconfig tap1
tap1 Protokoll:Ethernet Hardware Adresse 00:00:00:00:00:03
inet6 Adresse: fe80::200:ff:fe00:3/64 Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:444 errors:0 dropped:0 overruns:0 frame:0
TX packets:18557 errors:0 dropped:1 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:500
RX bytes:38115 (37.2 KiB) TX bytes:2090768 (1.9 MiB)
dragon ~ # brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.000000000002 no eth0
tap0
tap1
dragon ~ # sysctl net.ipv4.conf.eth0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.promote_secondaries = 0
dragon ~ # sysctl net.ipv4.conf.br0
net.ipv4.conf.br0.forwarding = 0
net.ipv4.conf.br0.mc_forwarding = 0
net.ipv4.conf.br0.accept_redirects = 1
net.ipv4.conf.br0.secure_redirects = 1
net.ipv4.conf.br0.shared_media = 1
net.ipv4.conf.br0.rp_filter = 1
net.ipv4.conf.br0.send_redirects = 1
net.ipv4.conf.br0.accept_source_route = 1
net.ipv4.conf.br0.accept_local = 0
net.ipv4.conf.br0.src_valid_mark = 0
net.ipv4.conf.br0.proxy_arp = 1
net.ipv4.conf.br0.medium_id = 0
net.ipv4.conf.br0.bootp_relay = 0
net.ipv4.conf.br0.log_martians = 0
net.ipv4.conf.br0.tag = 0
net.ipv4.conf.br0.arp_filter = 0
net.ipv4.conf.br0.arp_announce = 1
net.ipv4.conf.br0.arp_ignore = 0
net.ipv4.conf.br0.arp_accept = 0
net.ipv4.conf.br0.arp_notify = 0
net.ipv4.conf.br0.proxy_arp_pvlan = 0
net.ipv4.conf.br0.disable_xfrm = 0
net.ipv4.conf.br0.disable_policy = 0
net.ipv4.conf.br0.force_igmp_version = 0
net.ipv4.conf.br0.promote_secondaries = 0
dragon ~ # sysctl net.ipv4.conf.tap0
net.ipv4.conf.tap0.forwarding = 0
net.ipv4.conf.tap0.mc_forwarding = 0
net.ipv4.conf.tap0.accept_redirects = 1
net.ipv4.conf.tap0.secure_redirects = 1
net.ipv4.conf.tap0.shared_media = 1
net.ipv4.conf.tap0.rp_filter = 1
net.ipv4.conf.tap0.send_redirects = 1
net.ipv4.conf.tap0.accept_source_route = 1
net.ipv4.conf.tap0.accept_local = 0
net.ipv4.conf.tap0.src_valid_mark = 0
net.ipv4.conf.tap0.proxy_arp = 0
net.ipv4.conf.tap0.medium_id = 0
net.ipv4.conf.tap0.bootp_relay = 0
net.ipv4.conf.tap0.log_martians = 0
net.ipv4.conf.tap0.tag = 0
net.ipv4.conf.tap0.arp_filter = 0
net.ipv4.conf.tap0.arp_announce = 0
net.ipv4.conf.tap0.arp_ignore = 0
net.ipv4.conf.tap0.arp_accept = 0
net.ipv4.conf.tap0.arp_notify = 0
net.ipv4.conf.tap0.proxy_arp_pvlan = 0
net.ipv4.conf.tap0.disable_xfrm = 0
net.ipv4.conf.tap0.disable_policy = 0
net.ipv4.conf.tap0.force_igmp_version = 0
net.ipv4.conf.tap0.promote_secondaries = 0
dragon ~ # sysctl net.ipv4.conf.tap1
net.ipv4.conf.tap1.forwarding = 0
net.ipv4.conf.tap1.mc_forwarding = 0
net.ipv4.conf.tap1.accept_redirects = 1
net.ipv4.conf.tap1.secure_redirects = 1
net.ipv4.conf.tap1.shared_media = 1
net.ipv4.conf.tap1.rp_filter = 1
net.ipv4.conf.tap1.send_redirects = 1
net.ipv4.conf.tap1.accept_source_route = 1
net.ipv4.conf.tap1.accept_local = 0
net.ipv4.conf.tap1.src_valid_mark = 0
net.ipv4.conf.tap1.proxy_arp = 0
net.ipv4.conf.tap1.medium_id = 0
net.ipv4.conf.tap1.bootp_relay = 0
net.ipv4.conf.tap1.log_martians = 0
net.ipv4.conf.tap1.tag = 0
net.ipv4.conf.tap1.arp_filter = 0
net.ipv4.conf.tap1.arp_announce = 0
net.ipv4.conf.tap1.arp_ignore = 0
net.ipv4.conf.tap1.arp_accept = 0
net.ipv4.conf.tap1.arp_notify = 0
net.ipv4.conf.tap1.proxy_arp_pvlan = 0
net.ipv4.conf.tap1.disable_xfrm = 0
net.ipv4.conf.tap1.disable_policy = 0
net.ipv4.conf.tap1.force_igmp_version = 0
net.ipv4.conf.tap1.promote_secondaries = 0
dragon ~ # sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
dragon ~ # lsmod
Module Size Used by
iptable_filter 984 0
ip_tables 7681 1 iptable_filter
x_tables 9464 2 iptable_filter,ip_tables
kvm_intel 107912 3
kvm 243019 1 kvm_intel
tun 9802 5
bridge 52516 0
ipv6 205677 17 bridge
stp 1051 1 bridge
llc 2538 2 bridge,stp
r8169 34883 0
mptsas 27327 1
8250_pnp 3391 0
mptscsih 14369 1 mptsas
i7core_edac 12552 0
i2c_i801 6174 0
edac_core 24556 3 i7core_edac
mptbase 44384 2 mptsas,mptscsih
serio_raw 2913 0
mii 2823 1 r8169
ac 2373 0
e1000e 107805 0
8250 15537 1 8250_pnp
serial_core 12029 1 8250
dragon ~ # netstat -rn
Kernel IP Routentabelle
Ziel Router Genmask Flags MSS Fenster irtt Iface
0.0.0.0 172.16.20.5 0.0.0.0 UG 0 0 0 br0
127.0.0.0 - 255.0.0.0 ! - - - -
172.16.0.0 0.0.0.0 255.255.224.0 U 0 0 0 br0
|
As you can see from the arp-Tables, arp seems to work
| Code: |
dragon ~ # arp -a
celeborn (172.16.1.2) auf 00:19:99:7d:fa:78 [ether] auf br0
elrond (172.16.1.7) auf 00:00:00:00:00:03 [ether] auf br0
celeborn# arp -a|grep 172.16.1.7
? (172.16.1.7) at 00:00:00:00:00:03 on bge0 expires in 1146 seconds [ethernet]
celeborn# arp -a | grep 172.16.1.4
dragon (172.16.1.4) at 00:00:00:00:00:02 on bge0 expires in 1087 seconds [ethernet]
elrond# arp -a
? (172.16.1.4) at 00:00:00:00:00:02 on re0 expires in 907 seconds [ethernet]
? (172.16.1.7) at 00:00:00:00:00:03 on re0 permanent [ethernet]
? (172.16.1.2) at 00:19:99:7d:fa:78 on re0 expires in 1115 seconds [ethernet]
|
The VM is attached to tap1. tap0 isn't in use at this point. Interestingly br0 has the same MAC as tap0.
The guest is started with
| Code: |
qemu-kvm -hda /usr/local/util/freebsd.img -hdb /dev/sdb \
-net nic,macaddr=00:00:00:00:00:03 -net tap,ifname=tap1,script=no,downscript=no \
-cdrom /dev/sr0 -curses -k de 2
|
I have tried fiddeling with proxy_arp and forwarding to no avail. As far as I understand at least forwarding should not matter here, as arp shows the tap Interfaces outside the host-Machine (on celeborn).
Iptables isn't involved either:
| Code: |
dragon ~ # iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
|
So what am I missing here?
Last edited by AgBr on Wed Jun 06, 2012 8:46 am; edited 1 time in total |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21619
|
| Posted: Fri Jun 01, 2012 10:39 pm Post subject: |
|
|
| Please post a command that fails to connect in the way you want, and the full output generated by that command. Also post the output of /sbin/ip route or the equivalent for both endpoints of the attempted connection. |
|
| Back to top |
|
|
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
| Posted: Fri Jun 01, 2012 11:25 pm Post subject: |
|
|
| Hu wrote: | | Please post a command that fails to connect in the way you want, and the full output generated by that command. Also post the output of /sbin/ip route or the equivalent for both endpoints of the attempted connection. |
| Code: |
user@gandalf ~ $ ssh root@172.16.1.7
ssh: connect to host 172.16.1.7 port 22: Connection timed out
user@gandalf ~ $ ping 172.16.1.7
PING 172.16.1.7 (172.16.1.7) 56(84) bytes of data.
Ping doesn't return anything ^C after quite some while.
user@gandalf ~ $ netstat -rn
Kernel IP Routentabelle
Ziel Router Genmask Flags MSS Fenster irtt Iface
172.16.0.0 0.0.0.0 255.255.224.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 172.16.20.5 0.0.0.0 UG 0 0 0 eth0
user@gandalf ~ $ /sbin/arp -a
dragon (172.16.1.4) auf 00:00:00:00:00:02 [ether] auf eth0
celeborn (172.16.1.2) auf 00:19:99:7d:fa:78 [ether] auf eth0
? (172.16.1.7) auf 00:00:00:00:00:03 [ether] auf eth0
ulli@gandalf ~ $
|
Logged into the guest (elrond) via the host (dragon) and ping to gandalf (172.16.20.9)
| Code: |
elrond# ping 172.16.20.9
PING 172.16.20.9 (172.16.20.9): 56 data bytes
^C
--- 172.16.20.9 ping statistics ---
60 packets transmitted, 0 packets received, 100.0% packet loss
elrond# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 172.16.20.5 UGS 0 0 re0
127.0.0.1 link#3 UH 0 25 lo0
172.16.0.0/19 link#1 U 0 1385 re0
172.16.1.7 link#1 UHS 0 0 lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 ::1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%re0/64 link#1 U re0
fe80::200:ff:fe00:3%re0 link#1 UHS lo0
fe80::%lo0/64 link#3 U lo0
fe80::1%lo0 link#3 UHS lo0
ff01::%re0/32 fe80::200:ff:fe00:3%re0 U re0
ff01::%lo0/32 ::1 U lo0
ff02::/16 ::1 UGRS lo0
ff02::%re0/32 fe80::200:ff:fe00:3%re0 U re0
ff02::%lo0/32 ::1 U lo0
elrond# arp -a
? (172.16.20.9) at 00:30:05:40:68:aa on re0 expires in 1042 seconds [ethernet]
? (172.16.1.4) at 00:00:00:00:00:02 on re0 expires in 1061 seconds [ethernet]
? (172.16.1.7) at 00:00:00:00:00:03 on re0 permanent [ethernet]
? (172.16.1.2) at 00:19:99:7d:fa:78 on re0 expires in 1141 seconds [ethernet]
|
Arp-Tables presented again for your convenience. As you can see, arp does work. If arp works, the theory is, that layer 2 must work. Otherwise the machines would not have any means to know the mac-addresses for the target-interfaces of the respective IP-addresses. If layer 2 works, on the local net IP-traffic should work too as there is no router involved. I can log into the guest (172.16.1.7/19) from the host (172.16.1.4) but not from any other machine on the same subnet (172.16.20.9/19 or 172.16.1.2/19) for instance) |
|
| Back to top |
|
|
AngelKnight
Tux's lil' helper
Joined: 14 Jan 2003
Posts: 127
|
| Posted: Mon Jun 04, 2012 2:04 pm Post subject: |
|
|
You didn't mention whether or not you turned forwarding on on the host, with
| Code: | | sysctl net.ipv4.ip_forward=1 |
|
|
| Back to top |
|
|
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
| Posted: Mon Jun 04, 2012 3:06 pm Post subject: |
|
|
| AngelKnight wrote: | You didn't mention whether or not you turned forwarding on on the host, with
| Code: | | sysctl net.ipv4.ip_forward=1 |
|
I have tried this too despite that it should not be necessary. It did not make a difference. |
|
| Back to top |
|
|
AngelKnight
Tux's lil' helper
Joined: 14 Jan 2003
Posts: 127
|
| Posted: Tue Jun 05, 2012 6:08 pm Post subject: |
|
|
Argh, yes you're correct: there's no L3 forwarding here. Sorry, I thought I'd read more carefully.
If I read this correctly, you set the host's tap1 to be a forced MAC (0000.0000.0003) -and- set the inside guest to use the same forced MAC.
If you did, don't: let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside. I've already tested this; when the host TAP has the same MAC as the guest's simulated ethernet, the host TAP will tend to eat the frames, not pass them through. Don't worry about collisions too much: the autoassigned address has most-sig-byte's bit 1 set (marking it local-admin).
I vaguely remember being bit by this but evidently misremembered why earlier, sorry about that. |
|
| Back to top |
|
|
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
| Posted: Tue Jun 05, 2012 6:51 pm Post subject: |
|
|
| AngelKnight wrote: | Argh, yes you're correct: there's no L3 forwarding here. Sorry, I thought I'd read more carefully.
If I read this correctly, you set the host's tap1 to be a forced MAC (0000.0000.0003) -and- set the inside guest to use the same forced MAC.
If you did, don't: let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside. I've already tested this; when the host TAP has the same MAC as the guest's simulated ethernet, the host TAP will tend to eat the frames, not pass them through. Don't worry about collisions too much: the autoassigned address has most-sig-byte's bit 1 set (marking it local-admin).
I vaguely remember being bit by this but evidently misremembered why earlier, sorry about that. |
I was following the above mentioned instructions here but I will try that. Meanwhile I have found out by contemplating some tcpdump output, that all broadcast frames go in through the tap interface and all direct frames from the guest and all broadcasts go out through the tap but not through the bridge-Interface. So all frames having specific MAC-Addresses coming from the tap are blocked by the bridge interface.
The box in question is remote. I hope fiddeling with the tap interface will not kill the bridge. Otherwise I will have to wait until tomorrow. |
|
| Back to top |
|
|
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
| Posted: Wed Jun 06, 2012 8:45 am Post subject: |
|
|
| AngelKnight wrote: |
... let the host's tap interface float and use qemu-kvm commandline to force the MAC on the adapter inside. |
The problem is solved this way. Thank you. The wiki seems to be wrong in this respect. |
|
| Back to top |
|
|
AngelKnight
Tux's lil' helper
Joined: 14 Jan 2003
Posts: 127
|
| Posted: Wed Jun 06, 2012 11:01 pm Post subject: |
|
|
This wiki specifically says to make the host machine the defaultrouter in the guest, not the actual network's bridge. This is probably why.
Trying to use the real network segment's defaultrouter will not work for this configuration as you have discovered.
This wiki needs quite a bit of fixing around networking actually. Ugh. I'm too drunk to fix this now. Anyone?  |
|
| Back to top |
|
|
AgBr
Apprentice
Joined: 06 Nov 2010
Posts: 195
|
| Posted: Thu Jun 07, 2012 6:20 am Post subject: |
|
|
| AngelKnight wrote: | This wiki specifically says to make the host machine the defaultrouter in the guest, not the actual network's bridge. This is probably why.
Trying to use the real network segment's defaultrouter will not work for this configuration as you have discovered.
This wiki needs quite a bit of fixing around networking actually. Ugh. I'm too drunk to fix this now. Anyone? |
I am shy to do it as my knowledge about these things is too limited as you know. But I have made a comment about my experiences in the talk section already with reference to this thread. |
|
| Back to top |
|
|
|
Networking & Security
