Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] QEMU ipv6 traffic blocked
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Sun Feb 16, 2020 4:30 pm    Post subject: [SOLVED] QEMU ipv6 traffic blocked Reply with quote

TLDR: if you go to my last post you will see that this was caused (in my case) by the lack of settings in /etc/qemu/bridge.conf. /TLDR
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

At least I think that's what is happening - QEMU has an almost vertical learning curve :-)

I have a gentoo VM that has been running under Virtualbox for many years. I now wish to migrate it to run under KVM/QEMU. I'm really close - only issue is that ipv6 network traffic appears to being blocked by something.

Details. My libvirt XM for the network parts of the VM is as follows:
Code:
<interface type="bridge">
  <mac address="08:00:27:93:2d:e5"/>
  <source bridge="br0"/>
  <target dev="vnet0"/>
  <model type="e1000e"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>


When the VM is run, this gets translated into the /usr/bin/qemu-system-x86_64 command line as follows:
Code:
-netdev tap,fd=30,id=hostnet0 \
-device e1000e,netdev=hostnet0,id=net0,mac=08:00:27:93:2d:e5,bus=pci.1,addr=0x0

(There is a ton of other stuff on the command line of course, left out to focus on networking.)

On the host (also a gentoo system), the bridge looks like this:
Code:
ian2 ~ # brctl show
bridge name   bridge id      STP enabled   interfaces
br0      8000.1c6f65d226f5   yes      eth0
                     vnet0
br1      8000.0050b6083341   no      eth1
ian2 ~ #


In the guest VM, the network looks like this:
Code:
dns2 ~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:93:2d:e5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.32/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:8b0:fb5e:0:289e:16ed:3ff:e4fd/64 scope global temporary dynamic
       valid_lft 600995sec preferred_lft 82197sec
    inet6 2001:8b0:fb5e:0:a00:27ff:fe93:2de5/64 scope global dynamic mngtmpaddr
       valid_lft forever preferred_lft forever
    inet6 2001:8b0:fb5e::32/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe93:2de5/64 scope link
       valid_lft forever preferred_lft forever
3: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
5: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
8: ip6_vti0@NONE: <NOARP> mtu 1428 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
9: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
10: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
dns2 ~ #


From the host, I can ping the guest in both ipv4 and ipv6:
Code:

ipic@ian2 ~/bin $ ping -4 dns2
PING dns2.pickworth.me.uk (192.168.1.32) 56(84) bytes of data.
64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=1 ttl=64 time=0.396 ms
64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=2 ttl=64 time=0.715 ms
64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=3 ttl=64 time=0.832 ms
^C
--- dns2.pickworth.me.uk ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 11ms
rtt min/avg/max/mdev = 0.396/0.647/0.832/0.186 ms
ipic@ian2 ~/bin $ ping -6 dns2
PING dns2(dns2.pickworth.me.uk (2001:8b0:fb5e::32)) 56 data bytes
64 bytes from dns2.pickworth.me.uk (2001:8b0:fb5e::32): icmp_seq=1 ttl=64 time=0.786 ms
64 bytes from dns2.pickworth.me.uk (2001:8b0:fb5e::32): icmp_seq=2 ttl=64 time=0.742 ms
^C
--- dns2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 0.742/0.764/0.786/0.022 ms
ipic@ian2 ~/bin $


However, a connection to ipv6 ssh hangs, whilst an ipv4 sshd connection works:
Code:

pic@ian2 ~/bin $ ssh -Y -t -4 -p 4899 root@dns2
X11 forwarding request failed on channel 0
dns2 ~ # hostname
dns2
dns2 ~ # logout
Connection to dns2 closed.
ipic@ian2 ~/bin $ ssh -Y -t -6 -p 4899 root@dns2
^C
ipic@ian2 ~/bin $


I have spent the day trying to find documentation on this, but failed. Only things I can find involve setting up the QEMU shared network, which isn't want I want to do.

I can see that there is a nwfilter rule for ipv4 in virsh:
Code:
virsh # nwfilter-list
 UUID                                   Name
-----------------------------------------------------------------
 17f8df51-4f61-499e-9d88-fc9177c7b834   allow-arp
 fdb7d1a9-3b1a-43d5-afd3-1515f0228039   allow-dhcp
 88bc7ab0-9006-42da-9561-9d5d2c1410de   allow-dhcp-server
 4a461fb8-1703-4fef-9953-098002f3a8c4   allow-incoming-ipv4
 86d9e00c-1a44-4fc8-926c-2ee54b489713   allow-ipv4
 76630340-576a-43e8-b716-b833d94679c2   clean-traffic
 8094dec2-4f63-45d0-8c8d-b74b7eefec5c   clean-traffic-gateway
 4010010e-be0d-450d-b1e4-e5e5f8cc7076   no-arp-ip-spoofing
 e3e8319a-9336-4427-b179-b505522b84ae   no-arp-mac-spoofing
 4cc1ebfb-2640-48c4-a5f7-5ec4ef7653ef   no-arp-spoofing
 04908ecd-c0b4-4aeb-aebb-75c77471eded   no-ip-multicast
 ad6a1c85-ccf8-4f1f-8bf7-cc830cac54a9   no-ip-spoofing
 98d4c7a9-9e3c-4047-ac6f-182cedcc4aa6   no-mac-broadcast
 1d90c2d9-b02d-4561-91c6-6b209520548c   no-mac-spoofing
 001a8f78-cc2b-4892-8e13-f8b7ef1ada58   no-other-l2-traffic
 213ce201-07f1-4428-9a8e-759dc68b7e59   no-other-rarp-traffic
 1dc26217-d085-41c6-8e30-f9e33f4dba42   qemu-announce-self
 9d956941-7e32-478c-8964-afbb0b899de3   qemu-announce-self-rarp

virsh #


..but I don't know if this is being applied.

I can also see in QEMU documentation that there is a ipv6=on|off option, but it states "If neither is specified both protocols are enabled".

So I am at a bit of a dead end on this. Any help appreciated.
Thanks


(PS: Sorry for half cocked post half way through - pressed "submit" rather than "preview" Doh.)


Last edited by ipic on Mon Feb 17, 2020 6:05 pm; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Sun Feb 16, 2020 5:27 pm    Post subject: Reply with quote

  1. Code:
    However, a connection to ipv6 ssh hangs, whilst an ipv4 sshd connection works:

    Does you SSH daemon listen on an IPv6 socket? Or does it listen on a IPv4 socket only? Look at the output of 'netstat -l -n' and check your sshd config file (ListenAddress option).

  2. run 'tcpdump -v -n -i eth0' in your VM and look at the packets. Do you see the expected packets when your run ping -4, ping -6, ssh -4, ssh -6 from your host?
Back to top
View user's profile Send private message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Sun Feb 16, 2020 6:56 pm    Post subject: Reply with quote

I am using the same disk image (it's an LVM partition) under both Virtualbox and QEUM. On Virtualbox it works OK, so I anticipated seeing the same output for netstat, which I did:
Code:

dns2 ~ # netstat -l -n
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 192.168.1.32:53         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:953             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:4899            0.0.0.0:*               LISTEN     
tcp6       0      0 ::1:53                  :::*                    LISTEN     
tcp6       0      0 2001:8b0:fb5e:0:1c00:53 :::*                    LISTEN     
tcp6       0      0 2001:8b0:fb5e:0:a00::53 :::*                    LISTEN     
tcp6       0      0 2001:8b0:fb5e::32:53    :::*                    LISTEN     
tcp6       0      0 :::4899                 :::*                    LISTEN     
udp        0      0 192.168.1.32:53         0.0.0.0:*                         
udp        0      0 127.0.0.1:53            0.0.0.0:*                         
udp6       0      0 ::1:53                  :::*                               
udp6       0      0 2001:8b0:fb5e:0:1c00:53 :::*                               
udp6       0      0 2001:8b0:fb5e:0:a00::53 :::*                               
udp6       0      0 2001:8b0:fb5e::32:53    :::*                               
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     SEQPACKET  LISTENING     4926     /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     5760     /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     6617     /run/syslog-ng.ctl
dns2 ~ #


The routes look good as well:
Code:

dns2 ~ # ip route
default via 192.168.1.5 dev eth0 metric 2
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.32
dns2 ~ #
dns2 ~ # ip -6 route
2001:8b0:fb5e::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via 2001:8b0:fb5e::2 dev eth0 metric 2 pref medium
default via fe80::a2e4:cbff:fe26:5d0b dev eth0 proto ra metric 1024 expires 29sec hoplimit 64 pref medium
dns2 ~ #


The sshd config is as follows (snippet with relevant bits):
Code:

#Port 224
Port 4899

#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::


Commented lines mean they are the defaults, as explained a bit higher up:
Code:

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.


During a ping, I can see the following traffic: (leaving out lots of "spurious" bits)
Code:

....

18:13:42.526977 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 1
18:13:42.527020 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 1
......
18:13:43.527562 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 2
18:13:43.527632 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 2
.....
18:13:44.554650 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 3
18:13:44.554683 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 3
...


I think that means that icmp6 is making the round trip OK.
Getting on to the attempted ipv6 ssh connection I see this:
Code:

18:14:11.497785 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96
   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms
     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity
     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021
     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b
18:14:14.314389 STP 802.1d, Config, Flags [none], bridge-id 8000.1c:6f:65:d2:26:f5.8002, length 43
   message-age 0.00s, max-age 20.00s, hello-time 10.00s, forwarding-delay 2.00s
   root-id 8000.1c:6f:65:d2:26:f5, root-pathcost 0
18:14:14.745695 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96
   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms
     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity
     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021
     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b
18:14:15.616542 IP6 (flowlabel 0x5a0e3, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5904132 ecr 0,nop,wscale 7], length 0
18:14:15.616667 IP6 (flowlabel 0x7d490, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824091023 ecr 5904132,nop,wscale 7], length 0
18:14:16.618558 IP6 (flowlabel 0x8b1c8, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x110d (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5905134 ecr 0,nop,wscale 7], length 0
18:14:16.618613 IP6 (flowlabel 0xe2fd2, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd5ad), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824092025 ecr 5904132,nop,wscale 7], length 0
18:14:17.621885 IP6 (flowlabel 0xc5943, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd1c1), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824093029 ecr 5904132,nop,wscale 7], length 0
18:14:18.666553 IP6 (flowlabel 0x41ef2, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x090d (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5907182 ecr 0,nop,wscale 7], length 0
18:14:18.666616 IP6 (flowlabel 0x89bfb, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xcdad), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824094073 ecr 5904132,nop,wscale 7], length 0
18:14:19.331646 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96
   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms
     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity
     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021
     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b
18:14:20.693952 IP6 (flowlabel 0x96918, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xc5c1), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824096101 ecr 5904132,nop,wscale 7], length 0
18:14:22.698550 IP6 (flowlabel 0xf2c0d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xf94c (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5911214 ecr 0,nop,wscale 7], length 0
18:14:22.698619 IP6 (flowlabel 0x1ac16, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xbded), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824098105 ecr 5904132,nop,wscale 7], length 0
18:14:24.474280 IP (tos 0x0, ttl 64, id 4366, offset 0, flags [DF], proto UDP (17), length 159)
    192.168.1.11.17500 > 255.255.255.255.17500: UDP, length 131
18:14:24.475146 IP (tos 0x0, ttl 64, id 21757, offset 0, flags [DF], proto UDP (17), length 159)
    192.168.1.11.17500 > 192.168.1.255.17500: UDP, length 131
18:14:24.554552 STP 802.1d, Config, Flags [none], bridge-id 8000.1c:6f:65:d2:26:f5.8002, length 43
   message-age 0.00s, max-age 20.00s, hello-time 10.00s, forwarding-delay 2.00s
   root-id 8000.1c:6f:65:d2:26:f5, root-pathcost 0
18:14:26.709902 IP6 (flowlabel 0x58102, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xae41), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824102117 ecr 5904132,nop,wscale 7], length 0


I don't know what to make of this. It seems that the incoming connect requests are arriving from 2001:8b0:fb5e:0:f91b:1d04 but I can't make out what 2001:8b0:fb5e::32 is responding with. Its not what the host was expecting, hence a seemingly endless dialogue.

This is reminding me of https://xkcd.com/2259/ :cry:

My make.conf has ipv6 set, so everything that has that use option is built with it. I'm still struggling with exactly the same disk image doing something different when run under QEMU.

Stumped :-(
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Sun Feb 16, 2020 9:19 pm    Post subject: Reply with quote

The output shows that the SSH daemon listens on IPv4 as well as on IPv6 (good!):
Code:
tcp        0      0 0.0.0.0:4899            0.0.0.0:*               LISTEN     
tcp6       0      0 :::4899                 :::*                    LISTEN     

The TCP dump shows that the 3 way TCP handshake doesn't work. It shows only the first 2 packets. The third packet (ACK) is missing:
Code:
18:14:15.616542 IP6 (flowlabel 0x5a0e3, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5904132 ecr 0,nop,wscale 7], length 0
18:14:15.616667 IP6 (flowlabel 0x7d490, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824091023 ecr 5904132,nop,wscale 7], length 0

There are several possible reasons for this:
  1. the answer packet from the guest to the host (packet 2, SYN/ACK) doesn't reach the host
  2. The third packet from the host to the guest (ACK) doesn't reach the guest
  3. Something is wrong with the contents of the packets
  4. Something is wrong with the SSH software on the host or on the guest.

It's likely that (1) will help us. The IPv6 default route on your guest is:
Code:
default via 2001:8b0:fb5e::2 dev eth0 metric 2 pref medium

What is '2001:8b0:fb5e::2'? Does it exist and does it act as a router? Does it route the answer packets from the guest to your host (2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3)?


Last edited by mike155 on Sun Feb 16, 2020 9:31 pm; edited 1 time in total
Back to top
View user's profile Send private message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Sun Feb 16, 2020 9:24 pm    Post subject: Reply with quote

I did a trace on the guest when it is running under Virtualbox, when I connected to ssh using ipv6:
Code:

21:11:07.991306 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x43ee (correct), seq 3457914931, win 64800, options [mss 1440,sackOK,TS val 16516659 ecr 0,nop,wscale 7], length 0
21:11:07.991376 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3
     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5
21:11:07.991766 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]
     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5
21:11:07.991783 IP6 (flowlabel 0xd8451, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250: Flags [S.], cksum 0xbdb0 (correct), seq 331762219, ack 3457914932, win 64260, options [mss 1440,sackOK,TS val 3325977624 ecr 16516659,nop,wscale 7], length 0
21:11:07.992163 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [.], cksum 0xe571 (correct), ack 1, win 507, options [nop,nop,TS val 16516660 ecr 3325977624], length 0
21:11:07.993087 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 53) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [P.], cksum 0x24ac (correct), seq 1:22, ack 1, win 507, options [nop,nop,TS val 16516660 ecr 3325977624], length 21
21:11:07.993124 IP6 (flowlabel 0xd8451, hlim 64, next-header TCP (6) payload length: 32) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250: Flags [.], cksum 0xe55f (correct), ack 22, win 502, options [nop,nop,TS val 3325977626 ecr 16516660], length 0
.....


Now some of the trace in the previous post makes a little bit of sense, the key bit being:
Code:

....2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct).....
followed by
.....2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997)....


A little knowledge is a dangerous thing, but I'll take a stab. It seems that the initial challenge from the host '[S]' is met with a response saying "I don't agree with your checksum".

OK. Not sure I know what to do with that.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Sun Feb 16, 2020 9:32 pm    Post subject: Reply with quote

ipic, forget about the checksum. tcpdump and Wireshark often show invalid checksums, for various reasons.

I added a paragraph to my last post. Please look at it.

EDIT: I was probably wrong. Guest and host are on the same /64 subnet. There should be no router involved.

Please repeat the SSH test and look at traffic on your host, preferably on the bridge. Do you see the second packet (SYN/ACK) on the bridge? Or does it get lost in the VM software between the guest and the host?


Last edited by mike155 on Sun Feb 16, 2020 9:51 pm; edited 2 times in total
Back to top
View user's profile Send private message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Sun Feb 16, 2020 9:43 pm    Post subject: Reply with quote

'2001:8b0:fb5e::2' is my internet router, i.e.the gateway to the outside world.

All my machines are configured to use it as a gateway, and it can reach all the machines on the local network.

On the host, for example, the routes are thus:
Code:

ian2 ~ # ip -6 route
2001:8b0:fb5e::/64 dev br0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev br0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev br1 proto kernel metric 256 pref medium
default via 2001:8b0:fb5e::2 dev br0 metric 4 pref medium
default via fe80::a2e4:cbff:fe26:5d0b dev br0 proto ra metric 1024 expires 20sec hoplimit 64 pref medium
ian2 ~ #


I think the fact that the icmp6 ping goes back and forth OK shows basic routing is OK?
And it works when running under Virtualbox.

Actually, looking at the two route outputs, they have both picked up the link level address for the router: ''fe80::a2e4:cbff:fe26:5d0b". Is it a better approach to let ipv6 work out the gateway for itself?
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Sun Feb 16, 2020 9:50 pm    Post subject: Reply with quote

OK, good. I was probably wrong. Guest and host are on the same /64 subnet. There should be no router involved. My fault. But it's good to know that the router works!

Let's try to find out where the second packet (SYN/ACK) gets lost. Please repeat the SSH test and look at traffic on your host, preferably on the bridge (try all interfaces). Do you see the second packet (SYN/ACK) on the bridge? Or does it get lost in the VM software between the guest and the host?
Back to top
View user's profile Send private message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Sun Feb 16, 2020 10:57 pm    Post subject: Reply with quote

I took tcpdump traces in the guest, on the host vnet0 and on the host br0. In all three I could see the same packets:

Guest eth0:
Code:

22:23:29.958342 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfa4e (correct), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0
22:23:29.958408 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3
     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5
22:23:29.958708 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]
     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5
22:23:29.958722 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0xfcdf (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0


Host vnet0:
Code:

22:23:30.105910 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfa4e (correct), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0
22:23:30.106308 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3
     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5
22:23:30.106371 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]
     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5
22:23:30.106626 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0x7607 (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0


Host br0:
Code:

22:23:30.105901 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfcdf (incorrect -> 0xfa4e), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0
22:23:30.106308 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3
     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5
22:23:30.106367 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]
     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5
22:23:30.106626 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0x7607 (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0


I have the full files - there is a lot of traffic on br0. above is edited down to show the trace through the interfaces.
Following this, the pattern is repeated over and over as before.

What this all has done (and thank you very much for helping me with this), is make me see that the networking approach between QEMU and Virtualbox is different.
- In Virtualbox the guest connects direct to the bridge (br0 in my case) which is placed in promiscuous mode.
- In QEMU the guest inserts a virtual NIC into the bridge.

So, it is possible that something in ipv6 land doesn't like this setup. The frustrating thing is that it works with ipv4 :-(

I am going to try some different options on the guest NIC setup, to see if any make a difference.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Sun Feb 16, 2020 11:47 pm    Post subject: Reply with quote

Quote:
I have the full files - there is a lot of traffic on br0

You could add 'port 4899' to the tcpdump statement
Code:
tcpdump -v -i <interface> -n port 4899

This will show only TCP and UDP packets to and from port 4899. Beware that it won't show ICMP (ping) packets. Look at 'man tcdump' and 'man pcap-filters' for details.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2179
Location: Frankfurt, Germany

PostPosted: Mon Feb 17, 2020 12:05 am    Post subject: Reply with quote

Quote:
In all three I could see the same packets:
  1. So the SYN/ACK packets reach the bridge on the host. Great!

  2. Please show us the output of 'ip addr' on your host.

  3. I would like to see the MAC addresses. Please run the SSH test again. Start tcpdump on your host with option '-e':
    Code:
    tcpdump -v -e -i br0 -n port 4899

Back to top
View user's profile Send private message
ipic
Guru
Guru


Joined: 29 Dec 2003
Posts: 303
Location: UK

PostPosted: Mon Feb 17, 2020 6:01 pm    Post subject: Reply with quote

First, I want to say thank you, mike155, for all your help in this. By guiding me through the fault finding, you lead me to the root cause. Reminded me of my old days in operations!

The problem was permissions in the end. I had started to sniff this at the end of my last post, since besides the different approaches to networking, virtualbox and qemu ran under different users.

Completely by accident I stumbled across this file: /etc/qemu/bridge.conf
Since I am using libvirtd as the manager, it didn't occur to me to look under a qemu folder for settiings (for example, the conf file is here: /etc/libvirt/qemu.conf).
So full marks to libvert/qemu for being inconsistent and obtuse!

Inside the bridge.conf file is this (after I edited it):
Code:

ian2 ~ # cat /etc/qemu/bridge.conf
# This should have the following permissions: root:qemu 0640

# IRP Changes #
allow br0
allow br1
allow virbr0

# allow br0
# Uncommenting the above would allow users in the 'qemu' group
# to add devices to 'br0'

# allow virbr0
# Uncommenting the above would allow users in the 'qemu' group
# to add devices to 'virbr0'

# include /etc/qemu/bob.conf
# Uncommenting the above would allow users in the 'bob' group
# to have permissions defined in it, iff it has the following
# permissions: root:bob 0640
ian2 ~ #


With the permissions set as described, and the entries uncommitted (well added), I started up the VM, and the ipv6 ssh session just connected straight up - like it should.

phew.

I suppose a good question would be - why only ipv6 with the problem, surely permissions would stop ipv4 as well??? At this stage I can't be arsed.

And again many thanks again for the patient help.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum