Kernel panic with macvtap/macvlan

[BradN]

Hi all, I'm running kernel 3.9.0 with a qemu-kvm/libvirt virtual machine inside communicating to the network via macvtap. I can run as much traffic as I like within the host (ie, between host's macvlan and guest's macvtap) without problems.

As soon as I try doing any traffic external to the machine from within the guest (all it takes is a remote ssh shell and a couple minutes), the host kernel panics.

Update: Problem also occurs with gentoo-sources-3.9.1-r1
Update: Problem does NOT appear to occur on 3.8.5 (or is much harder to trigger?)


Panic screenshot: http://imgur.com/TsDAAMF

Anyone run into this issue?

Keywords for search engines to find this:

Code: Select all

BUG: unable to handle kernel paging request at
dev_queue_xmit
skb_copy_datagram_from_iovec
macvlan_start_xmit macvlan
macvtap_get_user macvtap
macvtap_sendmsg macvtap
handle_tx vhost_net
vhost_worker vhost_net
vhost_atach_cgroups_work vhost_net
kthread
mm_update_next_owner
kthread_freezable_should_stop
ret_from_fork

[_______0]

Do you absolutely need vhost? It appears to some sort of fancy bling for networking. In other words, forgoing vhost wont deprive you from proper and reliable networking. From the site:

http://www.linux-kvm.org/page/VhostNet

VhostNet provides better latency (10% less than e1000 on my system) and greater throughput (8x the normal virtio, around 7~8 Gigabits/sec here) for network.

And the dude's math is off. latency and throughput are not related and 10% is not 8x more!!! lololol!! Besides he doesn't mention for which situation is vhost with a 8GB/s ideal. Hardrive rw is about 60mb/s, ssd about 200mb/s, gig ethernet cards 1000mb/s. So where on earth is the advantage of 8GB/s transfer rate?? RAM to RAM??

Related to your crash there's small-print link at the end of that page. Read it carefully and see if it applies to your app.

Vhost is designed for ultra-low latency, but what's the point if it crashes your HOST?

I'd do this:

a) evaluate whether vhost is worth in your situation.
b) don't use vhost
c) if it's a case of ultra-low latency need get a dedicated hardware adapter and use passthrough.

Just curious, how do you set up macvlan/macvtap? How are you able to communicate host/guest with macvlan/macvtap? I am looking for a solution to share/communicate host/guest data.

thanks.

[BradN]

Hmm, I wasn't aware I was using vhost but apparently virt-manager or libvirt turns it on without giving an option. I just basically turned on anything in the kernel config that looked like it might be useful, i'm pretty new to virtualization.

Also vhost is just a speed enhancement for the virtio network device. It's entirely possible to have 10% lower latency but 8x throughput, but that's usually the kind of thing that happens from implementing zero-copy, not a better interrupt mechanism. I kinda doubt it too. On that note, I don't get why he compares it to e1000, apples and oranges, he should be comparing virtio with and without vhost.

At any rate, I don't think that caveat wrt DHCP is related to crashes, as it makes it through dhcp fine and nothing else is talking on that port. I will investigate though as anything that's involved with packet transmission may be an issue though.

Here is my conf.d/net on the host which allows communication between host and guest (note that BOTH eth0 and macvlan network devices must be turned on via rc-update, and note that macvlan0 becomes the network device for the host to use):

Code: Select all

config_eth0="null" #use macvlan instead

macvlan_macvlan0="eth0" #for libvirtd/qemu guests to be able to talk to host
mode_macvlan0="bridge"  #

Guest uses blank conf.d/net.

How does this allow host/guest to communicate? Simple, macvtap (uses macvlan internally) only allows packets to go out to the physical hardware or get redirected to the proper recipient when coming in - there's no mechanism to redirect a packet going out to one coming in, at least when you're dealing with the host using the actual eth0 device. Once it's gone that far it has to go out the cat5. But, the limitation doesn't exist between macvlan's using the same underlying hardware - it can bounce packets between them before taking them to eth0. So the answer is have the host use macvlan, guests use macvtap devices (which see themselves as peers and act like a bridge), and it works. The caveat i've seen is that NetworkManager doesn't understand anything about this (on the host) but screw networkmanager anyway.

[TomWij]

Please file kernel bugs on http://bugs.gentoo.org so we can keep track of them in one place and attempt to resolve them.

On a side note, I just pushed 3.9.2 to the Portage tree.

[BradN]

Thank you for monitoring the forums, I plan on filing a bug report when I finish my bisection attempt (it's irritating when it desync's my raid array and I have to wait for it to rebuild out of fear of really screwing something up otherwise).

[BradN]

Please file kernel bugs on http://bugs.gentoo.org so we can keep track of them in one place and attempt to resolve them.

On a side note, I just pushed 3.9.2 to the Portage tree.

Bug reported: https://bugs.gentoo.org/show_bug.cgi?id=470640

3.9.2 is still affected.