[SOLVED] Multicast & mythtv tuning issues with kernel>2.6.30

[Phoenix84]

I've been using Gentoo for many years, but I recently set up a new machine as a mythtv box. I originally set it up with kernel 2.6.29. I have an Hauppauge PVR 1600 (which I don't use anymore), and I have second NIC for my IPTV service. It seems something changed in the kernel configuration between 2.6.30 and 2.6.31, because I can no longer tune ANY capture card in mythtv after upgrading.
I originally saw this issue when 2.6.31 first came out, but though it was a regression or some odd configuration with my capture card, so I ignored it. However now I have my IPTV service (using multicast) and the EXACT same issue appears as the dvb card. I've been searching and fighting this issue for almost 6 months, and have gotten nowhere. This has also happens with kernel .32, .33, .34 and mythtv .21, .22, .23.

The steps I take to upgrade my kernel are as follows:
copy .config from currently running kernel into new directory
run make menuconfig (I've also tried running make oldconfig first)
verify various drivers are still available (multimedia tree changed between those version, so I have to reselect my V4L drivers for my PVR1600).
Rebuild and reboot

About the multicasting:
I successfully followed the mythtv wiki entry for the IPTV service I use, and it works perfectly under 2.6.30. I can test by using mplayer on the multicast address and can watch the stream from there. However when I go ANY kernel > 2.6.30, I get the following error:

Code: Select all

MPlayer SVN-r29796-4.3.4 (C) 2000-2009 MPlayer Team

Playing udp://225.1.100.1:2001.
STREAM_UDP, URL: udp://225.1.100.1:2001
Timeout! No data from host 225.1.100.1
udp_streaming_start failed
No stream found to handle url udp://225.1.100.1:2001


Exiting... (End of file)

At this point, it doesn't seem to be a mythtv issue anymore. I have verified that the multicast options are set correctly:

Code: Select all

dragon ~ # cat /proc/sys/net/ipv4/conf/all/rp_filter 
0
dragon ~ # cat /proc/sys/net/ipv4/conf/default/rp_filter 
0
dragon ~ # cat /proc/sys/net/ipv4/conf/all/force_igmp_version 
2
dragon ~ # cat /proc/sys/net/ipv4/conf/default/force_igmp_version 
2
dragon ~ # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.xxx.0.0      *               255.255.248.0   U     0      0        0 eth1
172.20.0.0      *               255.255.0.0     U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
BASE-ADDRESS.MC *               240.0.0.0       U     0      0        0 eth1
default         wrt600n         0.0.0.0         UG    0      0        0 eth0

eth0 is my LAN interface, and eth1 is the IPTV service (direct to my cable company).

I don't know what I'm missing. Thanks.

My hardware:
ASUS M3N78-VM
Phenom II X4 810
Intel Pro/100+ (IPTV NIC)

[Hu]

I originally saw this issue when 2.6.31 first came out, but though it was a regression or some odd configuration with my capture card, so I ignored it.

If you see a regression in kernel behavior and no one else has reported it, you should strongly consider reporting it upstream as soon as possible.

However now I have my IPTV service (using multicast) and the EXACT same issue appears as the dvb card. I've been searching and fighting this issue for almost 6 months, and have gotten nowhere. This has also happens with kernel .32, .33, .34 and mythtv .21, .22, .23.

Ask for help sooner next time. Have you checked that multicast works at all on the new kernels? We need to determine whether the problem is a complete failure of multicast or just a bad interaction between the kernel and your particular multicast IPTV service.

However when I go ANY kernel > 2.6.30, I get the following error:

Code: Select all

MPlayer SVN-r29796-4.3.4 (C) 2000-2009 MPlayer Team

Playing udp://225.1.100.1:2001.
STREAM_UDP, URL: udp://225.1.100.1:2001
Timeout! No data from host 225.1.100.1
udp_streaming_start failed
No stream found to handle url udp://225.1.100.1:2001

For the kernel where it works, do you need MythTV to configure the stream correctly first? That is, if you disabled automatic start of MythTV, rebooted, and immediately ran the mplayer command on a 2.6.30 kernel, would it work? If not, what steps are required to configure the stream to provide data to your system?

It could also be useful to see a summary network capture of eth1 on both a 2.6.30 and 2.6.31 kernel with the mplayer command shown. However, since I have not worked with multicast, it is possible that such a request is not meaningful in this context.

[Phoenix84]

Thanks for replying.

If you see a regression in kernel behavior and no one else has reported it, you should strongly consider reporting it upstream as soon as possible.

I forgot to mention, I have friends who are using later kernels with the same IPTV service without any issues. I even got a .config from one of them (using 2.6.31) and tried it out without success as well. That made me think later that it may not be a regression. Sorry for forgetting that information.
Thinking it might be a NIC issue, I also swapped with the onboard one (nForce), and still have the same issue.

For the kernel where it works, do you need MythTV to configure the stream correctly first? That is, if you disabled automatic start of MythTV, rebooted, and immediately ran the mplayer command on a 2.6.30 kernel, would it work? If not, what steps are required to configure the stream to provide data to your system?

MythTV is not required. The stream should still work without it.

It could also be useful to see a summary network capture of eth1 on both a 2.6.30 and 2.6.31 kernel with the mplayer command shown. However, since I have not worked with multicast, it is possible that such a request is not meaningful in this context.

I'd love to post more information like that, as well as my .config. Is there a way to attach files in this forum, or should I just post in a message?

[Hu]

For small configuration bits (<= 30 lines), you can post it here in a

Code: Select all

 block.  For large postings, such as [b].config[/b] files or build logs, post them to a pastebin and provide a link here.  For the [b].config[/b], please post a comment-stripped version.  You can remove comments using [b]grep -e '^[^#]' .config[/b].

[Phoenix84]

Thanks!

I've uploaded the .config for the kernel 2.6.30 I'm running that works: http://pastebin.com/tzY8MSi3

I've also included the .config for 2.6.31 that doesn't work: http://pastebin.com/0AmkfAbF

I also included diff of the two configs (just to be thorough): http://pastebin.com/njG5AF7g

Mostly, things were added to the new kernel, and only a handful were removed:
CONFIG_UNEVICTABLE_LRU
CONFIG_DMAR_GFX_WA
CONFIG_COMPAT_NET_DEV_OPS

Other options look like they were either renamed, or disabled in my config during the upgrade:
CONFIG_STACKTRACE
CONFIG_NOP_TRACER
CONFIG_RING_BUFFER
CONFIG_TRACING
CONFIG_BLK_DEV_IO_TRACE
CONFIG_BINARY_PRINTF

Here's the successful tcpdump (IP's masked to protect the innocent ):

Code: Select all

dragon ~ # tcpdump -i eth1 -n -c10 -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
14:47:01.455419 IP (tos 0x60, ttl 60, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 172.xx.xx.xx > 10.xx.xx.xx: ICMP echo request, id 3413, seq 1, length 64
14:47:02.465384 IP (tos 0x60, ttl 60, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 172.xx.xx.xx > 10.xx.xx.xx: ICMP echo request, id 3413, seq 2, length 64
14:47:16.722796 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.xx.xx.xx > 225.1.100.1: igmp v2 report 225.1.100.1
14:47:16.731739 IP (tos 0x60, ttl 4, id 42271, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.734318 IP (tos 0x60, ttl 4, id 42272, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.738614 IP (tos 0x60, ttl 4, id 42273, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.741041 IP (tos 0x60, ttl 4, id 42274, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.745749 IP (tos 0x60, ttl 4, id 42275, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.748411 IP (tos 0x60, ttl 4, id 42276, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:47:16.752782 IP (tos 0x60, ttl 4, id 42277, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
10 packets captured
10 packets received by filter
0 packets dropped by kernel

And the unsuccessful one (looks very similar though ):

Code: Select all

dragon ~ # tcpdump -i eth1 -n -c10 -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
14:55:01.094977 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 32, options (RA)) 10.xx.xx.xx > 225.1.100.1: igmp v2 report 225.1.100.1
14:55:01.104904 IP (tos 0x60, ttl 4, id 52687, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.107221 IP (tos 0x60, ttl 4, id 52688, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.111264 IP (tos 0x60, ttl 4, id 52689, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.114065 IP (tos 0x60, ttl 4, id 52690, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.117967 IP (tos 0x60, ttl 4, id 52691, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.120590 IP (tos 0x60, ttl 4, id 52692, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.124940 IP (tos 0x60, ttl 4, id 52693, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.127225 IP (tos 0x60, ttl 4, id 52694, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
14:55:01.131575 IP (tos 0x60, ttl 4, id 52695, offset 0, flags [DF], proto UDP (17), length 1344) 172.xx.xx.xx.53770 > 225.1.100.1.2001: UDP, length 1316
10 packets captured
10 packets received by filter
0 packets dropped by kernel

Looking at that it appears I may be incorrect about the multicast issues, as you can already see the data coming in (and the mythtv specific issue does affect my dvb tuner as well).

Maybe V4L? I'm running out of ideas.

[Hu]

None of that looks suspicious. I would expect that multicast IPTV would not require any kernel video support, since applications could receive the traffic directly off the network. The only remaining problem I can think of is that you built some key component as a module and 2.6.30 happens to load it, but 2.6.31+ happen not to load it. Check the lsmod output of each kernel to see if there is a difference in what modules are loaded.

[Phoenix84]

The only remaining problem I can think of is that you built some key component as a module and 2.6.30 happens to load it, but 2.6.31+ happen not to load it. Check the lsmod output of each kernel to see if there is a difference in what modules are loaded.

I hadn't thought of that. Unfortunately that doesn't seem to help. The only module difference was the 'it87' module I use for hardware monitoring. That module doesn't load in 2.6.31 (http://bbs.archlinux.org/viewtopic.php?pid=644705).

For a test, I loaded VLC on a windows machine on my network, and streamed a video over UDP multicast (to the same address and port). I switched the route to use eth0 instead, and it seems to work fine! Mplayer sees my stream, and I can even watch it in mythtv.

I also tested the IPTV stream with forcedeth again, and this time it worked too! It seems like a regression in e100.

I'm going to keep testing, then look at the commits and see if I can find some answers.

EDIT:
Perhaps I spoke too soon, it seems to have stopped working again on forcedeth. It got DNS settings from the wrong interface, so my network wasn't set up properly and it worked (not sure why), but when I fix the configuration and restarted the interfaces, it broke.

Mplayer seems to work if a single interface is active. Even on the e100 (eth1) I got the streaming to work again, but only while eth0 is inactive (I unloaded the module to be safe). Mythtv still cannot lock though (L__).

EDIT (again):
AH! It's a routing issue. Normally the default gateway is my LAN (obviously), but the multicast streaming won't work unless the default gateway is across eth1 (my IPTV service). I had mplayer streaming successfully for 10 mins, when I removed the default gateway, the stream immediately stopped. This is even though my multicast route is over eth1 (route add -net 224.0.0.0/4 dev eth1). This seems to what changed in 2.6.31 (assuming the mythtv channel locking issue is unrelated to the kernel).

[Phoenix84]

Found it!
It actually was the rp_filter setting. In 2.6.31, rp_filter calculation code was changed, see: http://linux.derkeiler.com/Mailing-List ... 04929.html

Anyway, I originally had in my /etc/conf.d/local.start:
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter

I guess that wasn't good enough, so I instead put those in /etc/sysctl.conf (better place anyway), and put this in local.start instead:
echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter

And now everything works!

Thanks for your help!

[Hu]

Good to see that you found a solution. You might be able to use loose filtering instead of strict filtering, to get some of the benefit without breaking the stream entirely. However, although disabling rp_filter is sometimes necessary for asymmetric routing, since you appear to be using symmetric routing, it seems likely that the problem here is that your routes are set incorrectly. Disabling the filter allows it to work since you do not need to send anything, so you do not notice that the kernel is routing traffic out the wrong interface. As mentioned in the thread you referenced, the 2.6.30 kernel mishandled the rp_filter, so the routes have probably been wrong for a very long time and you never noticed until 2.6.31 fixed the check.