Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Kernel panicing when WAN down, IPv6 in v4 tunnel to blame?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 101
Location: Central Washington

PostPosted: Sat Nov 08, 2014 3:41 pm    Post subject: Kernel panicing when WAN down, IPv6 in v4 tunnel to blame? Reply with quote

So my internet went down a few days ago. First I tried to SSH into my router box, but couldnt connect. So I connected to my Moxa which is connected to the serial port.
I could watch it boot and then panic within a few seconds of coming up.
They always looked like this: http://pastebin.com/MsMfY95V
So I finally modified the kernel command line so it wouldnt start my 'services' softlevel, only booting the minimal set of services. Then I manually started one by one until I discovered that having the v6 tunnel up with named would cause a panic. I change the named config to disable v6 and left my IPv6 tunnel down. That seemed to fix it. At least, it was no longer panicing. It was after I did this I found out my cable modem had been disabled because my ISP somehow lost my card info then didnt tell me until they just shut it off for 2 months of backpayments due.
Looking at the stack, now Im no expert, but I see a lot of ipv6 stuff in it. I just tried it now, and I can reliably cause a panic by pinging a v6 address and rebooting my cable modem.
wtf is going on?
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 6132
Location: almost Mile High in the USA

PostPosted: Sat Nov 08, 2014 6:39 pm    Post subject: Reply with quote

What version of the kernel are you using? Using latest?
May just be a kernel bug...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 101
Location: Central Washington

PostPosted: Sat Nov 08, 2014 9:47 pm    Post subject: Reply with quote

3.15.10-hardened-r1
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 6132
Location: almost Mile High in the USA

PostPosted: Sun Nov 09, 2014 12:41 am    Post subject: Reply with quote

Tried a later kernel? non-hardened kernel?

Not sure of any bugfixes but typically repeatable oops/aieees are kernel bugs, whether it real or someone sending you a bad packet.

I am currently running a dual stack IPV4/IPV6 but haven't seen any crashes, but I don't have the exact same setup as you....
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 101
Location: Central Washington

PostPosted: Sun Nov 09, 2014 2:35 am    Post subject: Reply with quote

Tried latest hardened 3.17.2
It still died, but slightly different output
http://pastebin.com/N1qEwE3D
Ill try a vanilla kernel next, after I decide to expend the energy.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 101
Location: Central Washington

PostPosted: Mon Nov 10, 2014 2:25 pm    Post subject: Reply with quote

I think its something to do with the hardened patch set. Vanilla 3.17.2 doesnt do it. I pinged a v6 address and rebooted my modem and I got "Destination unreachable: Address unreachable" while it was down.
It never got that far before. It died trying to give that message, if Im reading the stack correctly.
Actually, I figured out some more. I looked through the stack and googled the first one that looked a little off, "report_size_overflow"
Seems to be a PAX function.
From: https://rstforums.com/forum/62312-inside-size-overflow-plugin.rst
Quote:
report_size_overflow

The plugin creates the report_size_overflow declaration in the start_unit_callback, but the definition is always in the current program. The plugin inserts only the report_size_overflow calls. This is a no-return function.

This function prints out the file name, the function name and the line number of the detected overflow. If the stmt's line number is not available in gcc then it prints out the caller's start line number. The last two strings are only debug information.
The report_size_overflow function's message looks like this (without PaX it uses SIZE_OVERFLOW instead of PAX):
Code:
PAX: size overflow detected in function main tests/main12.c:27 cicus.4_21 (max)


Ive tried to 'dmesg -w' while causing the crash in order to see if I can pop this message, but it seems to be dying too quickly or something, as it never shows up. So the only info I have are the stack traces.
Ill probably disable this particular PAX feature until this is fixed. From what Im seeing, its a kernel bug that PAX is catching. Kind of a poor work around, but I don't know enough about programming to even know where to start looking for the problem.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 11168

PostPosted: Mon Nov 10, 2014 11:26 pm    Post subject: Reply with quote

I interpret that stack dump to mean that a problem was detected, so PaX tried to force-exit the current task. Unfortunately, the current task was interrupt context, so the kernel panicked instead. You should check higher up to see if there are more informative messages preceding the kernel panic message.
Back to top
View user's profile Send private message
Colt45
Tux's lil' helper
Tux's lil' helper


Joined: 05 Sep 2007
Posts: 101
Location: Central Washington

PostPosted: Tue Nov 11, 2014 12:22 am    Post subject: Reply with quote

Hu wrote:
I interpret that stack dump to mean that a problem was detected, so PaX tried to force-exit the current task. Unfortunately, the current task was interrupt context, so the kernel panicked instead. You should check higher up to see if there are more informative messages preceding the kernel panic message.

I agree.
Unfortunately, I dont get anything useful before the stack dumps.
_________________
"Hopps" - AMD Athlon X2 4450B, 4GB, 128GB Samsung SSD, (Gentoo Hardened amd64; gcc-4.9.3; 4.4.8-hardened-r1)
"Dash" - 2x Intel Xeon 5148, 12GB FB-DIMM, 3x Samsung 1TB (mdadm RAID 5), AMD HD6950, (Gentoo amd64-multilib; gcc-5.3.0; 4.4.0-pf4)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 11168

PostPosted: Tue Nov 11, 2014 2:49 am    Post subject: Reply with quote

In that case, file it as a bug with the PaX team. Crashing the kernel is a bug. Crashing it without even reporting adequate debugging information is worse.
Back to top
View user's profile Send private message
creideiki
n00b
n00b


Joined: 19 Feb 2003
Posts: 61
Location: Linköping, Sweden

PostPosted: Wed Nov 26, 2014 6:42 am    Post subject: Reply with quote

Did you ever report this, and/or find a workaround? I think I'm seeing the same crash after upgrading my router and IPv6-in-IPv4 tunnel endpoint from 3.16.5-hardened to 3.17.3-hardened. Unfortunately, the backtrace doesn't fit on my screen, and I was more interested in getting my Internet back than in digging out a serial cable to catch the whole message, but the last 30-something lines look similar.
Back to top
View user's profile Send private message
creideiki
n00b
n00b


Joined: 19 Feb 2003
Posts: 61
Location: Linköping, Sweden

PostPosted: Wed Nov 26, 2014 4:38 pm    Post subject: Reply with quote

Never mind, I found
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum