2.6.35 hangs badly

Ormaaj · Guru Joined: 28 Jan 2008 Posts: 319

2.6.35 kernel has been causing everything to freeze up after running certain heavy loads like compiling. All processes freeze except when triggering interrupts by moving the mouse or switching between virtual desktops. For example - watching a video will freeze completely except when moving the mouse - everything returns to normal. The only indication is that my xmobar displays "cpu total not found" rather than the load %.

This has been going on since about rc1-rc2. It occurs in both unstable and stable zen kernel (2.6.34 with some backported things), all vanilla 2.6.35 rcs, but not vanilla 2.6.34.1. I really don't know where to start looking to diagnose this. A git bisect would take ages because it occurs unpredictably within an hour of booting. Arch is amd64. Not sure what kind of info would be useful in diagnosing.

d2_racing · Posted: Thu Aug 05, 2010 12:47 pm Post subject:

Are you using the latest vanilla-sources or the gentoo-sources ?

Ormaaj · Guru Joined: 28 Jan 2008 Posts: 319

dufeu · l33t Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST

Timbers2k · Apprentice Joined: 03 Oct 2003 Posts: 215

I'm having the exact same problem. The worst case for me is playing Wow! After a few minutes the sound start stuttering and then everything starts to crawl. Once I exit Wow my desktop is very slow, and gkrellm is showing cpu use staying high, but nothing shows in top.

I'm also using a Core i7, and I did set up the new cpuidle driver. I'll try disabling that and see if it makes a difference.

Timbers2k · Apprentice Joined: 03 Oct 2003 Posts: 215

It works fine if you disable the "Cpuidle Driver for Intel Processors" option under "Power management and ACPI options". Seems that this option is not quite ready for prime time.

maj · Tux's lil' helper Joined: 22 Nov 2002 Posts: 92

See, its the first thing I disabled when I found the problem - still exists for me, mplayer will output it's your system is too slow message, which given its an i7 system, I think not! Besides, video playback is handed off to the GPU!

Edit - ok, removed "ACPI Processor P-States driver" aswell as "Cpuidle Driver for Intel Processors", now the system is behaving itself!

Edit 2 - Or not, still have the issue - its less pronounced, but it still happens

Cffeine · n00b Joined: 09 Aug 2010 Posts: 1

Hey all,

Just thought I'd throw in my experience. I also have an i7 and was experiencing the same problem with 2.6.35 (gentoo-sources). Any heavy load would bring my computer to a crawl (with the exception of moving the mouse, clicking menus, etc. Similar to what the OP was experiencing). My system never seemed to recover but I'm not sure if I gave it adequate time to do so as the system would usually be so slow I would have difficulty getting the offending process to close. I would usually have to use SysReq+REISUB to get it to reboot. I tried disabling "ACPI Processor P-States driver" as well as "Cpuidle Driver for Intel Processors" as mentioned but with no luck.

I finally got it to run properly by also disabling all the "CPU Frequency scaling --->" options. Not sure which one was causing the problem as I didn't have time over the weekend to test every governor combination.

My system now remains responsive as it should, but I have a couple of other strange issues that are probably not related. Namely: if I try and manually mount one of my hard disks on the command line, mount just hangs and the system goes slightly unstable -- I can log out, but If I try and shut the computer down, it just hangs. Not sure if this is a 2.6.35 kernel specific problem or if its because I disabled CONFIG_IDE as instructed by udev. I do know I didn't have this problem with 2.6.34 but I didn't take udev's advice for disabling CONFIG_IDE until I started messing with 2.6.35.

Not trying to wander off-topic, but here is the output of dmesg | tail after trying to do a mount operation:

dufeu · l33t Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST

Ormaaj · Guru Joined: 28 Jan 2008 Posts: 319

I can also confirm that the new intel idle driver isn't the problem - but if the solution is disabling all power saving then that would really suck as I have quite an overclocked system. P/C-states along with frequency scaling do a very nice job of keeping the temps down and saving power when idle (which is 99% of the time) so I don't really want to disable them keeping my CPU buzzing away at 4 Ghz at all times.

Maybe I'll just stick to 2.6.34.x until this gets fixed upstream. There should really be no reason to disable ACPI C-states.

EDIT: I haven't noticed the problem yet with the performance governor and ACPI P-state driver enabled along with intel cpuidle. Only been testing for about 20 minutes though so we'll see.

Ormaaj · Guru Joined: 28 Jan 2008 Posts: 319

Still seems fine after switching from ondemand to performance.

Also... just judging from the description relative to the symptoms, this one looks like a very likely culprit:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=1f85f87d4f81d1e5a2d502d48316a1bdc5acac0b
http://lwn.net/Articles/386990/

dufeu · l33t Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST

Ormaaj · Guru Joined: 28 Jan 2008 Posts: 319

I don't think this should affect PCIe because none of this should have an effect on board frequencies or even the input frequency to the CPU. Current intel frequencies on i7 are quite complex. There's sort of a hierarchy of things which affect actual core frequencies: FSB (BCLK) -> multiplier -> ACPI P-States -> Intel Turbo. The latter two happen within the chip and should be pretty transparent to everything else, and Turbo you really don't have any control over and I don't really understand why Turbo isn't just another P-state because I would think that should have the same effect. Additionally, there are the C-states which control sleeping, and even more beyond that which is controlled by that intel idle driver... I guess ACPI wasn't good enough for Intel so they had to add their own proprietary extensions.

P-states are controlled in discrete increments by the frequency scaling driver and work independently for each core. It shoudln't cause side-effects in other subsystems. You can see the true frequencies and P-states and C-states in action if you install i7z (there's no ebuild afaik)

http://code.google.com/p/i7z/

and also powertop gives some C-states info

http://www.lesswatts.org/projects/powertop/

If you notice in i7z, p-states are still doing their thing even if you use the performance governor. Enabling the on-demand driver seems to make p-states more aggressive for some reason... when your cpu is idle it really gets down to lower frequencies than without it. Disabling the p-states driver entirely causes you to always run at the highest p-state.

ferrarif5 · Posted: Thu Aug 12, 2010 7:05 am Post subject:

I've got the same issue as described in the thread, lagging response from keyboard, mouse, screen refresh with my CPU hitting 100% load, think I'll roll back to gentoo-sources-2.6.34 for now.
_________________
Asus P6X58D-E Mobo
Intel Core i7 920
18GB Corsair DDR3
User:335876 | Screenshot

drescherjm · Posted: Wed Aug 18, 2010 4:21 am Post subject:

I may be having a different issue but instead of my kernel booting in 20s or so on my i7 3.0GHz it takes over 90s and typing or anything else in the terminal reminds me of the early 1990s and dialing into my university with a 1200 baud modem, heck 1200 baud was faster..
_________________
John

My gentoo overlay
Instructons for overlay

dufeu · l33t Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST

drescherjm · Posted: Sat Aug 21, 2010 1:08 pm Post subject:

Thanks. I will try disabling C1E. However that is only a temporary solution since I find it unacceptable to have to disable this power saving feature.

[EDIT]Well at least on my machine power management is not the cause. I disabled CxE and SpeedStep in my BIOS and the problem continued. So back to 2.6.34. [/EDIT]
_________________
John

My gentoo overlay
Instructons for overlay

the_bard · n00b Joined: 03 Dec 2002 Posts: 60 Location: Albany, NY

I'm running across symptoms as experienced by the above: Running gentoo-sources 2.6.35-r1, I'm getting odd "soft" hanging on boot. I've got a boot splash screen configured, so leaving it in silent mode simply displays the loading bar as normal, but it hangs. The system does respond to input... switching to the verbose boot splash causes the system to unhang, whereupon it appears to hang at the next event.

Any keyboard activity seems to temporarily resolve the issue, very briefly. I've babysat the boot process, hitting keys repeatedly, until my system tries to load KDM. I don't have the patience to continue at that point.

I upgraded to 2.6.35-r1 by copying the .config from 2.6.34, then performing a `make oldconfig`. Removing the new intel idle CPU and APCI P-States drivers have done nothing. If I remember correctly, I even disabled ACPI completely within the kernel... it made no difference. I swapped back to 2.6.34.

I'm running a Core i7-860 on an Intel DP55WG board, for what it's worth.

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

2.6.35 with intel cpuidle caused really bad hangs and panics in KVM for me. 2.6.35.3 without anything new (answered N to all when make oldconfig from 2.6.34.x) does not have bad hangs, but in less than 6 hours I already had a KVM machine crashing again. There is no problem whatsoever with 2.6.34.x. So something is definitely odd with 2.6.35 for me, on a Intel i7 920 machine. I'll try disabling anything power saving related next since disabling the cpu intel idle driver already improved things a lot. Other than that I'm out of ideas as to what could be the culprit.

dufeu · l33t Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

The VMs are idle themselves most of the time. With the host running 2.6.35, they get kernel soft lockups when they are not.

And all that does not explain why it works fine in 2.6.34 (and older), but not in 2.6.35, using basically the same config.

Edit: Disabling the Power Management section entirely seems to have further improved the situation.
Edit2: Nah, still soft lockups in KVM. Unrelated to power management after all. Must be KVM doing something strange.

zx2c4 · Developer Joined: 09 Jun 2005 Posts: 177

@the_bard

Your symptoms are exactly the same as mine. Did you find a solution eventually?

optiluca · Posted: Sun Sep 05, 2010 9:03 am Post subject:

Genewb · Apprentice Joined: 09 Jan 2007 Posts: 165

There's a tracker ticket--which hasn't been answered--and another report on the mailing list was sent yesterday, which also hasn't been answered. It seems that no kernel hackers care much.
_________________
I don't give a darn about "experience", just functional copyleft software.

drescherjm · Posted: Fri Sep 10, 2010 12:29 am Post subject:

Genewb

Thanks. The second thread led me to the solution. Nix found by bisecting the kernel that this problem was with the clocksource. In his case it was hpet. I was using tsc. Switching to acpi_pm and all is well.

https://forums.gentoo.org/viewtopic-t-842775-highlight-.html
_________________
John

My gentoo overlay
Instructons for overlay