[SOLVED] Unhappy with CPU frequency / performance

[regox]

I have a laptop with a fairly recent CPU, an i7-10610U. It has 8 Cores and a maximum frequency of 4.9 GHz.

What I don't understand is why my CPU frequency (and therefore noticeably performance) is dropping as soon as I perform computationally heavy tasks. In Idle, the CPU is around 800 MHz - 1 GHz. During normal browsing, PDF viewing etc. it jumps up to and beyond 4 GHz, so far so good.

But when I'm compiling a kernel or use QEMU/KVM, the frequency is stuck at 400 MHz (or sub-1GHz most of the time) and my system feels very sluggish to use. In my understanding, the CPU should be at its maximum frequency in those cases. The frequency should only go down when the task is finished or thermal throttling kicks in. I don't think it's a thermal problem, the CPU reports 52 °C (125° F) right now, that should be okay.

I am using "performance" governor

Code: Select all

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance

I am not using any userspace tool for CPU frequency governance or any fancy desktop environment that might do things on its own. I use OpenRC.

Is this something I can fix or is there a flaw in my reasoning? What causes the CPU to throttle?

[mike155]

1. Please post the output of

   Code: Select all

   uname -a

   and of

   Code: Select all

   cpupower frequency-info

2. Please run

   Code: Select all

   while [ 1 ]; do i=1; done &

   That will run a job with a constant load in the background. Use 'top' to verify that this job is running. What are CPU frequency and CPU temperature? Let this test run for at least two minutes. Do CPU frequency and temperature reach stable values? Or do they jump up and down?
3. Run the command 3 more times. Use 'top' to verify that 4 jobs are running. What are CPU frequency and CPU temperature? Let this test run for at least two minutes. Do CPU frequency and temperature reach stable values? Or do they jump up and down?
4. Run this command 4 more times. Use 'top' to verify that 8 jobs are running. What are CPU frequency and CPU temperature? Let this test run for at least two minutes. Do CPU frequency and temperature reach stable values? Or do they jump up and down?
5. Kill the jobs running in the background

   Code: Select all

   jobs
   kill %1
   kill %2
   ...

[Hu]

52C seems rather high for an idle CPU, though that would be quite reasonable under load. What were the system circumstances when you observed the 52C? If the system was idle then, what temperature does it reach when you have it under full load? My system idles in the 34C-38C range. Under full load for an extended period, I have seen it go as high as 80C.

[regox]

Thank you both for the replies.

Code: Select all

uname -a
Linux thinkpad 5.18.0-rc5 #22 SMP PREEMPT_DYNAMIC Wed May 11 12:46:57 CEST 2022 x86_64 Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz GenuineIntel GNU/Linux

I know I run a "pre-release" vanilla kernel and not a gentoo-sources kernel. I was running a self-configured and a genkernel Gentoo kernel before, without any noticeable difference.
My config is here.

Code: Select all

cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 4.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 4.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.00 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

I did the test like you described.
In Idle, the CPU is at around 46°C. This sounds relatively high, but actually all the cores are at 2.30 GHz except for one jumping to around ~4 GHz occasionally (CPU load <1%). I don't know why the frequency is not lower. I thought I observed a baseline of 800 MHz with performance governor after some time, but maybe I am mistaken. With the "powersave" governor the CPU is actually at 800 MHz in idle. This leads to a temperature of around 35 °C.

Starting the first load command, one core jumps to 100% utilization and a steady 3.60 GHz @ 75° C. This goes on for about a minute, when it goes down to 1.60 GHz @ 57° C. Now the fan kicks in. Everything looks very stable, the values do not jump around.
Running 4 load commands, four cores at 100% are reporting that they're running at 700 MHz @ 50°C. Again, really stable; no change in 2 minutes.
Running 8 load commands does not change anything much. 8 cores are now at 700 MHz @ 50°C. No change either.

So it seems to me the CPU frequency controlling is rather cautious and conservative. Is it normal that the governor does not raise the frequency although the CPU is only at 50°C ? I also have manual control over the fan, leaving it on on full speed does not change the frequency much.

I don't know if I am just asking too much. But being used to the Gentoo way, I expected to have a little more control over my CPU Can't I "force" the kernel to keep the CPU frequency constant (at least for testing)? I mean there should still be thermal emergency shut-offs and such, right?

[mike155]

Thanks for the test results.

Something is totally wrong with either your kernel or with your notebook. Maybe both. If I had to guess I would say that sleep states don't work.

Please post your kernel config using wgepaste.

[grknight]

You should read /usr/src/linux/Documentation/admin-guide/pm/intel_pstate.rst on how the intel_pstate driver works and how the governor is ignored to some extent.

[regox]

Thanks for your quick reply.

Something is totally wrong with either your kernel or with your notebook. Maybe both. If I had to guess I would say that sleep states don't work.

Please post your kernel config using wgepaste.

I have uploaded my config file here: https://file-share.ch/b6b13b34973a99f4e ... 6f1/config You must have missed it in my previous post.

You should read /usr/src/linux/Documentation/admin-guide/pm/intel_pstate.rst on how the intel_pstate driver works and how the governor is ignored to some extent.

Thanks for the hint. I added "intel_pstate=disable" to kernel parameters, to prevent intel_pstate from registering as a CPUFreq agent. This makes acpi-cpufreq the driver in use.

Code: Select all

analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 10.0 us
  hardware limits: 400 MHz - 2.30 GHz
  available frequency steps:  2.30 GHz, 2.30 GHz, 2.20 GHz, 2.00 GHz, 1.80 GHz, 1.70 GHz, 1.60 GHz, 1.50 GHz, 1.30 GHz, 1.20 GHz, 1.10 GHz, 900 MHz, 800 MHz, 700 MHz, 500 MHz, 400 MHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 1.80 GHz and 1.80 GHz.
                  The governor "userspace" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.40 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

With this, the stress command from above runs at > 4 GHz for a few seconds, pushing the corresponding core to almost 100 °C. After that, I throttles down. With all 8 cores in use, it seems to stabilize at around 1.5 GHz @ 60°C. So that already looks like an improvement.

[NeddySeagoon]

regox,

Starting the first load command, one core jumps to 100% utilization and a steady 3.60 GHz @ 75° C.

It gets to 75° C on one core not even flat out. You want to dissipate amount 8 times the power for only another 5° C or at most, 10° C temperature rise.
The cooling system isn't up to it.

Throttling will start around 80° C

You have other problems too.

[figueroa]

Not the same CPU, but I get very good CPU responsiveness on my i7-2600 (SandyBridge at least 11 years old) with intel_pstate=active on the kernel commandline and the powersave governor. It took me a long time to wrap my mind around PSTATE.

sensors show 47, 38, 37, and 47 on the four cores, in a warm room, Thunderbird in the background and Firerfox with two tabs, a Youtube video running in the background. CPU frequency bouncing around on all threads, but in the viscinity of minimum MHz. I don't believe I get any overheating at full loads.

[regox]

regox,

Starting the first load command, one core jumps to 100% utilization and a steady 3.60 GHz @ 75° C.

It gets to 75° C on one core not even flat out. You want to dissipate amount 8 times the power for only another 5° C or at most, 10° C temperature rise.
The cooling system isn't up to it.

Throttling will start around 80° C

You have other problems too.

I see. The laptop is only about 16 months old, so I would expect the thermal paste etc. to still be good. I know that there is not much dust because I recently opened it up. Would you recommend to replace the thermal paste? I was also thinking about writing a script that runs the fan more often. It could also be that the heat pipes are just badly designed, which I can't fix of course... But then I would expect others to have problems too with this model (haven't found much searching around).

I understand Intel p-states are more complex than I thought... From the suggested documentation I also learned there is some hardware-based controlling. Is is possible (with some other driver?) to control my CPU frequency entirely from userspace? I think in such a setup I would also be able to find thermal/physical problems better.

What would you recommend I try next to find the culprit?

[Goverp]

If you're running bleeding edge kernels, are you also doing anything with fan controls? My current desktop has all sorts of ways to screw up the cooling! Or maybe the fan is broken.

[pietinger]

The behavior of Intels P-State Performance for a DESKTOP is:
When your cores are idle it switches all cores to BOOST frequency (if CPU has a boost; if not then to maximum). If your cores are under pressure it switches BACK to MAXimum frequency. The minimum frequency is almost not really needed ! (Sometimes it goes below MAX and of course it is not allowed to go below MIN) (I dont know for a notebook; never tried).

Recommendation for a NOTEBOOK (with a newer CPU than Intels 4.Gen) is P-State Powersave. This governor uses of course also MINimum frequency You should use only this one and not ACPI ... BUT ... you should also configure your kernel for using a Notebook: LPSS and all other options needed for A / THIS special notebook model. e.g.

Code: Select all

[*] Intel Low Power Subsystem Support

Thanks for the hint. I added "intel_pstate=disable" to kernel parameters, to prevent intel_pstate from registering as a CPUFreq agent. This makes acpi-cpufreq the driver in use.
[...]
With this, the stress command from above runs at > 4 GHz for a few seconds, pushing the corresponding core to almost 100 °C. After that, I throttles down. With all 8 cores in use, it seems to stabilize at around 1.5 GHz @ 60°C. So that already looks like an improvement.

If you have P-State Powersave and your Notebook never reaches higher frequencies under load, then there must be a problem ... either with cooling or a defect HWP. For correct HWP handling you should (always) load the newest microcode for your CPU.

[mike155]

I added "intel_pstate=disable"

I would not do that. Try to stay below 100°C.

Enter the BIOS Setup and make sure that sleep states (C3, C6, C7) are enabled.

Disable hyperthreading. It doesn't make sense to have 8 CPUs if the thermal limit is already reached when two or three jobs are running.

Run powertop. The second and the third tab show frequencies and idle states. Run one of the while loops above. Make sure that one CPU is running and that all other CPUs are in one of the higher sleep states (C3, C6, C7).

The fifth tabs shows other tunables of your machine. Enable as many energy saving options as possible to reduce power consumption of your machine.

Run "grep -i intel .config". Some of the drivers for Intel hardware are not enabled in your kernel. CONFIG_INTEL_IDLE for example. You may also want to enable some drivers like CONFIG_X86_INTEL_LPSS & friends or CONFIG_INTEL_PMC_CORE. It's difficult to predict which drivers are needed on your machine.

Boot from the latest SystemRescueCD or any other distribution. Repeat the test with the while loops above. Do you get a better result? If you do: run lsmod to get a list of the modules loaded - and enable those in your kernel.

[regox]

Thanks for all the help. I really appreciate it.

If you're running bleeding edge kernels, are you also doing anything with fan controls? My current desktop has all sorts of ways to screw up the cooling! Or maybe the fan is broken.

No, currently I am not using any additional fan control. Although I enabled manual control using thinkpad_acpi kernel module, the fan state is set to "auto". I can also force it to run at full speed, that does not influence the frequency control that much, but lowers the temperature a few degrees as expected. The rpm seems okay to me (0...>5000 rpm) so I think it's doing okay.

Recommendation for a NOTEBOOK (with a newer CPU than Intels 4.Gen) is P-State Powersave. This governor uses of course also MINimum frequency You should use only this one and not ACPI ... BUT ... you should also configure your kernel for using a Notebook: LPSS and all other options needed for A / THIS special notebook model. e.g.

Okay, that does make sense. I changed the default governor to p-state powersave and enabled CONFIG_X86_INTEL_LPSS and CONFIG_INTEL_IDLE.

Enter the BIOS Setup and make sure that sleep states (C3, C6, C7) are enabled.

I am not sure if this is the case when I select "Linux" for "Sleep State". In the BIOS, I got the following options:

Code: Select all

Intel(R) SpeedStep Technology         (On/Off)
  Mode for AC                         (Performance/Battery optimized)
  Mode for Battery                    (Performance/Battery optimized)
  Adaptive Thermal Management  
     Scheme for AC                    (Performance/Balanced)
     Scheme for Battery               (Performance/Balanced)
CPU Power Management                  (On/Off)
Sleep State                           (Windows/Linux)

Disable hyperthreading.

Done. I thought I had 8 physical cores, but now I found out that I only have 4, but 8 threads. Good to know.

I also disabled "SpeedStep" and "CPU Power Management" in the BIOS menu (for AC, I selected "Performance").
I also removed intel_pstate=disabled, so I am using this driver again.

That improved things already a lot! If I run 1 job, the corresponding core settles at 3.80 GHz @ 85°C. Four simultaneous jobs settle at 2.2 GHz @ 69°C.

Run powertop. The second and the third tab show frequencies and idle states. Run one of the while loops above. Make sure that one CPU is running and that all other CPUs are in one of the higher sleep states (C3, C6, C7).

I can confirm, that is exactly what happens now.

Enable as many energy saving options as possible to reduce power consumption of your machine.

I added that to my WM start script. Everything is "Good".

From these tests I'd say the performance is way better now. I will use it like this for a few days to see how it's going when in normal use. Thanks a lot!

[mike155]

I have a Dell Latitude 5289 notebook with an Intel 5-7200U CPU @ 2.50GHz. I use the Intel pstate driver and the governor "powersave".

Below are the results on my machine. '# jobs' is the number of while loops running in parallel:

Code: Select all

# jobs  CPU Freq  CPU Temp
---------------------------------------------------------------------------------------------
   0    0.9 GHz   30°C, no fan
   1    3.1 GHz   70°C, fan starts (slowly) after 2 minutes, keeps temp at 70 °C 
   2    2.9 GHz   80°C, fan starts to run a little faster after 2 minutes, keeps temp at 80 °C
   3    2.9 GHz   86°C, fan runs full speed
   4    2.9 GHz   90°C, fan runs full speed

[regox]

Okay, so I think I finally found the problem. Although it was better with the fixes you suggested, the behavior was not repeatable and sometimes my laptop would still fall back to this 400 MHz-state sometimes. Especially on battery, this occurred more often. I tried several things over several months, but nothing was really satisfactory.

After some more searching, also specifically for my laptop model (Lenovo Thinkpad P14s), I found that other people (even on Windows OS sometimes!) seem to have similar problems. What finally helped me was throttled[1]. Seems like the limiting factor was the package power limit, that explains why my CPU frequency would collapse randomly under heavy load on all cores (because it trips some power limit). Now it is very stable for prolonged periods of time, even under full load at 3-4GHz while also getting reasonably hot, but never exceeding 95°C. I am very happy now with performance. Hopefully it stays that way now.

There is even an Overlay available for Portage and support for OpenRC! [2]

[1] https://github.com/erpalma/throttled
[2] https://github.com/erpalma/throttled-overlay

[AstroFloyd]

What finally helped me was throttled

Oh yes, that seems to have fixed my issue on a ThinkPad T14 Gen2 (what's in a name?) with a 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz CPU. Your message set me onto the correct path, thank you so much!

My steps: I added the throttled overlay, installed the latest version (v0.9.2), started the service and ... nothing changed. Then I keyworded v9999, updated the package, restarted the service (whilst emerging) and the CPU frequencies shot up from 1.2-1.3GHz to >4GHz (initially) and then 3-4GHz (for a longer time), temperatures jumped from ~45degC to 70-100degC and my fan went into helicopter mode.

[mark4]

Okay, so I think I finally found the problem. Although it was better with the fixes you suggested, the behavior was not repeatable and sometimes my laptop would still fall back to this 400 MHz-state sometimes. Especially on battery, this occurred more often. I tried several things over several months, but nothing was really satisfactory.

After some more searching, also specifically for my laptop model (Lenovo Thinkpad P14s), I found that other people (even on Windows OS sometimes!) seem to have similar problems. What finally helped me was throttled[1]. Seems like the limiting factor was the package power limit, that explains why my CPU frequency would collapse randomly under heavy load on all cores (because it trips some power limit). Now it is very stable for prolonged periods of time, even under full load at 3-4GHz while also getting reasonably hot, but never exceeding 95°C. I am very happy now with performance. Hopefully it stays that way now.

There is even an Overlay available for Portage and support for OpenRC! [2]

[1] https://github.com/erpalma/throttled
[2] https://github.com/erpalma/throttled-overlay

Under no circumstances what so ever will I install anything like this that is not in portage proper. I will not trust some random person to not destroy my $4000 laptop with a script that jacks up important safety measures in my CPU. I do believe this is exactly what my processor is experiencing, an over zealous power throttling crippling performance but WHY do I have to run some random script kiddie skript from github to "fix" this at all? are the linux kernel developers incompetent to fix it in the kernel? Can intel not supply the information those developers need to fix this in the kernel?

I truly believe that I am being RIPPED OFF by Intel selling me a 5000000000000 rpm car that will only do 3000 rpm if there are any passengers in it and will only idle at 1/1000000 the advertised rpm. NOT happy

[regox]

Hi mark4

Meanwhile the package is in the official Gentoo repository: https://packages.gentoo.org/packages/sy ... hrottled[b][/b].
Nobody is forcing you to install it, nor is anybody preventing you from implementing a fix in the kernel or from buying a new computer.

Cheers,
regox