Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Help understanding performance drop with kernel upgrade [s]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 5:57 am    Post subject: Help understanding performance drop with kernel upgrade [s] Reply with quote

I'm running 5.7.19-gentoo right now, and everything works well.

When I tried upgrading to either 5.8.12 or 5.9.1, my performance turns really terrible across the system.
Mouse pointer feels janky, applications start slowly, rendering in games is slow, I see a lot of "wait" in top. Pretty much everything turns to molasses.

I haven't experienced this before, where everything seems to be impacted, so I'm not sure where to even begin investigating.
I moved back to 5.7.19 again and I'm now compiling 5.8.16 to try the lastest 5.8, but since the problem is also in 5.9 I don't think it's going to help much.

Maybe I can bisect the kernel, but from where would I clone the source?

Perhaps the problem isn't really in the kernel, but the upgrade triggers a problem elsewhere?


Last edited by Letharion on Thu Oct 22, 2020 3:17 pm; edited 1 time in total
Back to top
View user's profile Send private message
Ionen
Veteran
Veteran


Joined: 06 Dec 2018
Posts: 1602

PostPosted: Tue Oct 20, 2020 6:08 am    Post subject: Reply with quote

Don't know if the same here but heard a similar story recently... turns out they had forgotten to run emerge @module-rebuild after updating their kernel and were missing nvidia modules for the newer kernels.
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 6:19 am    Post subject: Reply with quote

Ionen wrote:
Don't know if the same here but heard a similar story recently... turns out they had forgotten to run emerge @module-rebuild after updating their kernel and were missing nvidia modules for the newer kernels.


I suppose that could be the issue, but I would expect to be dumped out of X if that was case, as X would fail to start. As it is now I get can start a game and get 20 FPS where I normally expect to get 100+.

Having said that, here's how I normallly do a kernel upgrade, not sure if that sheds any light on the situation:
Code:
mount /boot/; cd /usr/src/linux; cp -i /usr/src/linux-$(awk '{ print $3 }' /proc/version)/.config .; make oldconfig; make && make install && make modules_install && grub-mkconfig -o /boot/grub/grub.cfg; emerge -q @module-rebuild
Back to top
View user's profile Send private message
Ionen
Veteran
Veteran


Joined: 06 Dec 2018
Posts: 1602

PostPosted: Tue Oct 20, 2020 6:22 am    Post subject: Reply with quote

Nowadays you have libglvnd that will let things still work, but give you software rendering instead that's unsurprisingly slow.

Can just check if your modules are loaded anyway, maybe something went wrong. Also output of glxinfo | head which normally should mention NVIDIA if that's what you use.
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 6:27 am    Post subject: Reply with quote

I'm now using 5.8.16 and the problem is still there.

Quote:
Nowadays you have libglvnd that will let things still work, but give you software rendering instead that's unsurprisingly slow.

I saw libglvnd show up as a flag recently, but didn't realize this would happen. Doesn't look like that's the case here though.

Code:
$ lsmod
Module                  Size  Used by
nvidia_drm             49152  17
nvidia_modeset       1167360  35 nvidia_drm
vboxnetadp             28672  0
vboxnetflt             32768  0
vboxdrv               421888  2 vboxnetadp,vboxnetflt
x86_pkg_temp_thermal    20480  0
snd_usb_audio         253952  4
snd_usbmidi_lib        28672  1 snd_usb_audio
snd_rawmidi            32768  1 snd_usbmidi_lib
nvidia              27209728  1933 nvidia_modeset
efivarfs               16384  1


The Xorg.0.log part where nvidia gets loaded:
Code:

[    12.528] (II) Applying OutputClass "nvidia" to /dev/dri/card0
[    12.528]    loading driver: nvidia
[    12.529] (==) Matched nvidia as autoconfigured driver 0
[    12.529] (II) LoadModule: "nvidia"
[    12.529] (II) Loading /usr/lib64/xorg/modules/drivers/nvidia_drv.so
[    12.536] (II) Module nvidia: vendor="NVIDIA Corporation"
[    12.541] (II) NVIDIA dlloader X Driver  455.28  Wed Sep 30 01:04:06 UTC 2020
[    12.541] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
[    12.554] (II) NVIDIA(0): Creating default Display subsection in Screen section
[    12.554] (==) NVIDIA(0): Depth 24, (==) framebuffer bpp 32
[    12.554] (==) NVIDIA(0): RGB weight 888
[    12.554] (==) NVIDIA(0): Default visual is TrueColor
[    12.554] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
[    12.555] (**) NVIDIA(0): Enabling 2D acceleration
[    12.555] (II) Loading sub module "glxserver_nvidia"
[    12.555] (II) LoadModule: "glxserver_nvidia"
[    12.555] (II) Loading /usr/lib64/xorg/modules/extensions/libglxserver_nvidia.so
[    12.633] (II) Module glxserver_nvidia: vendor="NVIDIA Corporation"
[    12.633] (II) NVIDIA GLX Module  455.28  Wed Sep 30 01:01:28 UTC 2020
[    12.635] (II) NVIDIA: The X server supports PRIME Render Offload.
[    13.098] (--) NVIDIA(0): Valid display device(s) on GPU-0 at PCI:1:0:0
[    13.098] (--) NVIDIA(0):     CRT-0
[    13.098] (--) NVIDIA(0):     DFP-0 (boot)
[    13.098] (--) NVIDIA(0):     DFP-1
[    13.098] (--) NVIDIA(0):     DFP-2
[    13.098] (--) NVIDIA(0):     DFP-3
[    13.098] (--) NVIDIA(0):     DFP-4
[    13.100] (II) NVIDIA(0): NVIDIA GPU GeForce GTX 770 (GK104) at PCI:1:0:0 (GPU-0)


Code:
$ glxinfo | head
name of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_ARB_context_flush_control, GLX_ARB_create_context,
    GLX_ARB_create_context_no_error, GLX_ARB_create_context_profile,
    GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float,
    GLX_ARB_multisample, GLX_EXT_buffer_age,
Back to top
View user's profile Send private message
Ionen
Veteran
Veteran


Joined: 06 Dec 2018
Posts: 1602

PostPosted: Tue Oct 20, 2020 6:34 am    Post subject: Reply with quote

Well I don't see anything wrong in those outputs at least, not sure then :|
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 7:53 am    Post subject: Reply with quote

Another point of comparision. I opened a web-app I frequently use in FF. On 5.8.16 it hovered around 150% CPU usage.
Rebooting back to 5.7.19 this drops down to low double-digits.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2564
Location: Frankfurt, Germany

PostPosted: Tue Oct 20, 2020 11:33 am    Post subject: Reply with quote

In order to avoid mistakes building the kernel, I run
Code:
dmesg -t >dmesg-$(uname -r)

whenever I compile and boot a new kernel. After that, I compare the newly created dmesg file with the dmesg file of the previous kernel (diff). The diff quickly shows missing drivers, different setting or other issues. If I had modules enabled, I would probably also run 'lsmod >modules-$(uname -r)' whenever I compile and boot a new kernel - and compare the modules-files.
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 11:59 am    Post subject: Reply with quote

mike155 wrote:
In order to avoid mistakes building the kernel, I run
Code:
dmesg -t >dmesg-$(uname -r)

...


Clever, I'll definitely incorporate that into my update workflow.

Having tested now, I can't spot much interesting unfortunately. The order of many things differ somewhat, but after sorting the files, the major difference is minor things like the actual kernel version, some process ids that naturally differ and a few other details.

The one little thing that stands out is:
Code:
750 i2c i2c-0: 2/4 memory slots populated (from DMI)
751 i2c i2c-0: Successfully instantiated SPD at 0x51
752 i2c i2c-0: Successfully instantiated SPD at 0x53

It certainly doesn't look like an error though, but I also don't know what it means.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 5047
Location: Dallas area

PostPosted: Tue Oct 20, 2020 12:14 pm    Post subject: Reply with quote

what does cat /proc/sys/kernel/random/entropy_avail return?

If you run it during the slowdowns, what does it show?

Also what CPU are you on and which governor are you running? And are you by any chance running conky?
_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie) amd64-no-multilib, eudev, openrc, openbox

The New OTW
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 12:27 pm    Post subject: Reply with quote

Code:
$ grep name /proc/cpuinfo
model name   : Intel(R) Core(TM) i5-5675C CPU @ 3.10GHz


For reference, here's what the entropy and governor say when the system is normal:
Code:
$ cat /proc/sys/kernel/random/entropy_avail
3523
# cat /sys/devices/system/cpu/cpu{0,1,2,3}/cpufreq/scaling_governor
powersave
powersave
powersave
powersave


Gonna reboot and check the slow kernel in a minute.

Edit:
Code:
# cat /sys/devices/system/cpu/cpu{0,1,2,3}/cpufreq/scaling_governor
userspace
userspace
userspace
userspace
$ cat /proc/sys/kernel/random/entropy_avail
1761


Ah, that scaling governor change looks very suspiscous.
In fact, I'm not even allowed to change it back:
Code:
# echo powersave > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
bash: echo: write error: Invalid argument

Which is probably the reason why it now defaults to a different value.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 5047
Location: Dallas area

PostPosted: Tue Oct 20, 2020 12:38 pm    Post subject: Reply with quote

Why are you using powersave?

Code:
      Use the CPUFreq governor 'powersave' as default. This sets
      the frequency statically to the lowest frequency supported by
      the CPU.

_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie) amd64-no-multilib, eudev, openrc, openbox

The New OTW
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 12:38 pm    Post subject: Reply with quote

Code:
echo performance | tee /sys/devices/system/cpu/cpu{0,1,2,3}/cpufreq/scaling_governor


This makes the system much more responsive right away, yey!
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 5047
Location: Dallas area

PostPosted: Tue Oct 20, 2020 12:40 pm    Post subject: Reply with quote

Letharion wrote:

Edit:
Code:
# cat /sys/devices/system/cpu/cpu{0,1,2,3}/cpufreq/scaling_governor
userspace
userspace
userspace
userspace
$ cat /proc/sys/kernel/random/entropy_avail
1761


Ah, that scaling governor change looks very suspiscous.
In fact, I'm not even allowed to change it back:
Code:
# echo powersave > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
bash: echo: write error: Invalid argument

Which is probably the reason why it now defaults to a different value.


userspace is wrong :lol:

But that amount of entropy is awfully small and will cause the system to pause a fair amount.
_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie) amd64-no-multilib, eudev, openrc, openbox

The New OTW
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 12:40 pm    Post subject: Reply with quote

Anon-E-moose wrote:
Why are you using powersave?

Code:
      Use the CPUFreq governor 'powersave' as default. This sets
      the frequency statically to the lowest frequency supported by
      the CPU.


I don't actually know, this is a desktop, so there's no strong reason to that I can think off.
I may have had a reason at some point and just forgotten about it.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 5047
Location: Dallas area

PostPosted: Tue Oct 20, 2020 12:42 pm    Post subject: Reply with quote

You might check the "default governor" in the kernel config, sounds like userspace got set by default.

Code:
$ grep FREQ_DEFAULT_GOV /usr/src/linux/.config

_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie) amd64-no-multilib, eudev, openrc, openbox

The New OTW
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 12:47 pm    Post subject: Reply with quote

Anon-E-moose wrote:
But that amount of entropy is awfully small and will cause the system to pause a fair amount.


I'll keep an eye on that now, does it maybe take the system some time to accumulate events? It's up now:
Code:
$ cat /proc/sys/kernel/random/entropy_avail
4001

If things slow down again later, I'll know to look at that. I/O is a source of entropy, is that right?
So ensuring I have my backup raid-discs mounted should be helpful then?
Back to top
View user's profile Send private message
Letharion
Veteran
Veteran


Joined: 13 Jun 2005
Posts: 1333
Location: Sweden

PostPosted: Tue Oct 20, 2020 12:51 pm    Post subject: Reply with quote

Anon-E-moose wrote:
You might check the "default governor" in the kernel config, sounds like userspace got set by default.

Code:
$ grep FREQ_DEFAULT_GOV /usr/src/linux/.config


Well, yes, but the value doesn't appear to have changed?

Code:
$ grep FREQ_DEFAULT_GOV /usr/src/linux-5.7.19-gentoo/.config
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
$ grep FREQ_DEFAULT_GOV /usr/src/linux-5.8.16-gentoo/.config
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set


I _did_ try to compare the configs before I posted my question here, and I couldn't spot anything interesting then.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 5047
Location: Dallas area

PostPosted: Tue Oct 20, 2020 1:44 pm    Post subject: Reply with quote

You shouldn't use the userspace driver UNLESS you're going to run some program to handle changing the frequency changes, which is what it's designed for.



As far as entropy, it's generated by a lot of things, keyboard input, mouse movement.

I use rng-tools to add to entropy, put it in the boot runlevel
Code:

sys-apps/rng-tools-6.10::gentoo  USE="jitterentropy nistbeacon pkcs11 (-selinux)"

 * Found these USE flags for sys-apps/rng-tools-6.10:
 U I
 + + jitterentropy : Enable Jitter RNG entropy support
 + + nistbeacon    : Enable NIST beacon entropy support
 + + pkcs11        : Enable PKCS11 entropy support

_________________
PRIME x570-pro, 3700x, RX 550 - 5.8 zen kernel
Acer E5-575 (laptop), i3-7100u - i965 - 5.5 zen kernel
---both---
gcc 9.3.0, profile 17.1 (no-pie) amd64-no-multilib, eudev, openrc, openbox

The New OTW
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum