Rebuild your kernel after upgrading GCC to 7.3!

[nsoveiko]

today i decided to update the system and ended up with an unloadable nvidia-drivers module:

Code: Select all

Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_edx (err 0)
Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_ecx (err 0)
Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_edi (err 0)
Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_ebp (err 0)
Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_esi (err 0)
Jul  3 14:19:21 alerik kernel: nvidia: Unknown symbol __x86_indirect_thunk_eax (err 0)

these types of errors usually indicate that modules were built against the wrong kernel version. in my case kernel has not changed in a couple of months, and the sources were current. after much soul searching i recalled that since then system gcc was upgraded from sys-devel/gcc-6.4.0-r1 to 7.3.0-r3. que procedure outlined in https://wiki.gentoo.org/wiki/Upgrading_GCC and warnings about ABI incompatibility.

the problem was resolved by recompiling kernel with gcc-7.3 and rebooting into the new kernel. after that modules loaded fine.

note that revdep-rebuild will not catch this as kernel sources are not compiled nor installed by emerge.

i guess gcc upgrade guide could have mentioned this.

this concludes today's public service announcement.

[asturm]

Looks like an nvidia-driver specialty.

[nsoveiko]

i'm willing to bet that this affects other external modules too (e.g. app-emulation/virtualbox-modules)

[Marcih]

Thanks for the heads up and although I've recompiled my kernel after 7.3.0 got stabilised because I wanted full support for retpoline and few days after 4.14 got stabilised too (yay!), it's still useful for others to know, especially since there most likely is a number of people like you (or me) running Novideo with proprietary drivers.

[Josef.95]

Looks like an nvidia-driver specialty.

No, this is not driver specialty.
Ideally, the kernel image and modules should always be created with the same toolchain and compiler version.

[asturm]

Yes, other missing modules are less fatal though.

In this case however it means that nvidia-drivers module must have been rebuilt after GCC switch, without rebuilding the kernel +modules. That information is missing from the first post. That stuff does not break automatically, and rebuilding the kernel without rebuilding nvidia-drivers would be equally fatal.

It is thus not an advice applicable for everyone, mostly for those relying on external kernel modules to print stuff on their screens. Maybe also ZFS users, but I suspect they are aware of the issue.

[NeddySeagoon]

nsoveiko,

To generaise your topic title, the kernel and all of its modules must be built with the same gcc version.
Modules built with a different gcc won't load and the module loader will leave an error message in dmesg.

[krinn]

are you sure neddy? i didn't test it, but this would mean binary modules must match kernel version and gcc version ; making virtually almost impossible to have binary modules working no?

i think the real issue with what user has done is that he have build gcc with reptoline and rebuild nvidia-drivers with reptoline, and rebuild (or not) the kernel with reptoline, but didn't install that kernel.

And you endup with nvidia-drivers trying to use a symbol (x86_indirect_thunk) over a kernel that doesn't provide that symbol.

[asturm]

That is a good explanation for us not seeing such reports after every GCC update...

[NeddySeagoon]

krinn,

I thought that the gcc version was in the "version magic" that that module loader checks.
Looking at a random module, it not there.

Code: Select all

retpoline:      Y
intree:         Y
name:           snd_hda_intel
vermagic:       4.17.1-gentoo SMP preempt mod_unload modversions 

Good call.

[nsoveiko]

i think the real issue with what user has done is that he have build gcc with reptoline and rebuild nvidia-drivers with reptoline, and rebuild (or not) the kernel with reptoline, but didn't install that kernel.

i didn't do any particular configuration for gcc, if 7.3 was built with reptoline, then it must have been the new defaults.

what actually happened was:

• kernel was updated to gentoo-sources-4.4.128 on april 26 with current system compiler (which at the time was sys-devel/gcc-6.4.0-r1); nvidia-drivers were rebuilt for the new kernel at the same time
• sys-devel/gcc was updated to 7.3.0-r3 on june 28, everything done by the book
• on luly 3, a @world update saw there's a newer version of nvidia-drivers in the tree and installed it
• after trying to modprobe the new nvidia module onto kernel compiled with gcc 6.4.0 got errors posted at the beginning of this thread

and i'm sorry i did not make it clear in the original post

btw, today i encountered another case of ABI incompatibility: media-video/vlc-3.0.3-r1 refused to build until i've rebuilt dev-libs/libebml-1.3.5 with gcc-7.3, see viewtopic-t-1077132-highlight-vlc.html for a similar problem

[NeddySeagoon]

nsoveiko,

The kernel builds with reptoline by default, when gcc supports it.
reptoline was new in gcc-7.3

Here you have your old kernel and out of tree modules built with gcc-6.x, (no reptoline)
Then your out of tree nvidia module was built with reptoline (gcc-7.x)
Thats the kernel build system applying reptoline when it can.

There are some options that must be common between the kernel and all of its modules, reptoline must be one of those.

[nsoveiko]

The kernel builds with reptoline by default, when gcc supports it.
reptoline was new in gcc-7.3

so the topic title should refer to gcc 7.3 only, not to any major gcc upgrade, right?

[NeddySeagoon]

nsoveiko,

In this case yes. It was only an issue for you as you updated an out of tree module (without the rest of the kernel) and happened to get an updated gcc meanwhile.
The reptoline capability was introduced in gcc-7.3. The kernel build system changed at about 4.15.0 but the reptoline patches have been back ported too.