Kernel 6.1.81 smartd won't run; trace in dmesg [SOLVED]

figueroa · Last edited by figueroa on Tue Mar 19, 2024 3:12 am; edited 2 times in total

I've just upgraded two machines to gentoo-sources-6.1.81. On both machines now, upon boot, I get a trace that looks like the following:

figueroa · Posted: Thu Mar 14, 2024 3:33 am Post subject:

I greatly dislike answering my own question, but afraid I must. smartd (/etc/init.d/smartd) will not start under gentoo-sources-6.1.81.

Recompiling smartmontools didn't help. Rebooting to gentoo-sources-6.1.74 all back to normal.

This failure to start smartd happened on two VERY different systems. One system is an x86 Gigabyte mother board running an AMD Phenom(tm) 8650 Triple-Core Processor, and the second system is an x86_64 HP h8-1260t motherboard running an Intel(R) Core(TM) i7-2600 CPU.

I'm assuming many people are running this kernel, so I'm not calling it SOLVED. It's just solved for me for the time being.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

Hu · Moderator Joined: 06 Mar 2007 Posts: 21637

Although that failure to start is inconvenient for you, it is a very good thing for debugging. First, it means that this is a user-visible regression, and fixing those is considered second only to preserving security fixes. If you can find the specific commit that broke it, I think it highly likely you can either argue to get it reverted or argue to get a fix so that smartd works on future 6.1.x kernels. Second, since it fails to start, this is a relatively straightforward test case: (1) boot a suspect kernel; (2) try to start smartd; (3) check whether smartd started without error and without kernel warnings. This is much nicer than bugs where the kernel must run for hours before it fails, or where a failure happens only once out of every several reboots.

To your specific problem: Normally, the next step would be to try to find the specific commit (not just kernel release) that broke this. I see 1512 commits present in v6.1.81 and not present in v6.1.74. If we assume this was broken upstream, rather than by a Gentoo-specific patch, then we need to find which of those 1512 commits is at fault. A straight git bisect will need log2(1512) =~ 11 steps to find this. However, after the first draft of this post, I did some basic analysis of the faulting function, and I see the WARN introduced in a commit that is present in v6.1.81 and absent in v6.1.74. Therefore, without any attempt to understand what the code is meant to do, I posit that this check is just wrong as committed. The WARN was introduced in scsi: core: Add struct for args to execution functions, which makes it v6.1.80-4-gcf33e6ca12d8. That is, it is the fourth commit added after 6.1.80, so I expect you would find 6.1.80 to be good and 6.1.81 to be bad. It may (or may not) be the case that the corresponding upstream commit was correct as written, due to a change present in the later kernel that is absent in 6.1. That would require either testing a newer kernel series, or getting input from someone who understands this code and can explain what the check was meant to prevent.

figueroa · Posted: Thu Mar 14, 2024 3:44 pm Post subject:

Thanks, Hu. That's a good lead and findings. I guess bugzilla is my friend.

ADDED 20240315: Reported: https://bugs.gentoo.org/927079 (date edited 20240323)
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

figueroa · Posted: Tue Mar 19, 2024 3:11 am Post subject:

The problem that began with an update to gentoo-sources-6.1.81 preventing smartd from running was resolved by upgrading the kernel to gentoo-sources-6.8.1.

You can follow the saga at the bug report https://bugs.gentoo.org/927079#c9 but the reason this upgrade solved the problem has not yet been revealed.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

Hu · Moderator Joined: 06 Mar 2007 Posts: 21637

figueroa · Posted: Sun Mar 24, 2024 3:32 am Post subject:

figueroa · Posted: Sun Apr 07, 2024 6:37 pm Post subject:

Hu recommended that I install kernel gentoo-sources-6.8.1 since I'd found that 6.1.81 would not start smartd. Kernel 6.8.1 solved the problem and I've been running it for two weeks now without and issue. But thinking long term, what kernel series do I follow now to keep from being left behind? I'm not comfortable with and have no need on my older hardware to stay on the bleeding edge of kernel development.

Noticing that the newest LTS kernel is the 6.6 series, I merged, built, and installed gentoo-sources-6.6.21, the current stable LTS version. That seems to have hit the sweet spot for me. SMART (/etc/init.d/smartd) runs as intended, and dmesg reveals no important errors or warnings, and /var/log/boot shows all processes called for have started normally.

That is all. Original problem still solved.

_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi