TSC unstable! How to update firmware?

Message

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 2:59 pm

I was getting the error message on warm boot:

clocksource: timekeeping watchdog on CPU4: Marking clocksource 'tsc' as unstable because the skew is too large:
[    2.589702] clocksource:                       'hpet' wd_nsec: 495453193 wd_now: 225c7ac wd_last: 1b988c8 mask: ffffffff
[    2.589706] clocksource:                       'tsc' cs_nsec: 503192819 cs_now: 347d06d8f cs_last: 308f28ac5 mask: ffffffffffffffff
[    2.589710] clocksource:                       'tsc' is current clocksource.
[    2.589739] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[    2.589934] clocksource: Checking clocksource tsc synchronization from CPU 0 to CPUs 1-2,4-5,7,9.

on Lenovo ideapad3 15ALV6 Ryzen 5 5500U. The problem is that hpet is then used as the clocksource, instead of tsc
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet
which is not what I want since tsc is using cpu registers instead of memory so it is much faster.
I have found this thread https://bugzilla.kernel.org/show_bug.cgi?id=216166, where the same error occurs on Ryzen processors and there it is suggested the problem is in bios. I have played around a bit though and have figured out the problem is in deep idle states, which is where tsc breaks. I am not sure why, but from what I understand, registers tsc is using are not designed to be the system clocksource, which is why tsc is not as reliable as hpet. I guess deep idle states are one of the things that break it. Since I have put in my /etc/default/grub

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="processor.max_cstate=1 amd_pstate=passive"

error does not occur anymore and tsc clocksource is being used.

Code: Select all

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource 
tsc

I do not have a proper understanding of the underlying issue and I am worried this might only push the problem somewhere else or sacrifice the performance somehow. Does anybody know is this acceptable solution or I should definitely update the appropriate firmware? What I understood from the mentioned thread, the version of the SMC should be at least 55.93.0, while I have

Code: Select all

$ cat /sys/kernel/debug/dri/0/amdgpu_firmware_info | grep -i smc
SMC feature version: 0, program: 0, firmware version: 0x00375100 (55.81.0)

However, I have no idea how to update this. Lenovo support page offers only one update for my ideapad3 and it is an exe file!!! What am I supposed to do with an exe file? Moreover, it is not said what is included in that exe file and I do not feel comfortable installing something I do not know what it is. I do not even want bios update apart from SMC, unless necessary. So, my question would be does anybody please know how to update this? Do I have to make a windows partition just to do this and then delete it??? I tried using fwupd but it did not find any firmware updates for my devices.

logrusx · Post by **logrusx** » Sat Oct 28, 2023 5:21 pm

Which kernel is that?

I used to have problems with 5.x but now I don't see anything related in dmesg

Legion 5 Pro 16ACH6H here with Ryzen 7 5800 H.

Also what does `dmesg | grep firmware` say? Do you have the latest firmware?

What does `equery list linux-firmware` say?

Do not update bios unless you want something like "connected standby" or other crap introduced.

Best Regards,
Georgi

Post by **NeddySeagoon** » Sat Oct 28, 2023 5:39 pm

Freedom1947,

Some of the deeper sleep states interfere with the tsc. It stops running.
The work around is to not use those sleep states.

Downloading the exe file is harmless.
Then you can use the file command to see what sort of file it actually is.

Once that is determined we can work out the next steps.

My curionity got the better of me.

Code: Select all

$ file downloads/gkcn60ww.exe 
downloads/gkcn60ww.exe: PE32 executable (GUI) Intel 80386, for MS Windows, 10 sections

So, yes, you need Windows to use that file.

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 5:52 pm

Hi Georgi,

Which kernel is that?

I used to have problems with 5.x but now I don't see anything related in dmesg

Legion 5 Pro 16ACH6H here with Ryzen 7 5800 H.

It is my custom kernel based on 6.1.57. I tried with the original 6.1.57 as well and the error was the same.
It is not up to kernel to deal with shit like this and kernel developers said they even do not have a workaround for this problem.

This is from 'dmesg | grep -i firmware'.

Code: Select all

$ dmesg | grep -i firmware
[    0.213634] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.225772] acpi PNP0A08:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge
[    0.774113] Loading firmware: regulatory.db
[    0.774904] Loading firmware: regulatory.db.p7s
[    0.867961] Loading firmware: rtw88/rtw8822c_fw.bin
[    0.868432] Loading firmware: rtw88/rtw8822c_wow_fw.bin
[    0.869437] rtw_8822ce 0000:01:00.0: Firmware version 9.9.4, H2C version 15
[    0.869598] rtw_8822ce 0000:01:00.0: Firmware version 9.9.15, H2C version 15
[    0.927356] Loading firmware: amdgpu/renoir_sdma.bin
[    0.928344] Loading firmware: amdgpu/renoir_asd.bin
[    0.929081] Loading firmware: amdgpu/renoir_ta.bin
[    0.929380] Loading firmware: amdgpu/renoir_dmcub.bin
[    0.930043] [drm] Loading DMUB firmware via PSP: version=0x01010027
[    0.930053] Loading firmware: amdgpu/renoir_pfp.bin
[    0.930260] Loading firmware: amdgpu/renoir_me.bin
[    0.930456] Loading firmware: amdgpu/renoir_ce.bin
[    0.930835] Loading firmware: amdgpu/renoir_rlc.bin
[    0.931131] Loading firmware: amdgpu/renoir_mec.bin
[    0.932553] Loading firmware: amdgpu/renoir_vcn.bin
[    0.933368] [drm] Found VCN firmware Version ENC: 1.20 DEC: 6 VEP: 0 Revision: 0
[    0.933376] amdgpu 0000:03:00.0: amdgpu: Will use PSP to load VCN firmware

And my firmware should be up to date:

Code: Select all

$ equery list linux-firmware
 * Searching for linux-firmware ...
[IP-] [  ] sys-kernel/linux-firmware-20230919:0

Do not update bios unless you want something like "connected standby" or other crap introduced.

I know, that is why I want to avoid it. But if there are serious drawbacks to using 'processor.max_cstate=1', I'll have to use hpet. Ideally, I would want to update smc only.

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 5:57 pm

Hi NeddySeagon,

thanks for the fast reply.

Some of the deeper sleep states interfere with the tsc. It stops running.
The work around is to not use those sleep states.

But are there any side effects of not using the deeper sleep states or their only purpose is to preserve more power?

So, yes, you need Windows to use that file.

So, if I want to update this, creating a windows partition is the only way?

Post by **NeddySeagoon** » Sat Oct 28, 2023 6:06 pm

Freedom1947,

Short answer ... yes.

Longer answer ... as its a 32 bit file, it may run under WINE. However, run is not equal to work.
It's a very bad thing to 'brick' your system due to a BIOS update not working.

The BIOS is not the same as the firmware loaded by the kernel and used by bits of your hardware to make it work.

The BIOS is all that there is for your CPU to execute at power on.

Yes, the main purpose of sleep/low power states is to reduce power consumption to extend battery life.

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 6:21 pm

Hi NeddySeagon,
thanks again for the clarification.
Can I ask you do you think this is worth updating the whole bios? And how big the difference between 'processor.max_cstate=1' and 'processor.max_cstate=2' is; I mean, I do not have even a slight feeling of how much power each deeper state preserves?
Would you personally rather have this option set or hpet being used over tsc?

Post by **NeddySeagoon** » Sat Oct 28, 2023 6:49 pm

Freedom1947,

Read the change note on the BIOS download page.

If it claims to fix the things you want it to fix, then maybe.
The TSC probably won't be fixed by a BIOS change. That's internal to the CPU. As its a CPU thing, the kernel can do that It's called CPU microcode. No need for a whole BIOS update.
BIOSes can contain CPU microcode updates too.

Your CPU microcode is provided by sys-kernel/linux-firmware

Its one of these. if its is linux-firmware

Code: Select all

$ ls /lib/firmware/amd/
amd_sev_fam17h_model0xh.sbin  amd_sev_fam17h_model3xh.sbin  amd_sev_fam19h_model0xh.sbin  amd_sev_fam19h_model1xh.sbin

Code: Select all

$ dmesg | grep microcode
[   10.038734] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[   10.038736] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[   14.093128] microcode: CPU2: patch_level=0x0a201025
[   14.093131] microcode: CPU3: patch_level=0x0a201025
[   14.093139] microcode: CPU0: patch_level=0x0a201025

It looks like I have a problem there. It should say ... microcode update early.
That line will not appear if there is no microcode update to apply

Microcode updates need to be built into the kernel or in an initrd. See the Wiki page.

The microcode is what tells the CPU how to execute each individual instruction.

logrusx · Post by **logrusx** » Sat Oct 28, 2023 7:19 pm

@Freedom1947

I'm sorry, I made a mistake, I meant `dmesg | grep microcode`
...

Best Regards,
Georgi

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 7:30 pm

I have the appropriate microcode built into the kernel since I am not using initramfs. This is my output:

Code: Select all

$ dmesg | grep -i microcode
[    0.082565] Zenbleed: please update your microcode for the most optimal fix
[    0.269770] microcode: CPU0: patch_level=0x08608103
[    0.269790] microcode: CPU1: patch_level=0x08608103
[    0.269806] microcode: CPU2: patch_level=0x08608103
[    0.269825] microcode: CPU3: patch_level=0x08608103
[    0.269840] microcode: CPU4: patch_level=0x08608103
[    0.269859] microcode: CPU5: patch_level=0x08608103
[    0.269874] microcode: CPU6: patch_level=0x08608103
[    0.269893] microcode: CPU7: patch_level=0x08608103
[    0.269907] microcode: CPU8: patch_level=0x08608103
[    0.269917] microcode: CPU9: patch_level=0x08608103
[    0.269927] microcode: CPU10: patch_level=0x08608103
[    0.269945] microcode: CPU11: patch_level=0x08608103
[    0.269960] microcode: Microcode Update Driver: v2.2.

The Zenbleed message is in regards to the vulnerability that has not been addressed by AMD for all processors yet. For my 5500U, it is planned to be resolved in December this year https://www.amd.com/en/resources/produc ... -7008.html. It is therefore not included in the linux-firmware https://git.kernel.org/pub/scm/linux/ke ... ode/README (mine would be 'Model=0x68'). This should not have anything to do with tsc and apart from that, I think my microcode is loaded correctly. The reason I think this has to do with bios update is because of what an AMD developer said on the aforementioned forum:

this is what it says:

root@akarin:/sys/kernel/debug/dri# cat 0/amdgpu_firmware_info | grep 'SMC'
SMC feature version: 0, program: 0, firmware version: 0x00374700 (55.71.0)

Comment 22 Mario Limonciello (AMD) 2023-08-23 21:30:50 UTC

OK. That's too old, it doesn't pick up the fix. you'll need a BIOS upgrade.

Now, what I do not understand is how to get the bios upgrade he is referring to. Should I contact lenovo or it might be included in the official microcode in December?
I am sorry for mixing firmware, microcode and bios. I know the difference clearly, while it does make sense to me that this concerns the microcode, I found this comment from a kernel developer https://bugzilla.kernel.org/show_bug.cgi?id=216967:

The CPU vendor doesn't matter - it is the OEMs who don't care about their laptops supporting Linux properly. I, myself, wouldn't take ASUS hardware even for free, judging by past experience.

What you could do next time you buy, is search the net whether someone else has run Linux on that hw and what her/his feedback about it is. Or, if you can get your hands on the machine you wanna buy, you boot a live CD on it and stare at dmesg and see whether everything gets detected properly and so on.

I have a Zen2 laptop here from Lenovo and a perfectly fine TSC which is simply f***ed by the OEM BIOS. Guess what'll happen the next time I have to buy a Lenovo machine...

I'm sorry but this is the reality, unfortunately.

logrusx · Post by **logrusx** » Sat Oct 28, 2023 7:42 pm

Bios update and microcode update are two separate things. AFAIK, a microcode update might be included in a bios update, however I might be wrong on that.

The comment is for a different CPU and device.

You're alright running with tsc. To my knowledge, from when I was searching for information on the same issue, it only has a minor impact on battery life. Maybe less significant that a single cstate as you've currently instructed your cpufreq governor to work.

Best Regards,
Georgi

Post by **NeddySeagoon** » Sat Oct 28, 2023 7:44 pm

Freedom1947,

Code: Select all

root@akarin:/sys/kernel/debug/dri# cat 0/amdgpu_firmware_info | grep 'SMC'

That's not related to the tsc issue, that's looking at amdgpu_firmware_info, which is GPU related.

Freedom1947 · Post by **Freedom1947** » Sat Oct 28, 2023 7:52 pm

That's not related to the tsc issue, that's looking at amdgpu_firmware_info, which is GPU related.

Well, the issue described there is the same as mine and only on warm boot as well. It was weird for me as well but I just assumed it might have to do something with integrated graphics. Nobody is talking about gpus in that thread. Cpus mentioned there are not the same but are very similar: I assume 4500U is quite similar to 5500U and the appearance of the same error message might have the same cause.

You're alright running with tsc. To my knowledge, from when I was searching for information on the same issue, it only has a minor impact on battery life. Maybe less significant that a single cstate as you've currently instructed your cpufreq governor to work.

How do you mean running with tsc without using single cstate. Without single cstate, it falls to using hpet. How can I make it use tsc otherwise? Have I misunderstood something?
Thanks for the reinsurance though, I am almost always on adapter anyways. I just didn't know if it is only power related or there are performance impacts as well.

logrusx · Post by **logrusx** » Sun Oct 29, 2023 5:42 am

Freedom1947 wrote:
You're alright running with tsc. To my knowledge, from when I was searching for information on the same issue, it only has a minor impact on battery life. Maybe less significant that a single cstate as you've currently instructed your cpufreq governor to work.
How do you mean running with tsc without using single cstate. Without single cstate, it falls to using hpet. How can I make it use tsc otherwise? Have I misunderstood something?
Thanks for the reinsurance though, I am almost always on adapter anyways. I just didn't know if it is only power related or there are performance impacts as well.

Sorry, I didn't mean tsc, I meant hpet. This is what information I got back when the kernel was reporting the same thing for my CPU. But it doesn't anymore and I don't think it's because a microcode update because there hasn't been any.

From what I read the kernel's requirements were too tight and they were talking about relaxing them. They were actually tightened too much in 5.15 I think, so they were stepping aback on that.

So back when I was searching for information, the main argument was that with a more precise timer, some battery power would be spared, but not much.

By the way why do you insist on using it?

Best Regards,
Georgi

Freedom1947 · Post by **Freedom1947** » Sun Oct 29, 2023 10:45 am

Hi Georgi,
I am very far from expert but I think you might be wrong. I think the clocksource is not directly related to power consumption, it is just the clock which tells the os what time it is (and thus how to measure it). Now, the clock does affect the power consumption in a way that a high precision clock allows for more efficient usage of cpu (so, as you said, some battery power would be spared), but that is not the purpose of a clock and both tsc and hpet are high resolution clocks. Hpet is a physical chip in the southbridge and communicates with your os via acpi. Tsc is a cpu register which counts the number of cpu cycles since its reset, which means it is extremely hot and has a very high resolution on the expense of reliability, since there is no guarantee that different cpu cores will have the same value in their tsc register. I guess the power saving measures mess this up since they change the value of tsc registers and cpus can thus stop being synchronized. I think kernel developers have a way to deal with this issue, but tsc registers in theory should not have different values on warm boot and this is thus considered a bug on the side of AMD, but there is also a possibility that bios is writing to the register incorrectly, which makes tsc unsynced from the beginning of the boot process. From what I understood from kernel bugzilla, AMD did solve this bug but I am not clear how did they distribute it. So, will this fix be included in microcode blobs or in the bios, I do not know.

Why do I want tsc? Well, because I care about performance more than about stability. It is not that I need that possible performance boost, but I hate to settle knowing there is a more elegant solution and, in my opinion, tsc is waaay more elegant way to measure time then hpet. I do not like unnecessarily using memory outside cache in general and I always make sure that software I make prefetches all the memory that is going to be used (hardware does this pretty good but there are a lot of ways one can improve it). I believe tsc can give a very significant performance boost over hpet (but that is just my humble opinion) and I am willing to invest some extra effort even if I don't need it, rather than settle for hpet. I also think tsc usage should not have reliability problems but bad software, like, for example, windows os, does, so they rather use something worse but more stable in order to have more room for making mistakes and bad software. And this is true in many other situations. Eventually, when such a messy software hits a wall and breaks or becomes incompetent, they release patches which usually just prolong the problem and make the software even messier. And normies buy it... Such is the circle of incompetence. Maybe I am wrong about this particular problem but this is a general tendency. Sorry for the digression.

Best regards!

logrusx · Post by **logrusx** » Sun Oct 29, 2023 11:49 am

I think you have a fixation on a problem that's not worth solving.

Best Regards,
Georgi

Goverp · Post by **Goverp** » Sun Oct 29, 2023 12:55 pm

An aside to the above discussion. I note there's a new BIOS for my AMD64 motherboard which is based on "ComboV2PI 1.2.0.B", whch fixes Inception and ZenBleed vulnerabilities properly, and should recover the performance lost to the kernel bypasses. I'm giving it a month, as usual, before installing. Note that, AFAICT, firmware fixes loaded from the kernel don't address the bugs in the same way, and the kernel still uses the bypass whatever firmware you have, because it chooses mitigations for the vulnerabilities before running the firmware update (see dmesg).

FWIW, I've been updating my BIOS pretty regularly, because the motherboard was pretty leading-edge when I bought it, and so there's been a steady stream of improvements.

I'm not sure whether this is worth a separate thread.

logrusx · Post by **logrusx** » Sun Oct 29, 2023 3:10 pm

Goverp wrote:An aside to the above discussion. I note there's a new BIOS for my AMD64 motherboard which is based on "ComboV2PI 1.2.0.B", whch fixes Inception and ZenBleed vulnerabilities properly, and should recover the performance lost to the kernel bypasses. I'm giving it a month, as usual, before installing. Note that, AFAICT, firmware fixes loaded from the kernel don't address the bugs in the same way, and the kernel still uses the bypass whatever firmware you have, because it chooses mitigations for the vulnerabilities before running the firmware update (see dmesg).

FWIW, I've been updating my BIOS pretty regularly, because the motherboard was pretty leading-edge when I bought it, and so there's been a steady stream of improvements.

I'm not sure whether this is worth a separate thread.

Thanks for mentioning. I find it related. Now you got me wondering if it's worth the risk of losing something like S3 sleep if I update. I think there was someone on the forum with the same platform laptop as me, but with newer EFI firmware than mine who had troubles sleeping their laptop. That'd be really unpleasant if it happens.

Best Regards,
Georgi

Freedom1947 · Post by **Freedom1947** » Mon Oct 30, 2023 11:33 am

@logrusx

I think you have a fixation on a problem that's not worth solving.

If you do not think help is necessary, you do not have to comment.
Also, please do not call work all the people have put into solving this problem worthless.

logrusx · Post by **logrusx** » Mon Oct 30, 2023 2:20 pm

Freedom1947 wrote:@logrusx
I think you have a fixation on a problem that's not worth solving.
If you do not think help is necessary, you do not have to comment.
Also, please do not call work all the people have put into solving this problem worthless.

Hey, I don't know what your native language is, nor how proficient you are in English, but mine definitely isn't English and my English is so called work level - enough to transmit information, very likely not enough to do it properly from a social perspective.

Having said that, I hope you understand I didn't mean any offence.

Now on the point - I did my research on that problem back when I had it and concluded it's not worth solving. The perceived power and performance gains are of little significance.

Of course it's your right to continue putting time and effort, just take it easy when you're presented with an opinion. Life is too short to waste your time taking it seriously.

And do not put words in my mouth please. I never said anybody's work is worthless. It's easy to see how worthy that work is for a data center or for high speed computing cluster, but for a laptop it can't be serious.

Best Regards,
Georgi

Post by Hu » Mon Oct 30, 2023 2:53 pm

Freedom1947 wrote:@logrusx
I think you have a fixation on a problem that's not worth solving.
If you do not think help is necessary, you do not have to comment.
Also, please do not call work all the people have put into solving this problem worthless.

I interpreted logrusx's comment as an attempt to help save your time, by encouraging you to abandon a problem that is consuming significant amounts of your time and which, even if solved, will likely take a long time (if ever) to pay back the time spent so far. If it cannot be solved because it relies on software updates that do not exist, then you could spend an unlimited amount of time trying to solve this.

Post by **pietinger** » Mon Oct 30, 2023 3:15 pm

logrusx wrote:
Freedom1947 wrote:@logrusx
I think you have a fixation on a problem that's not worth solving.
If you do not think help is necessary, you do not have to comment.
Also, please do not call work all the people have put into solving this problem worthless.
Hey, I don't know what your native language is, nor how proficient you are in English, but mine definitely isn't English and my English is so called work level - enough to transmit information, very likely not enough to do it properly from a social perspective.

Having said that, I hope you understand I didn't mean any offence.

[...]

logrusx,

I believe you that you did not want to offend @Freedom1947, and yes, one can already come on the thought, something would not be meaningful; only one does not have to express this thought. The decision not to express something is independent of the language.

logrusx wrote:[...]

Now on the point - I did my research on that problem back when I had it and concluded it's not worth solving. The perceived power and performance gains are of little significance.

[...]

I write to you because I think that it would be much more interesting if you had presented the results of your investigations. I know you are trying to help many users here and I applaud that. But please do not forget that new users do not (can not) know you.

...

In general, I would like to note that - especially in written communication with people of different nationalities - it is always helpful to assume that the other person has written something only with the best of intentions.

Freedom1947 · Post by **Freedom1947** » Mon Oct 30, 2023 3:45 pm

I am not offended, I just do not think the comment was helpful since no proper arguments were presented.
In this particular case, I think quite a performance gain can be achieved with tsc and proper cpu sync.
I understand the point of your comment @logrusx and I do agree that for a lot of people even 20% of performance difference would not matter for basic tasks, but if we find a possible solution, a lot of people can get that possible performance gain basically for free.
If I did not believe this question was important, I would not bother with it in the first place. But mate, do not worry, you did not offend me, we are good

Will update you all when I solve this

logrusx · Post by **logrusx** » Mon Oct 30, 2023 4:26 pm

Freedom1947 wrote:no proper arguments were presented

The same holds true for the reasoning behind your motivation to solve the issue. That's why I first asked you about it, before making that statement. I needed to find out if you had some urgent need, but you provided none and therefore my conclusion you're wasting your time on something nice to have. It's fine, but is it worth? Maybe, who knows.

Freedom1947 wrote: Will update you all when I solve this

I'd appreciate that.

Best Regards,
Georgi