[SOLVED] Migration to systemd: Flood from systemd-journald

[lcj]

I'm using ~x86, latest updates form this week, latest kernel:

Using the Gentoo Wiki, I've migrated to systemd. OpenRC is still installed, I managed to recompile all with -consolekit systemd.

Configured kernel - prete much line-by-line what wiki says
Using lilo as boot manager I've added the init line, just as the wiki says.
Now after reboot, system boots but is on console immediately flooded with following message (the first might be slightly different I had to record hi-speed video of the console to get this):

[ulenrich]

You could add to your kernel cmdline (grub or lilo)

[TomWij]

Try increasing the kernel message log buffer length with the kernel parameter log_buf_len, for example: log_buf_len=262144

[lcj]

Wow, the obvious thing of enabling kde helped, thanks ulenrich, my bad for not noticing that!

I see that NetworkManager is now talking to dbus, and I can controll it!
The VPN started to work too, bu it seems that passwords were not remembered in WiFi, but that's complety different thing.

TomWij - I still saw the messages...

I did recompile the kernel and changed the default 16=64k to 18...

system-journald is taking up to 95% of the CPU core, load is @ 1,53.
What I can see is that the cat /proc/kmsg shows repetetive power_supply BAT0: uevent

I'll investigate tomorrow and ask it to log something via syslog-ng and via the suggested lilo line, and remove the acpi debugs...

Big thanks - it's first time since 2004 that my problem was not already solved (using search) by the users of this forum.
_________________
--
Lukasz C. Jokiel via web

[TomWij]

[lcj]

The battery just last month started to complain because it's around 3 years old, so Dell utility says after every boot: Battery may load normaly, but it's performance might be decreased over time.

So after removing the support from kernel: battery & AC power supply my kernel log is no longer overflooded. Thanks.
_________________
--
Lukasz C. Jokiel via web

[Barvinok]

Having the same problem.
What if I need kernel support for the power supply and battery? How do I just stop journald flooding or redirect it elsewhere?
_________________
Unlike reality, stupidity is inescapable

[ulenrich]

[lcj]

I can suggest:

#1 Try removing the battery and check the status
#2 Expand the kmsg log buffer in kernel beyond 256K
_________________
--
Lukasz C. Jokiel via web

[ulenrich]

[lcj]

Yes, I'm @

[ulenrich]

[lcj]

Well - not that I complain, but flood is probably not what we would like to get. Windows reports battery getting old once, maybe the interpretation of the ACPI event from this particular vendor (DELL) should be supressed after all.
I will investigate further but I think we should not continue this thread - the actual problem IS SOLVED.
_________________
--
Lukasz C. Jokiel via web

[Barvinok]

[ulenrich]

[Barvinok]

Okay, now is the time for some ACPI mystery.
I was playing with journald config on this machine while ac and battery modules were turned off from the kernel. After that I re-enabled ac and battery modules only to see there's no more uevent flood. Worse still, the flood didn't reappear when I reverted journald settings back to previous. So now the issue on Dell Inspiron seem fixed but I still don't know how to fix it.

For next experiments, I took much newer laptop with stock battery, this time ASUS K52N, with exactly the same previous Gentoo setup and also in transition to systemd and GNOME3. It also flooded in the exactly same way as Inspiron did before, only this time I managed to take a peek at it:
1. modprobe ac results in the following flood which continues infinitely:

[Barvinok]

Here's my latest discoveries on the issue.
I'm no longer blaming systemd, this seems to be some specific hardware issue unaccounted for in the kernel.
Stuffing dump_stack() in various places of kernel source, namely at power_supply and battery modules, I came to see that the uevent flood is reported by power_supply_uevent() function called by this:

[lcj]

@Barvinok:

So it's not related to actual hardware problems ? I do know that I had connected PSU for this Dell that wasn't enough (65W should be 90W) and the battery is not BAD yet.
But it seems this ACPI flood is unrelated to actual hardware issues.
_________________
--
Lukasz C. Jokiel via web

[Barvinok]

[lcj]

I agree, elevate this!
_________________
--
Lukasz C. Jokiel via web

[Navar]

[ulenrich]

@Navar,
my "solved" statement refered to this message just two above it:

[Navar]

@ulenrich,
Let's not presume what I may prefer. My argument was not against the importance of system logging, on the contrary. Instead, seemingly non-existent rate limiting on messages with a syslog function process acting, frankly, out of control on reporting duplicate events is the issue. It's hardly the first time a freedesktop.org 'invention' has shown race conditions and run away event handling (evolution, pulseaudio, nautilus, etc.). Hey, it's software development, stuff happens. But why it always has to be yet another not-ready-for-prime-time affair with multi-year userbase 'testing' is beyond me.

And if we're going there (systemd), which we're not--I'm clarifying my stance on this particular issue. I personally loathe the Redmond Way binary closed format of system and event logging rather than the simple de facto text output we're used to that many simple, non-proprietary, tools can easily process. When have obscure closed binary formats ever been a good idea for system logs?

• When we care less about immediate performance and simplicity and more about large stores of metadata (now the metaphor on function is less a log and more a database with associated problems).
• When there is money to be made off forcing your users to learn your particular proprietary tools for any access and recovery.
• And apparently due to freedesktop documentation, when there is some burning need to store large chunks of binary data... what are we storing core file data now?

Have I utilized the systemd way of logging? Thankfully, no, but I've read enough frustrations of others it has been foist upon and screen shots displaying the situation to know I'm wary.

As convoluted and massive in scope as XML related technologies are, they're still far and away better in helping data exchange than dealing with binary formats. Efficiency is their main weak point. The only arguments for binary data have been efficiency and obscurity. If systemd were producing XML output (as an option) along with plain text like existing loggers do, I could see both sides. Hell, even if the option to opt out of binary journal use with their proprietary database scheme for simple text writes would be ok. But those options don't exist. In fact, the freekdesktop documentation strongly 'recommends' using their tools only rather than bothering to absorb their C API to produce your own parsing tools.
_________________
(I don't speak for/represent my current employer)

[TomWij]

[TomWij]