rc_logger saves no messages on shutdown if /var is mounted

[cz0]

Hi folks.

I just noticed that /var/log/rc.log contain no messages about system shutting down.

I was replacing platform on my aged headless home server and noticed some "Unable to.." red asterisk warning during reboots and checked /var/log/rc.log to investigate what is wrong finding nothing related to shutdown process. Even with monitor connected the error messages glimpses just way to fast to be recognized, but for the headless system it literally leaves no way to figure out that something goes wrong during shutdown process.

Am I misunderstanding something or we need shutdown/stop process also be reflected in rc.log?

P.S. I'm not quite sure, but I remember rc.log to have stopping messages. I'm not able to find any evidence in backup files, but I remember to investigate some random reboots (they came out to be power related) and remember rc.log to have messages the system is stopping making it clear crash/power reboot from a regular one.

Any ideas here?

[Goverp]

Just checked; my system's /var/log/rc.log has a section starting "rc shutdown logging" that contains all the output I'd expect, right down to "stopping udev".

[b11n]

old sysadmin trick: use a camera phone to capture error messages that only appear briefly. Another possibility is a serial console for a headless machine.

[cz0]

Just checked; my system's /var/log/rc.log has a section starting "rc shutdown logging" that contains all the output I'd expect, right down to "stopping udev".

Thank you very much, I was suspecting myself having a dementia with early beginning. Not this time, heh.

Now I would like to figure out how to get the same output cause I have this:

Code: Select all

# grep -i 'shutdown' /var/log/rc.log*
#

Added:
I also recalled that I have /var mounted as a separate partition:

Code: Select all

/dev/sda5               /var            ext4            relatime,nodiratime             0 1

Could this be the cause for rc.log having no shutdown messages because of /var been unmounted yearly?

[Goverp]

Bit short of ideas here. Maybe /var/log is on a mount that's getting removed or R/O'd by some early step in the shutdown, or disk full? Or maybe something gets mounted over the top of /var/log, so openrc is writing to a log file that becomes invisible. Or just rc_logger = "no", but then you'd have no logs at all...

[cz0]

Bit short of ideas here. Maybe /var/log is on a mount that's getting removed or R/O'd by some early step in the shutdown, or disk full? Or maybe something gets mounted over the top of /var/log, so openrc is writing to a log file that becomes invisible. Or just rc_logger = "no", but then you'd have no logs at all...

Yap, mount is exactly my guess after some investigation (see my edited message). Maybe there is a way to prevent /var been unmounted that early?
On the other hand, I always had /var as a separate partition and remember to have shutdown logs ever since I set up this system in late 2000.

Added:
And I'm starting to suspect the "Unable to.." messages are actually /var related. Something like "Unable to unmount /var" would be my guess.
Looks like something changed since the system was set up (well, 15+ y.o. things are expected to have "something changed") causing this issues.

[cz0]

So, yes, this is related to /var been mounted as separate partition.
Redefined rc.log path with rc_log_path parameter and since then all shutdown messages are present in rc.log with no "Unable to.." error message at the same time. Both my guesses are valid at the same time.

But I remember to have this working with /var been mounted as partition separately from root fs. No doubts here. And I always had separate /var on this servers (since 2009, AFAIR).

It looks like OpenRC behavior changed now causing shutdown logs get lost and errors appear during shutdown process if /var is mounted as a partition.
Here comes last lines from the shutdown process:

Code: Select all

 * Stopping syslog-ng ...
 [ ok ]
 * Unmounting loop devices
 *   Remounting /usr/portage read only ...
 [ ok ]
 * Unmounting filesystems
 *   Unmounting /mnt/extended ...
 [ ok ]
 *   Unmounting /var ...
 [ ok ]
 * Deactivating swap devices ...
 [ ok ]
 * Setting hardware clock using the system clock [Local Time] ...
 [ ok ]
 * Stopping udev ...
 [ ok ]

[cz0]

It looks like the localmount is guilty as it has no /var/log in no_umounts_r and RC_NO_UMOUNTS variables or something.
Maybe it has it previously that allowed the rc.log be located on a separate partition from / and make no problems at the same time.

Some OpenRC experts wanted.

[cz0]

Filled a bug report on OpenRC Github project page: https://github.com/OpenRC/openrc/issues/977.