syslogd error opening log file - SOLVED

[jankom]

/var/log files are reported as read only.
I've finally installed Gentoo trying to follow all instructions, 64-bit Desktop version after resolving a freetype and harfbuzz problem with the help of the Gentoo forum. However, when I boot I have 6 error messages before login:

syslogd ... error opening log file /var/log/... Read only filesystem /var/log/...

I don't understand since

Code: Select all

ls -ihl /var

or

Code: Select all

ls -iHl /var/log

will show that root does have access.

[cboldt]

Its possible that the entire filesystem whatever /var is part of being mounted on, is mounted as a read-only filesystem. This is a function of filesystem mounting, not a function of file permissions.

The command "mount" with no modifiers will show plenty of detail.

Being mounted "ro" (read only) instead of "rw" (read/write) will happen when there are issues confirming integrity of filesystem. In other words, a "ro" mount can happen even though the system is configured to make a rw mount.

To figure this out, of course more info is needed. What is your partition mounting arrangement? Either "df" or parts of /etc/fstab, or the output of the `mount` command.

[jankom]

Thx4 the quick reply.
The error shows up in the boot screen after starting sysklogd. I tried to do a

Code: Select all

touch /var/log/auth.log

but all 6 logging files, including auth.log were still reported as read only. Then I did an

Code: Select all

fsck /dev/sda2

from live CD boot, but the filesystem was clean.

Code: Select all

dmesg

does not show this error (because syslogd cannot function?). Anyway, here is my fstab:

Code: Select all

# /etc/fstab: static file system information.
# <fs>			<mountpoint>	<type>		<opts>		<dump/pass>

/dev/sda1		/boot		vfat		defaults	0 2
/dev/sda2		/		ext4		defaults,noatime		0 1
#/dev/cdrom		/mnt/cdrom	auto		noauto,ro	0 0

Note: (a) I have no swap because 8G ram memory is sufficient for my purposes,
(b) it had only the noatima, so added defaults to sda2 but made no difference;
(c) also tried w/o noatime, still no difference.

mount command shows:

Code: Select all

/dev/sda2 on / type ext4 (rw,noatime)

I configured and compiled the kernel myself. Could it be that

Code: Select all

sysklogd

needs some kernel options?

Otherwise my system seems to work OK (so far).
Janos

[cboldt]

Output of the 'mount' command shows that the problem isn't that the filesystem is mounted read only.

I assume when you say you get a report that the file or the folder is read only, that is literally what the report says. Maybe some extended attribute is set on a file or on /var/log. I don't think this is likely, for various reasons. the command `lsattr` is used to show extended attributes.

Can you write to a file there? Just open a file with an editor (for example, `nano /var/log/test-write.txt`) compose a novel, and see if it saves. Do that as root because the var/log directory is generally not regular user or world writable. Mine is ...

Code: Select all

cboldt@hypoid [5] 18 /home/cboldt > stat /var/log
  File: /var/log
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 806h/2054d      Inode: 2233        Links: 14
Access: (2771/drwxrws--x)  Uid: (    0/    root)   Gid: (   10/   wheel)

[Hu]

When the filesystem is reported as read-only, please show the output of cat -n /proc/self/mounts, at least for the line for that filesystem, but preferably the entire output. I wonder if OP has /etc/mtab as a real file, rather than a link to /proc/mounts, in which case the output of mount cannot be trusted when the root filesystem is read-only.

[GuntherDW]

Is this in a VM and/or a BIOS install? Because I have noticed similar issues when installing a "fresh" Gentoo in a KVM VM using a BIOS setup a couple weeks ago.

I don't know why, but these issues seemed to vanish when using UEFI boot methods instead of BIOS. That and/or using "rw" instead of "ro" in GRUB, but that's kind of dangerous.
Both methods were using the same methods of booting though. Both were using gentoo-kernel, dracut etc. The only major difference was BIOS booting and UEFI booting (and thus changing the partition table from MBR to GPT using gdisk).

The strange thing was that all of this showed up after the "remounting disks as r/w" message popped up.

I should do a new install to doublecheck but it might just be a fluke in a config file somewhere, or it just wasn't tested very well. Given that I used a ~arch install, including a stage1 bootstrap to get a ~amd64 toolchain before even starting to really install the distribution.

For clarification: the FS'es *were* mounted as read/write but maybe the command had a delay or was run too late or something, but by the time I could login it already "fixed itself". It was just rsyslog or syslog-ng and other services that were complaining about it.

[jankom]

@cboldt
Yes, I can write and save to /var/log/auth.log as root. I created that empty file when I did a touch command as root. But the sysklogd error is persistent after multiple boots.

@Hu
My /etc/mtab is a real file, not symlinked (using command ls -aiHl /etc/mtab and it shows read only for all and owned by root.
Output of cat --n /proc/self/mounts

Code: Select all

     1	/dev/root / ext4 rw,relatime 0 0
     2	devtmpfs /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=995700,mode=755 0 0
     3	proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     4	tmpfs /run tmpfs rw,nosuid,nodev,size=1593920k,nr_inodes=819200,mode=755 0 0
     5	sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     6	debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
     7	selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
     8	efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
     9	cgroup_root /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
    10	openrc /sys/fs/cgroup/openrc cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc 0 0
    11	none /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
    12	cpuset /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    13	cpu /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
    14	cpuacct /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0
    15	freezer /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    16	mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
    17	devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
    18	shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
    19	binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
    20	/dev/sda1 /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0
    21	/dev/sdb1 /mnt ext3 rw,relatime 0 0

@GuntherDW
This is a fitlet2 machine with SSD /dev/hda, EFI boot.

[GuntherDW]

Yeah, I just made that VM and it booted up just fine, I guess it was either a fluke or something to do with the state of portage/that stage3 initial install at the time.
But I can confirm at least that I have noticed it occurring, even to other people at the time. But whatever it was seems to be fixed now.

Either that or what happened to trigger said issue just didn't occur now. Just as a check though, if you do omit the "ro" (or change it to "rw") in GRUB, does the issue suddenly stop appearing?
It's not really safe to do, if there's any filesystem corruption before fsck gets to it it might cause other issues, but it might just rule out a couple avenues first.

EDIT: Hmm, after playing around in that VM a bit I haven't been able to trigger said bug yet. Looking at screenshots though rsyslog was getting triggered to launch right after localmount/fsck. Is this the case in your system as well?

screenshot : https://i.imgur.com/BgHKWth.png / https://i.imgur.com/YyJn99m.png

I'm almost ready to just attribute it myself to a borked stage3/1 tarball because I haven't been able to trigger it again just yet.

[jankom]

@GuntherDW
Tried eliminating ro from /boot/grub/grub.cfg and rebooted, but the issue is still there.

Interestingly, after playing with startx I noticed that in ~/.local/share/xorg/Xorg.0.log the "Kernel command line:" has BOOT_IMAGE=vmlinuz-5.10.61 ... root=/dev/sda2 ro even after rebooting without the grub ro and restarting X.

I compiled the kernel myself mainly to leave out drivers not needed for my machine and installed it without initramfs following standard procedures (make, make module-install, make install) then doing the grub thing.

[cboldt]

Booting is usually part ro, although I don't know the gory details. At some point during startup drives can be mounted rw, and are. The fact that you can write to a file implies the system is not read only, but something is reporting read only. This is a confusing pair of reports, they can't be reconciled. One of the reports is false or misleading.

The permissions on a file aren't changed by mounting rw vs. mounting ro - permissions look the same both ways, and from initial descriptions, sysklog is not complaining about permisisions, it says the file SYSTEM is ro.

I assume you've done the 'cat /proc/self/mounts' step to confirm the relevant mount is in fact rw. Assuming further, if the mount is rw, but sysklogd reports it is not, the issue is out of my league. Could be an artifact of some whiz-bang secure kernel (I know zero about selinux), or could be a sysklogd problem, but that I sort of doubt.

Edit to add - I see sysklogd has a "systemd" USE flag. Shooting in the dark, are you running systemd or some other itint, and second question, does your sysklogd USE flag there, correspond with your init system?

[jankom]

@cboldt
Yes, I did the cat /proc/self/mounts - see my previous post, and I do not use selinux (too complicated for me).
As far as the sysklogd USE flag is concerned: what is an easy way to find out that? I'm new to Gentoo and just learning the intriucacies of USE. My boot is not "systemd", I'm more familiar with the older init process.

Thanks again to all for your continued attention. - Janos

[cboldt]

Run `emerge -pv sysklogd`

The output of that will tell all the USE flags that pertain to the package sysklogd. Those with a dash/minus in front are not enabled on your install.

Edit to add - I don't know or remember for sure that you are using sysklogd. Maybe the package you installed is another of the logging packages. There are a few. Maybe you are using newsyslog or rsyslog or even something else. Just thinking that if the filesystem is writabable, the issue isn't the filesystem or how it is mounted. That means the issue is with the program trying to do the writing. I use syslog-ng, so would have to dig a bit to learn about other logging programs.

[jankom]

@cboldt

Code: Select all

USE="-logger -logrotate -systemd"

Additional info to all on this issue:
I cannot do a real screenshot (using the OS on the machine) only I could take a picture with a camera, BUT
here is a manual, and shortened copy of the messages appearing on console while booting. I stopped the screen with the "ScrollLock" key on my US keyboard.

Code: Select all

...
Starting sysklogd...    OK
syslogd[625]: Error opening log file: /var/log/auth.log
... <continued with similar errors>
...
...
Remounting root filesystem read/write...   OK
...

So it does look like the boot process initially mounts sda2 (where my / is kept) in ro mode. So it is not surprising that syslogd cannot write to sda2

Hope this helps.

Janos

[cboldt]

That does help. You are using open-rc to start, I think, so moving the place that sysklogd starts to AFTER the remount ought to cure the fail to write report.

`rc-status -a`will show all the services you have programmed to start, and whether they start as part of sysinit, boot, or default. My logging starts as part of the "default," and by this time the file systems are all writable.

As root, `rc-update` is the command that does service removal and service additions. I'm thinking ...

rc-update delete sysklogd sysinit (or rc-update delete sysklogd boot)
rc-update add sysklogd default

If there is some need or desire to log earlier events, then we get into kernel settings for kernel logging, and IIRC, openrc has some logging function for its startup, as well. Somehow I've managed to debug my way through startup issues without having to routinely log them - in other words, I'm satisfied with starting logging as part of default and not earlier.

Your observation here also reconciles up the "observations can't be reconciled" aspect. The filesystem IS ro when sysklogd tries to write, and it IS NOT ro when you inspect later, after startup is complete.

[jankom]

@cboldt
Yesss!
Case closed, error disappeared after I removed syklogd from run level sysinit. I did not have to add it to run level default because it was already there. Case closed.

unless

To all: is this a bug somewhere? I followed the instructions in installing the system, and service "sysklogd" was included twice. Maybe in "Installing system tools" section I entered command

Code: Select all

rc-update add sysklogd

without the default parameter. I remember doing something like this, but then I repeated the command with "default" in the command line.

So this may not be a bug, just my sloppiness.

Anyway, thx again you-all!

Janos