How to fix? EXT4-fs (nvme0n1p2): orphan cleanup on readonly

[GenIT]

Hi,

I compiled 6.12.74

I read Dracut is being too chatty. It mounts the root filesystem as Read-Only to check it, sees tiny leftovers in the journal (orphans), and complains because it can't "clean" them without write access.

So I tried running dracut as; dracut --force --no-hostonly

But in dmesg I keep getting these messages;

Code: Select all

[    2.548735] EXT4-fs (nvme0n1p2): orphan cleanup on readonly fs
[    2.549071] EXT4-fs (nvme0n1p2): mounted filesystem 99bcf0bd-7d3e-4940-8f1a-229efccbcf32 ro with ordered data mode. Quota mode: disabled.
[    2.557686] dracut: Remounting /dev/disk/by-uuid/99bcf0bd-7d3e-4940-8f1a-229efccbcf32 with -o defaults,noatime,ro
[    2.569180] EXT4-fs (nvme0n1p2): re-mounted 99bcf0bd-7d3e-4940-8f1a-229efccbcf32.
[    2.577108] dracut: Mounted root filesystem /dev/nvme0n1p2
[    2.594083] dracut: Switching root

But the system is also clean...

Code: Select all

sudo tune2fs -l /dev/nvme0n1p2 | grep "Filesystem state"
Filesystem state:         clean

How do I get it to stop this orphan cleanup?

THANKS

[zen_desu]

Hi,

I compiled 6.12.74

I read Dracut is being too chatty. It mounts the root filesystem as Read-Only to check it, sees tiny leftovers in the journal (orphans), and complains because it can't "clean" them without write access.

So I tried running dracut as; dracut --force --no-hostonly

But in dmesg I keep getting these messages;

Code: Select all

[    2.548735] EXT4-fs (nvme0n1p2): orphan cleanup on readonly fs
[    2.549071] EXT4-fs (nvme0n1p2): mounted filesystem 99bcf0bd-7d3e-4940-8f1a-229efccbcf32 ro with ordered data mode. Quota mode: disabled.
[    2.557686] dracut: Remounting /dev/disk/by-uuid/99bcf0bd-7d3e-4940-8f1a-229efccbcf32 with -o defaults,noatime,ro
[    2.569180] EXT4-fs (nvme0n1p2): re-mounted 99bcf0bd-7d3e-4940-8f1a-229efccbcf32.
[    2.577108] dracut: Mounted root filesystem /dev/nvme0n1p2
[    2.594083] dracut: Switching root

But the system is also clean...

Code: Select all

sudo tune2fs -l /dev/nvme0n1p2 | grep "Filesystem state"
Filesystem state:         clean

How do I get it to stop this orphan cleanup?

THANKS

I think you can set 'nofscks="yes"' in your dracut config, but generally it's good if it fscks when bringing up storage devices. I'm not sure why this error is repeating though

[NeddySeagoon]

GenIT,

The system is detecting orphan i-nodes but can't do anything about them, so yeu get te warning every boot.

Be sure you have a backup of the filesystem.
Check that. A backup is two or more validated copies.

Boot your install media.
Run fsck on the affected partition.

Orphan i-nodes indicates a shutdown when something was writing to the filesystem.

WARNING: fsck can make things much worse rather than better.
That's why you need validated backups. It all depends on what else it discovers.

[frostschutz]

"Orphan cleanup on readonly fs" temporarily lifts the readonly status and performs the cleanup anyway.

Readonly removed here https://github.com/torvalds/linux/blob/ ... #L420-L423

Readonly reinstated here https://github.com/torvalds/linux/blob/ ... han.c#L506

Several other error messages in this function deliberate state "skipping orphan cleanup", this one does not skip. There is even a message "write access unavailable, skipping orphan cleanup" meaning the block device itself is read-only. Maybe you'd get that message if you use `mount -o loop,ro` instead of just `mount -o ro`, with a read-only loop device in between.

In data recovery you don't trust read-only mounts to not modify your filesystems. You get a filesystem where *you* can't write any files, but the filesystem itself can still be modified. Some filesystems cannot perform recovery in-memory only so it's done on-disk.

But that doesn't solve your problem. As to why you got orphans in the first place, I can't say. Since this happens in kernel, removing fsck flag or whatever from dracut, probably won't help.

[GenIT]

I use i3 with OpenRC, I don't use a login manager, I simply login through tty user/pass - startx.

When I exit, reboot, or shutdown i3, I do it through keybinds with these settings in the config.

Code: Select all

# Logout, Reboot, Poweroff
mode "(L)ogout, (R)eboot, (P)oweroff" {
        bindsym l exit
        bindsym r exec --no-startup-id sudo reboot
        bindsym p exec --no-startup-id sudo poweroff

        # back to normal: Enter or Escape
        bindsym Return mode "default"
        bindsym Escape mode "default"
}

bindsym $mod+Shift+e mode "(L)ogout, (R)eboot, (P)oweroff"

I then tried to change this shutting down openrc, thinking I needed to do that, to make this cleaner, but it didn't help.

Code: Select all

# Logout, Reboot, Poweroff
mode "(L)ogout, (R)eboot, (P)oweroff" {
        bindsym l exit
        bindsym r exec --no-startup-id sudo openrc-shutdown --reboot now
        bindsym p exec --no-startup-id sudo openrc-shutdown --poweroff now

        # back to normal: Enter or Escape
        bindsym Return mode "default"
        bindsym Escape mode "default"
}

bindsym $mod+Shift+e mode "(L)ogout, (R)eboot, (P)oweroff"

I also use Slackware, CachyOS, and Arch, so along with Gentoo, I have 8 backups, 2 for each distro. All the backups are good, all the systems are good.
I also only run i3 on them the same way, no login manager, tty/startx, and all using the same config settings, as the first one.

I do not have any system or hardware issues, I just see that using OpenRC on a small WindowManager setup, with no login manager,
it seems like I need to do something different here, from the usual, say those on KDE through KDM, those end-users don't see this.

Hi @NeddySeagoon, the system came back showing clean, unless there is some possible fsck glitch, where it's giving false reports.

Code: Select all

sudo tune2fs -l /dev/nvme0n1p2 | grep "Filesystem state"
Filesystem state:         clean 

I have another idea to try here....

Code: Select all

# Logout, Reboot, Poweroff
mode "(L)ogout, (R)eboot, (P)oweroff" {
    bindsym l exit
    bindsym r exec --no-startup-id sudo sh -c "sync; mount -o remount,ro /; openrc-shutdown --reboot now"
    bindsym p exec --no-startup-id sudo sh -c "sync; mount -o remount,ro /; openrc-shutdown --poweroff now"

    bindsym Return mode "default"
    bindsym Escape mode "default"
}
bindsym $mod+Shift+e mode "(L)ogout, (R)eboot, (P)oweroff"

And then for dracut;

Code: Select all

/etc/dracut.conf.d/shutdown.conf
add_dracutmodules+=" shutdown "

I'm runnning Larry on a Samsung 990 Pro, from some diggging around looking at this, could it be when running an NVMe, we possibly have a race condition, because the NVMe is
faster than the OpenRC unmount script?

THANKS

[zen_desu]

I don't think you should need to adjust your initramfs to address your storage devices being unmounted incorrectly. I'm guessing there was a hard shutdown at some point and it damaged your FS.

[GenIT]

I don't think you should need to adjust your initramfs to address your storage devices being unmounted incorrectly. I'm guessing there was a hard shutdown at some point and it damaged your FS.

The box has never hard a shutdown from me, if the FS was damaged, I wouldn't expect it to report back clean.

My understanding is if you had a "Hard Shutdown" (power loss), you'd have possibly some major FS errors. To me Orphan Inodes, is the "smoking gun" for a failed unmount during a graceful shutdown.

THANKS

[GenIT]

I booted up the LiveKDE, and ran fsck it showed it was clean.

At first I thought this was a dracut initramfs issue, so I removed the initramfs because I don't need it to boot.

But I see the same thing in dmesg;

Code: Select all

[    2.337086] EXT4-fs (nvme0n1p2): orphan cleanup on readonly fs
[    2.337406] EXT4-fs (nvme0n1p2): mounted filesystem 99bcf0bd-7d3e-4940-8f1a-229efccbcf32 ro with ordered data mode. Quota mode: disabled.
[    2.337414] VFS: Mounted root (ext4 filesystem) readonly on device 259:2.

In i3 it's my understanding I should be using openrc-shutdown to reboot and shutdown, so I placed it in my i3 config, slightly different than before, but it doesn't help.
I also tried using loginctl in the config, but no success with it either.

Code: Select all

# Logout, Reboot, Poweroff
mode "(L)ogout, (R)eboot, (P)oweroff" {
        bindsym l exit
        bindsym r exec --no-startup-id sudo openrc-shutdown -r now
        bindsym p exec --no-startup-id sudo openrc-shutdown -p now

        # back to normal: Enter or Escape
        bindsym Return mode "default"
        bindsym Escape mode "default"
}

bindsym $mod+Shift+e mode "(L)ogout, (R)eboot, (P)oweroff"

This is my takeway here on this at the moment, because this really looks like an OpenRC situation I am running into.

1. Transition: You trigger shutdown. OpenRC moves to the shutdown runlevel.

2. Kill Phase: killprocs runs. It sends a SIGTERM to all processes (Pipewire, etc.).

3. The Handshake: Because mount-ro says after killprocs, it waits for killprocs to finish.

4. The Flaw: killprocs finishes as soon as it sends the signals. It does not wait for the processes to actually exit or for the NVIDIA driver to release the hardware.

5. The Collision: mount-ro starts its sync and remount. The kernel says: "No. The various processes are still 'technically' alive and holding a file open. Device Busy."

The End: mount-ro fails. OpenRC proceeds to the final kernel power-off.

Also what about /etc/init.d/mount-ro listed as start() , I thought OpenRC calls mount-ro. Since there is no stop() function, I thought it does nothing, and therefore the disk never gets remounted as it should causing the Orphan Inodes.

THANKS

[NeddySeagoon]

GenIT,

What happens if you shutdown with

Code: Select all

shutdown -h now

Run as root?

poweroff is like a power failure but I don't know what openrc does with that option.
I've used startx and that shutdown command for a long time with no problems.

[freke]

Isn't this just an information that it is running orphan cleanup?
If there was anything to cleanup wouldn't it be more like this

Code: Select all

[1840475.709644] EXT4-fs (dm-2): orphan cleanup on readonly fs
[1840475.710209] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 785094
[1840475.710782] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 797898
[1840475.710936] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 797897

[GDH-gentoo]

In i3 it's my understanding I should be using openrc-shutdown to reboot and shutdown, so I placed it in my i3 config, slightly different than before, but it doesn't help.

Only if you are using openrc-int as the PID 1 program, which is not the case on Gentoo unless you explicitly set that up.

[NeddySeagoon]

freke,

That's what happens on an unmouted filesystem.
When rootfsck runs on a ro mounted fs it can't do everything it wants to.
A journal replay happens on a ro filesystem, as long as the underlying block device is not ro.

[GenIT]

Hi NeddySeagoon,

This is a reboot issue at present, I've just exited out of i3 to the console, and just doing Ctrl Alt Del I still get this oprhan cleanup.

NeddySeagoon if you look in dmesg you don't see this exact message, or similar?

Since I also tried rebooting without initramfs, my understanding is this has now removed dracut out of the sitiuation, and it now becomes
a Kernel and OpenRC issue.

I also booted up the Live KDE iso and ran fsck a different way, now for a second time and still clean.

I truly believe there is either a glitch with OpenRC, or, when using a windowmanger without a login manager, there is a way in which I need to
be rebooting and shutting down, and that I also need a OpenRC script to properly handle the reboot/shutdown.

But I used these and it didn't work for the cmds part;

Code: Select all

sudo openrc-shutdown -r now
sudo openrc-shutdown -p now

Ok I didn't set up openrc-int as PID 1

And last, if everything is correct in my setup, then I've read, when running fast hardware, it moves faster, to fast for OpenRC to keep up with and it glitches like this, not allowing clean reboots/shutdowns.

On this fast rebooting part, of fast hardware, I will tell you this, I've used Linux a long time, and around 50-70 Distros, and in all my years, no system has moved as fast as this, it's the fastest. I can reboot and restart in like 7-10 secs,
so I'm starting to think there are aspects at play that can't keep up with it, and it is glitching out on this all the time.

This thing moves at the blink of an eye, it amazes the crap out of me how fast I've got it running.

THANKS

[GenIT]

Isn't this just an information that it is running orphan cleanup?
If there was anything to cleanup wouldn't it be more like this

Code: Select all

[1840475.709644] EXT4-fs (dm-2): orphan cleanup on readonly fs
[1840475.710209] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 785094
[1840475.710782] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 797898
[1840475.710936] EXT4-fs (dm-2): ext4_orphan_cleanup: deleting
unreferenced inode 797897

Excellent point too, because that is what I've also read, if it is cleaning it shows this, at least that's my understanding.

Wait, is this just the kernel saying this option is simply loaded?

Is this simply an option that is being shown it's just been loaded is all? Because this is all my system says, and I never started to think
of it this way, as simply a message that says been engaged/loaded, but then I would of assumed it would of said instead as; orphan cleanup loaded on readonly fs

Code: Select all

[    2.548735] EXT4-fs (nvme0n1p2): orphan cleanup on readonly fs
[    2.549071] EXT4-fs (nvme0n1p2): mounted filesystem 99bcf0bd-7d3e-4940-8f1a-229efccbcf32 ro with ordered data mode. Quota mode: disabled.
[    2.557686] dracut: Remounting /dev/disk/by-uuid/99bcf0bd-7d3e-4940-8f1a-229efccbcf32 with -o defaults,noatime,ro
[    2.569180] EXT4-fs (nvme0n1p2): re-mounted 99bcf0bd-7d3e-4940-8f1a-229efccbcf32.
[    2.577108] dracut: Mounted root filesystem /dev/nvme0n1p2
[    2.594083] dracut: Switching root

If you go online and lookup; orphan cleanup on readonly fs

You will see mention of it then cleaning up. not just showing only what I see.

I think this is only a Status Message, that it has entered this state it's telling you is all...

Ok I have found the code, of course this is for 5.15.180, but it's been around my understanding a long time.

https://elixir.bootlin.com/linux/v5.15. ... han.c#L422

Code: Select all

f (s_flags & SB_RDONLY) {
		ext4_msg(sb, KERN_INFO, "orphan cleanup on readonly fs");
		sb->s_flags &= ~SB_RDONLY;
	}

It's my understganding In the Linux kernel, ext4_msg is a wrapper for printk.

sb: The superblock (identifies which drive/partition is talking).

KERN_INFO: The log level. It tells the system this is just an informational message, not an error or a warning.

The String: The actual text that appears in your dmesg or /var/log/messages.

It is the kernel's version of a "print" or "echo" statement.

THANKS

[GenIT]

Here's a discussion on this over at OpenRC...

https://github.com/OpenRC/openrc/discussions/983

Of course I have orphan_file and orphan_present active in the features.

sudo tune2fs -l /dev/nvme0n1p2 | grep "Filesystem features"

Code: Select all

Filesystem features:      has_journal ext_attr resize_inode dir_index orphan_file filetype needs_recovery extent 64bit flex_bg metadata_csum_seed sparse_super large_file huge_file dir_nlink extra_isize metadata_csum orphan_present

It's my understanding when you have the orphan_file feature enabled, the kernel is much more sensitive. It doesn't mean the system crashed; I thought that means that at the moment of the "Read-Only" transition during shutdown, the kernel marked that file as "present"
to finish a housekeeping task, and it's simply clearing that bit on the next boot.

My understanding also is, it's not a dirty' shutdown issue, when the filesystem is using the orphan_file feature. The orphan_present flag is what's triggering the ext4_orphan_cleanup logic in the kernel source. Again my understanding, the kernel isn't fixing corruption; it's simply clearing the
orphan file bit as designed. The issue is just that OpenRC doesn't silence this specific kernel chatter during the mount transition.

THANKS

[zen_desu]

orphan file bit as designed. The issue is just that OpenRC doesn't silence this specific kernel chatter during the mount transition.

THANKS

openrc is not involved until after switch_root, it may be a kernel thing, may be a dracut thing

[GenIT]

orphan file bit as designed. The issue is just that OpenRC doesn't silence this specific kernel chatter during the mount transition.

THANKS

openrc is not involved until after switch_root, it may be a kernel thing, may be a dracut thing

Here are my understandings on my thoughts below...

Here's systemd vs OpenRC to compare.

Arch uses Systemd, which has a built-in "final shutdown" phase that automatically remounts the root filesystem as read-only and performs a global sync before the kernel takes back control.

OpenRC does not do this as aggressively by default; it relies on individual service scripts and the final init transition.

Code: Select all

Feature: -------------------------- Arch (Systemd) -------------------------------------- Gentoo (OpenRC)

Shutdown Logic: -----------  Hardcoded, centralized "final" phase ----------------- Script-based runlevels (shutdown)
Root Remount: ------------- Forces / to Read-Only early --------------------------- Often stays Read-Write until the end
Flush Behavior: ------------ Waits for all PID 1 children to die ------------------- Processes killed while buffers open

Is it an OpenRC "Glitch"?

Not a glitch, but a design difference. OpenRC is minimal; it doesn't babysit the NVMe cache unless you tell it to.

I use Arch too, and this is all I get on Arch...

Code: Select all

arch kernel: EXT4-fs (nvme0n1p2): mounted filesystem dd96df73-860a-4e56-8aa1-8897776f2599 r/w with ordered data mode. Quota mode: none.
arch kernel: EXT4-fs (nvme0n1p2): re-mounted dd96df73-860a-4e56-8aa1-8897776f2599.
arch systemd[1]: Finished Remount Root and Kernel File Systems.

With Arch showing clean, shows a managed shutdown sequence (Systemd's way) prevents the "orphan file", unless, but I am not aware, that someone on Arch simply patched it.
On Gentoo, you have to manually recreate that logic.

Systemd (Arch) performs a "pivot" at shutdown. It moves the entire OS back into RAM, unmounts your NVMe completely, and only then signals the hardware to stop.
Because the drive is unmounted before the power signal, the "orphan bit" never triggers.

Why Gentoo (OpenRC)?

OpenRC stops services and then tells the kernel "okay, I'm done." If the kernel or Dracut doesn't cleanly remount / as read-only fast enough before the reset hits, it looks like we get the "orphan" message.

So I would assume if it's a kernel thing, I would see this in Arch, and since switching my box over to NVMe I've used 4 other distros I've not seen this on as well.


THANKS

[GenIT]

I am not trying to make 25 posts in a row here.. LOL

I am just trying to be helpful, because I am 99.9% sure I did it by the book.

On eselect profile list I selected #3

Code: Select all

[3]   default/linux/amd64/23.0/desktop (stable) *

My understandings this is OpenRC installed, setup and ready to go out the box...

*** I am not using PID1 ***

From further looking, if I am understanding this correctly rc.log shows that OpenRC is stopping at udev. It never reaches the mount-ro service.

Because mount-ro never runs, the NVMe is never transitioned to a read-only state. When the kernel takes over a few milliseconds later to reboot,
the disk is still "Dirty" and "Open," which is why I see the "orphan cleanup" on every single boot.

Why is it stopping?
The inittab, the line l6:6:wait:/sbin/openrc reboot is firing. OpenRC starts the shutdown, but it seems to think its job is done once udev stops, or it's being interrupted.

So yes or no on my thoughts for this?

Here's the rc.log online;

https://dpaste.com/G3PEFYBWR

Hmm Arch wiki;

Code: Select all

During shutdown remounting root as read-only fails

If the above happens, edit the /etc/openrc/init.d/mount-ro file and put:

telinit u

after the following line:

# Flush all pending disk writes now
sync; sync

THANKS