| View previous topic :: View next topic |
| Author |
Message |
deadram
n00b
Joined: 20 Dec 2006
Posts: 46
|
 Posted: Sun Jul 06, 2025 2:19 am Post subject: After a power outage, initramfs can't find the root part |
 |
|
After a power outage, initramfs can't find the root partition at boot time. I have a special use case, which requires some explaining. Basically, lvm is missing it's RAM copy of the root partitions mirror, but it has the "backup" copy available and degraded on the hdd (at /dev/sdb3). initramfs (and my livecd boot partition) don't have access to the lv, until I manually recover it.
I've got my root partition as an lvm mirror with a single copy (0 mirrors) on sdb3. Once booted, it loads a ram drive, sets up a lvm mirror on that drive and begins a sync (1 mirror) to the RAM mirror. Then it disables the root hdd.
This is the "critical" part of the start script at boot time
| Code: | $ cat /etc/ramraid/ramraid.start | tail -n 39 | head -n 11
# add /dev/ram0 block ram device
modprobe brd rd_nr=${BLOCK_RAM_DISK_NUM} max_part=${BLOCK_RAM_DISK_PARTITIONS} rd_size=${ROOT_SIZE} || eexit "\tError: modprobe brd failed"
# add ram0p[1-4] as a physical drive for lvm use
pvcreate ${RAM_DEV} || eexit "\tError: \"pvcreate ${RAM_DEV}\" failed"
# add ram0p[1-4] as a drive on the vg
vgextend ${VG_NAME} ${RAM_DEV} || eexit "\tError: \"vgextend ${VG_NAME} ${RAM_DEV}\" failed"
# convert the lv to use the ram drives as mirror (this starts a sync from the ssd/hhd to your ram drive)
lvconvert -y -m ${NUM_MIRROR_RAM} ${LV_DEV} || eexit "\tError: \"lvconvert -y -m ${NUM_MIRROR_RAM} ${LV_DEV}\" failed" |
This script waits for the "Cpy%Sync" value to get to 100%, then it splits the hdd away from the mirror (Using only RAM, so everything runs faster and less writes to hdd/ssd).
| Code: | $ cat ramraid.forkwaitforsync.thensplit | tail -n 65 | head -n 40
# While sync is not 100 percent, sleep for 5 seconds
function waitforsync() {
TEST=0
while [[ x$TEST != x100 ]] ; do
# Sleep for 5 seconds
sleep 5
TEST=$(lvs -o sync_percent ${LV_DEV} | tail -n 1 | sed -ne 's/ \+//gm;s/\.[0-9][0-9]//;s/ \+//m;p')
# echo -ne "\r${TEST}" # Uncomment to debug
if ! ((TEST <= 100)) || ! ((TEST >= 0)) ; then
eexit "\tError reading sync percent: \"$TEST\""
fi
done
# Just take some time after sync, 1 second ought to do it
sleep 1
}
elog "$0 started on "$(date)
if [[ -f /etc/ramraid/config ]] ; then
. /etc/ramraid/config
else
eexit "\t$0 failed to load /etc/ramraid/config"
fi
touch ${TMP_DIR}/ramraid.start.forkwait
waitforsync
# You cannot use a number higher then 1, with "splitmirrors" and "trackchanges".
# trackchanges will only sync the changes, and not the whole image when the mirrors are merged again
lvconvert -y --splitmirrors 1 --trackchanges ${LV_DEV} /dev/disk/by-partuuid/${MAIN_PARTUUID}
elog "\t$0 completed on "$(date)
if [ -f ${TMP_DIR}/ramraid.start.forkwait ] ; then
rm -rf ${TMP_DIR}/ramraid.start.forkwait
fi
exit 0 |
Here's some of the output from lvs/pvs after boot:
| Code: | $ echo "First 20 min after boot" ; lvs -a
First 20 min after boot
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv-hdd-strips vg-hdd-strips -wi-ao---- 1.36t
lv-root-raid1 vg-root-raid1 rwi-aor--- 255.99g 63.97
[lv-root-raid1_rimage_0] vg-root-raid1 iwi-aor--- 255.99g
[lv-root-raid1_rimage_1] vg-root-raid1 Iwi-aor--- 255.99g
[lv-root-raid1_rmeta_0] vg-root-raid1 ewi-aor--- 4.00m
[lv-root-raid1_rmeta_1] vg-root-raid1 ewi-aor--- 4.00m
$ pvs -a
PV VG Fmt Attr PSize PFree
/dev/ram0 vg-root-raid1 lvm2 a-- <256.00g 0
/dev/ram1 --- 0 0
/dev/sda1 --- 0 0
/dev/sda2 --- 0 0
/dev/sda3 --- 0 0
/dev/sda4 --- 0 0
/dev/sda5 --- 0 0
/dev/sdb1 --- 0 0
/dev/sdb2 --- 0 0
/dev/sdb3 vg-root-raid1 lvm2 a-- <256.00g 0
/dev/sdb4 --- 0 0
/dev/sdc1 vg-hdd-strips lvm2 a-- <465.76g 0
/dev/sdd1 vg-hdd-strips lvm2 a-- <931.51g 0
$ echo "After sync completes" ; lvs -a
After sync completes
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv-hdd-strips vg-hdd-strips -wi-ao---- 1.36t
lv-root-raid1 vg-root-raid1 rwi-aor--- 255.99g 100.00
lv-root-raid1_rimage_0 vg-root-raid1 Iri-a-r--- 255.99g
[lv-root-raid1_rimage_1] vg-root-raid1 iwi-aor--- 255.99g
[lv-root-raid1_rmeta_0] vg-root-raid1 ewi-a-r--- 4.00m
[lv-root-raid1_rmeta_1] vg-root-raid1 ewi-aor--- 4.00m
|
/dev/sdb3 is the "hard" copy of the root mirror, /dev/ram0 is the mirror in ram. If I manually boot into the livecd, then I can recover the lv with the following:
| Code: | $ cat /boot/efi/ramraid.repair
#!/bin/sh
# Use this command first, from memory XD won't be able to mount drive until then. and this file is stored on that drive XD maybe copy this one file to boot?
#vgchange -ay vgrootraid1
#
# Old value for 64GB RAM Drive, new value is *4=268435456
# modprobe brd rd_nr=2 max_part=4 rd_size=67108864
cat << EOF
modprobe brd rd_nr=2 max_part=4 rd_size=268435456
pvcreate /dev/ram0
vgextend vg-root-raid1 /dev/ram0
vgreduce --removemissing --force vg-root-raid1
lvconvert --mergemirrors vg-root-raid1
lvchange -a y vg-root-raid1/lv-root-raid1
lvconvert --repair vg-root-raid1/lv-root-raid1
vgreduce --removemissing --force vg-root-raid1
EOF
|
There's also a stop script (for use during a normal shutdown), and a "flush" script, which merges the mirrors once a day in a cronjob. normal reboot works fine and the RAM mirror is removed from the lv and vg and pv groups cleanly before a reboot/shutdown.
What I'd like to do, is force the lvm partition to automatically "repair" and bring online the degraded root lv on /dev/sdb3, maybe from grub, or maybe with boot options? If that's not possible, I'll probably have to edit the initramfs, and add my ramraid.repair script to it, but I'd prefer if that wasn't required. Unfortunately this is a hobby server, so no access to a UPS, and too expensive for me to dump the money into one.
Please let me know if you want to see the full ramraid script, or any other details.
_________________
echo deadram; fortune |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 55402
Location: 56N 3W
|
| Posted: Sun Jul 06, 2025 11:30 am Post subject: |
|
|
deadram,
Verbatim error messages and the content of /dev when you get dumped into the initrd shell would be good.
Your initrd shell should tell you exactly which step failed, if not, it needs to be fixed.
Incidentally, if you have the RAM to run from RAM the way you are, it will save writes but it won't be any faster.
The kernel will use your RAM to cache everything that's used anyway.
You move the time to get a hot cache to the boot process instead of on first use.
You also cache the HDD free space, which is a waste of time and RAM.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
deadram
n00b
Joined: 20 Dec 2006
Posts: 46
|
| Posted: Sun Jul 06, 2025 7:21 pm Post subject: |
|
|
| NeddySeagoon wrote: | Verbatim error messages and the content of /dev when you get dumped into the initrd shell would be good.
Your initrd shell should tell you exactly which step failed, if not, it needs to be fixed. |
| Code: | $ cat gksosreport.txt | tail -n 53
[ 7.793667] input: Logitech Wireless Mouse PID:4091 Consumer Control as /devices/pci0000:00/0000:00:14.0/usb3/3-5/3-5.1/3-5.1:1.1/0003:046D:C534.0002/0003:046D:4091.0004/input/input13
[ 7.793777] hid-generic 0003:046D:4091.0004: input,hidraw3: USB HID v1.11 Mouse [Logitech Wireless Mouse PID:4091] on usb-0000:00:14.0-5.1/input1:2
[ 7.794944] udevd[1601]: failed to execute '/usr/lib/udev/fido_id' 'fido_id': No such file or directory
[ 7.795749] udevd[1602]: failed to execute '/usr/lib/udev/fido_id' 'fido_id': No such file or directory
[ 8.698633] raid6: skipped pq benchmark and selected avx2x4
[ 8.698638] raid6: using avx2x2 recovery algorithm
[ 8.700419] async_tx: api initialized (async)
[ 8.702304] xor: automatically using best checksumming function avx
[ 8.725704] device-mapper: raid: Failed to read superblock of device at position 1
[ 8.819045] md: pers->run() failed ...
[ 8.819065] device-mapper: table: 254:6: raid: Failed to run raid array (-EINVAL)
[ 8.819070] device-mapper: ioctl: error adding target to table
+ '[' -f /run/initramfs/init.log ]
+ cat /run/initramfs/init.log
+ sed -e 's/\(ftp:\/\/.*\):.*@/\1:*******@/g;s/\(cifs:\/\/.*\):.*@/\1:*******@/g;s/cifspass=[^ ]*/cifspass=*******/g;s/iscsi:.*@/iscsi:******@/g;s/rd.iscsi.password=[^ ]*/rd.iscsi.password=******/g;s/rd.iscsi.in.password=[^ ]*/rd.iscsi.in.password=******/g'
[ 2025-07-06 17:54:38.076294 ] Executed: 'busybox --install -s'
[ 2025-07-06 17:54:38.078456 ] [OK] Genkernel 4.3.16 (2025-04-04 14:10:26 UTC). Linux kernel 6.12.21-gentoo-x86_64-1
[ 2025-07-06 17:54:38.296100 ] Executed: 'mount -t devtmpfs -o exec,nosuid,mode=0755,size=10M udev /dev'
[ 2025-07-06 17:54:38.298750 ] Executed: 'mkdir -m 0755 /dev/pts'
[ 2025-07-06 17:54:38.300695 ] Executed: 'mount -t devpts -o gid=5,mode=0620 devpts /dev/pts'
[ 2025-07-06 17:54:38.302457 ] Executed: 'mkdir -m 1777 /dev/shm'
[ 2025-07-06 17:54:38.304577 ] Executed: 'mount -t tmpfs -o mode=1777,nosuid,nodev,strictatime tmpfs /dev/shm'
[ 2025-07-06 17:54:38.306481 ] Executed: 'mount -t sysfs sysfs /sys -o noexec,nosuid,nodev'
[ 2025-07-06 17:54:38.307586 ] [OK] Activating udev ...
[ 2025-07-06 17:54:38.318695 ] Executed: 'udevd --resolve-names=never --daemon'
[ 2025-07-06 17:54:38.353856 ] Executed: 'udevadm trigger --action=add'
[ 2025-07-06 17:54:42.713198 ] Executed: 'udevadm settle --timeout=120'
[ 2025-07-06 17:54:42.715036 ] [OK] Scanning for volume groups ...
Found volume group "vgrootraid1" using metadata type lvm2
WARNING: Couldn't find device with uuid 26LJ0X-FDk7-6lky-pnvx-keFe-Fq5r-Y92Xog.
WARNING: VG vg-root-raid1 is missing PV 26LJ0X-FDk7-6lky-pnvx-keFe-Fq5r-Y92Xog (last written to /dev/ram1).
Found volume group "vg-root-raid1" using metadata type lvm2
Found volume group "vg-hdd-strips" using metadata type lvm2
[ 2025-07-06 17:54:42.845773 ] Executed: 'lvm vgscan'
[ 2025-07-06 17:54:42.861767 ] Executed: 'udevadm settle --timeout=120'
[ 2025-07-06 17:54:42.863560 ] [OK] Activating volume groups ...
1 logical volume(s) in volume group "vgrootraid1" now active
WARNING: Couldn't find device with uuid 26LJ0X-FDk7-6lky-pnvx-keFe-Fq5r-Y92Xog.
WARNING: VG vg-root-raid1 is missing PV 26LJ0X-FDk7-6lky-pnvx-keFe-Fq5r-Y92Xog (last written to /dev/ram1).
device-mapper: reload ioctl on (254:6) failed: Invalid argument
1 logical volume(s) in volume group "vg-root-raid1" now active
1 logical volume(s) in volume group "vg-hdd-strips" now active
[ 2025-07-06 17:54:43.336854 ] Failed (5): 'lvm vgchange -ay --sysinit'
[ 2025-07-06 17:54:43.351575 ] Executed: 'udevadm settle --timeout=120'
[ 2025-07-06 17:54:43.357224 ] Executed: 'mkdir -p /newroot'
[ 2025-07-06 17:54:43.361219 ] [OK] Determining root device (trying /dev/mapper/vg--root--raid1-lv--root--raid1) ...
[ 2025-07-06 17:54:49.104324 ] [!!] Block device /dev/mapper/vg--root--raid1-lv--root--raid1 is not a valid root device ...
[ 2025-07-06 17:54:49.114147 ] [!!] Could not find the root block device in /dev/mapper/vg--root--raid1-lv--root--raid1.
[ 2025-07-06 17:54:49.123434 ] [!!] Please specify another value or:
[ 2025-07-06 17:54:49.132495 ] [!!] - press Enter for the same
[ 2025-07-06 17:54:49.141371 ] [!!] - type "shell" for a shell
[ 2025-07-06 17:54:49.150167 ] [!!] - type "q" to skip ...
[ 2025-07-06 17:55:27.597809 ] [**] To leave and try again just press <Ctrl>+D |
| Code: | $ ls /dev/vg-root-raid1/
lv-root-raid_rimage_0
$ ls /dev
...
dm-0
dm-1
dm-2
...
vg-root-raid1/
... |
If I go into shell, I can mount /dev/dm-2, which is the root partition.
To fully recover, I need to be in my live-cd boot, and manually run the commands in ramraid.repair, they don't seem to work, unless I have a /dev/ram0 device in the vg. I haven't tried "modprobe brd" from the initramfs shell recovery, but I'm guessing I'd have to move that kernel module into the initramfs file, or compile it into the kernel and change my boot options?
After boot, a start up script will run "ramraid.forkwaitforsync.thensplit". It watches the Sync status, and once it's 100%, it runs the following command:
| Code: | | lvconvert -y --splitmirrors 1 --trackchanges ${LV_DEV} /dev/disk/by-partuuid/${MAIN_PARTUUID}" |
That line triggers the mirrors to split, and the root to run all from RAM. I can manually merge the mirrors, with "lvconvert --mergemirrors ${SPLITMIRROR}" wait for sync to get to 100% then "lvconvert -y -m 0 ${LV_DEV} ${RAM_DEV}", then the RAM mirror is removed, and the system will reboot normally, this is basically what the ramraid.stop script does. The more changes to rootfs, the longer this sync will take, with no changes the sync occurs within 5 or less seconds.
| Quote: | Incidentally, if you have the RAM to run from RAM the way you are, it will save writes but it won't be any faster.
The kernel will use your RAM to cache everything that's used anyway. |
I'd probably have to change a few sysctl settings for that to be true? max size of cache at least? From my experience, most tools to test the speed don't work, because they don't actually start measuring, until the kernel has all the data in cache. And real world testing with the game "Elite Dangerous" with game files stored on a ssd raid0 (and 300+GB of free RAM), has trouble loading planet objects like "Geyser" and "Flora", unless I'm flying around the planet at 50km/h, even after 60+ minutes of flying around the same planet. When the Elite Dangerous game files are stored on the RAM drive mirror, I can travel 10 times as fast, at 500km/h and loading of geysers and flora is not affected.
| Quote: | You move the time to get a hot cache to the boot process instead of on first use.
You also cache the HDD free space, which is a waste of time and RAM. |
both true, but while experimenting with zram drives, the cpu got too much of a hit, and there would be occasional "freezes" while the data was decompressing, iirc. And without the lvm mirror, there is no way to boot to a ram drive, without heavily modifying the initramfs files, and forcing you to wait for the "sync". lvm mirror allows a seamless boot, while sync occurs in the background at boot time. The wasted RAM and time is of no consequence to me. I bought this box specifically to run the root from RAM. The extra empty space is a read from the hdd/ssd, and not a write (so it won't affect the drive as much as writes would). If I ever do have large computations (which occur 1 or 2 times every couple of years), I can turn off the RAM mirror, do the computations and then restore the RAM mirror.
Dell Poweredge R730, 72 cores, 512GB RAM, 6.7T ssd raid0, 1.4T ssd/hhd strips and an AMD Radeon RX 7800 XT on a riser, because it won't fit in the box! This is a beast of a PC, lol 
_________________
echo deadram; fortune |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
