Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Migrate MDADM autoassembly --> initramfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Fri Jan 31, 2014 1:13 pm    Post subject: [SOLVED] Migrate MDADM autoassembly --> initramfs Reply with quote

Hi all,

Currently I do not have an initramfs and the kernel auto assembles my 3 mirrors when LILO fires off the kernel. This is because all 3 arrays are still on superblock v0.90. I've read about this possibly being depricated in the future and I want to migrate to an initramfs to assemble the arrays. When this works I want to move the data off the arrays and re-create them with superblock v1.20.

So I've installed genkernel to create the initramfs only and ran something like
Code:
genkernel initramfs --mdadm


It generated the initramfs and I stuck it in /boot. I then re-compiled my kernel but I set:
Code:
CONFIG_MD_AUTODETECT=n


Hoping to completely disable auto detection. My theory was that if I disable this kernel option and my server boots it MUST be the initramfs assembling the arrays.

When I added a second entry to my lilo.conf and included the initramfs and the append options for real_root and domdadm etc it actually booted. However, once booted the dmesg was still showing.
Code:
md: Autodetecting RAID arrays.
md: blablabla considering bla bla adding bla bla


Isn't that kernel auto detection? How do I know after booting whether the initramfs assembled the arrays or whether the kernel auto assembled them? What does it look like when the initramfs assembles the arrays? How come it still shows this line even though I disabled CONFIG_MD_AUTODETECT for this kernel?

Obviously I've read all the documentation I could find but I'm slightly confused as I can't even tell whether the initramfs is now working or not. To make matters worse I've got no video card in this box.


Last edited by pakjebakmeel on Tue Feb 11, 2014 9:58 am; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54028
Location: 56N 3W

PostPosted: Fri Jan 31, 2014 10:01 pm    Post subject: Reply with quote

pakjebakmeel,

Danger Will Robinson ..... lilo.conf

I suspect you forgot to run /sbin/lilo, so the old kernel is still used.

What does
Code:
uname -a
show?

I get
Code:
Linux NeddySeagoon_SSD 3.12.5-gentoo #1 SMP PREEMPT Sat Dec 14 18:05:33 GMT 2013 x86_64 AMD Phenom(tm) II X6 1090T Processor AuthenticAMD GNU/Linux

The #1 shows this is the first build of 3.12.5-gentoo.
The Sat Dec 14 18:05:33 GMT 2013 is the build time of the running kernel.

You should be able to tell from the timestamp in uname -a if you are running the kernel you think you are.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Sat Feb 01, 2014 9:39 am    Post subject: Reply with quote

A shotgun-type solution may be changing 'fstype' on your partitions from 'raid autodetect' to anything else, then kernel, even if support autoassamble of arrays, should never do it.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Sat Feb 01, 2014 1:19 pm    Post subject: Reply with quote

do you have a /etc{/mdadm,}/mdadm.conf?

with autodetection it's not necessary to have one, but with initramfs you usually do
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Sat Feb 01, 2014 1:55 pm    Post subject: Reply with quote

also, if you want move to initramfs, you can just fetch better-initramfs (see my sig) and just add 'softraid' to kernel's boot params, thats all. No mdadm.conf neded.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Sat Feb 01, 2014 3:21 pm    Post subject: Reply with quote

NeddySeagoon wrote:
pakjebakmeel,

Danger Will Robinson ..... lilo.conf

I suspect you forgot to run /sbin/lilo, so the old kernel is still used.

What does
Code:
uname -a
show?

I get
Code:
Linux NeddySeagoon_SSD 3.12.5-gentoo #1 SMP PREEMPT Sat Dec 14 18:05:33 GMT 2013 x86_64 AMD Phenom(tm) II X6 1090T Processor AuthenticAMD GNU/Linux

The #1 shows this is the first build of 3.12.5-gentoo.
The Sat Dec 14 18:05:33 GMT 2013 is the build time of the running kernel.

You should be able to tell from the timestamp in uname -a if you are running the kernel you think you are.


I have changed the kernel config and re-compiled, I then copied the kernel to /boot over the existing file linked to the second boot entry, because the file has now changed on block level the entry would fail to boot unless I run /sbin/lilo as the blocks on the disk would not match. Everyone using LILO has probably done this a couple of times before adopting the habit of /sbin/lilo.

Because the entry booted fine I'm pretty sure I did /sbin/lilo although I'm only 99% sure because it's a routine to run this after kernel updates. Also, I checked the dmesg after booting and it was showing the appended options specified in the second boot entry so it did actually boot the second option which means it booted the correct kernel too?

I'll give it another try next Tuesday when I have the time to make sure the correct kernel has booted. It sounds you're fairly sure that if I disabled kernel auto assembly and booted the correct kernel this could not have happened?

SlashBeast wrote:
A shotgun-type solution may be changing 'fstype' on your partitions from 'raid autodetect' to anything else, then kernel, even if support autoassamble of arrays, should never do it.


Did I mention I have no monitor attached and no video card in this server? I see how this would work but because my main lilo entry relies on auto assembly this would render my box unbootable for sure in case the iniramfs doesn't do its job : :wink: I'm using a 1 minute delay on LILO so I can arrow down and enter to boot the second entry without a monitor attached.

frostschutz wrote:
do you have a /etc{/mdadm,}/mdadm.conf?

with autodetection it's not necessary to have one, but with initramfs you usually do


Yes I have one, although I did read that including this in the initramfs is optional and should only be used if the auto detection of the arrays fails somehow.

SlashBeast wrote:
also, if you want move to initramfs, you can just fetch better-initramfs (see my sig) and just add 'softraid' to kernel's boot params, thats all. No mdadm.conf neded.


Thanks, I'll check it out now.. It's not a case of "wanting to move" as I like to keep it simple, if I don't need one I'd like to live without it for simplicity's sake. However, back in 2009 kernel auto assembly was no longer the preferred method and I'm wondering how long it will take before the kernel option actually gets deprecated. I'd rather solve the issues now than when I can no longer update the kernel because of this.

edit: That looks very interesting actually, for sure the bit where I can SSH into the box as it has no video card.. Very promising indeed, nice work.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54028
Location: 56N 3W

PostPosted: Sat Feb 01, 2014 4:15 pm    Post subject: Reply with quote

pakjebakmeel

pakjebakmeel wrote:
I then copied the kernel to /boot over the existing file linked to the second boot entry, because the file has now changed on block level the entry would fail to boot unless I run /sbin/lilo ...


When you 'overwrite' a file the new file is allocated to free blocks on the drive, when the write is complete, the space used by the old file is marked an free and the directory index updated to point to the new file. The data from the old file is still there. There is no overwrite in place.
As lilo does not read the directory at boot time, it will load the old file if you omit to run /sbin/lilo.

Code:
uname -a
will answer a lot of questions.

initrd raid assembly looks like
Code:
[    1.730302] Write protecting the kernel read-only data: 8192k
[    1.732973] Freeing unused kernel memory: 816K (ffff880001534000 - ffff880001600000)
[    1.734699] Freeing unused kernel memory: 416K (ffff880001798000 - ffff880001800000)
[    1.781785] md: md125 stopped.
[    1.783585] md: bind<sdb1>
[    1.784501] md: bind<sdc1>
[    1.785425] md: bind<sdd1>
[    1.786355] md: bind<sda1>
Notice that it happens after kernel initialisation is complete.
The initrd init line that does that reads
Code:
/sbin/mdadm --assemble /dev/md125 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

as mdadm is told explicitly what to do, no mdadm.conf is required in the initrd.

I'm fairly sure that auto assembly happens earlier is the startup sequence
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Sat Feb 01, 2014 6:48 pm    Post subject: Reply with quote

I found the dmesg's I kept.. This is from the booted kernel that has CONFIG_MD_AUTODETECT=n

Code:

[    0.000000] Linux version 3.10.25-gentoo (root@gentoo) (gcc version 4.7.3 (Gentoo 4.7.3-r1 p1.4, pie-0.5.5) ) #8 SMP Fri Jan 31 11:16:46 CET 2014
[    0.000000] Command line: BOOT_IMAGE=Recovery ro raid=noautodetect rootdelay=10 domdadm real_root=/dev/md3


And this is the kernel from my main boot entry:

Code:

[    0.000000] Linux version 3.10.25-gentoo (root@gentoo) (gcc version 4.7.3 (Gentoo 4.7.3-r1 p1.4, pie-0.5.5) ) #7 SMP Fri Jan 31 11:01:11 CET 2014
[    0.000000] Command line: auto BOOT_IMAGE=Current ro root=903


So it did boot the new one.. But later on in the kernel that has

(1 raid=noautodetect set by kernel command line
(2 CONFIG_MD_AUTODETECT=n

It says the following:

Code:

[    0.476570] md: Waiting for all devices to be available before autodetect
[    0.476575] md: If you don't use raid, use raid=noautodetect
[    0.476764] md: Autodetecting RAID arrays.
...
**Further consideration and assembly takes place**
...


muh? :?: That does kinda look like autodetection doesn't it? Anyhow, I'm going to play with better-initramfs when I have time and physical access to the box. At least knowing what it *should* look like is a big help. Thanks.

But whether I did or did run /sbin/lilo properly and whether I did or did not disable the option, shouldn't raid=noautodetect prevent this mechanism even if it's enabled in the kernel?

SlashBeast wrote:
also, if you want move to initramfs, you can just fetch better-initramfs (see my sig) and just add 'softraid' to kernel's boot params, thats all. No mdadm.conf neded.


I'm playing around with this now, 2 question arise at the moment.

1. -stupid question about labels removed-
2. What do I do with the lilo root=/dev/md3 parameter as I specify that on the append line too..

Does below look allright to you?

Code:

image=/boot/kernel-current
        label=Current
        read-only
        root=/dev/md3
        vga=791

image=/boot/kernel-recovery
        label=Recovery
        read-only
        root=/dev/md3
        append="softraid root=mirror_root=rootfs sshd sshd_wait=60 sshd_port=20 sshd_interface=eth0 sshd_ipv4=192.168.x.x/24"
        initrd=/boot/initramfs-recovery
        vga=791


kernel-current symlinks to my current kernel, kernel-recovery is now pointing to the recompiled kernel without autodetection. Second boot entry used to be recovery but is now more testing initramfs :roll:

*** I just found e2label, will that be sufficient? *hence my first question removed and some elaboration on that aswell to shorten the post.

Code:

gentoo ~ # blkid | grep mirror
/dev/md1: UUID="9a54f416-e6ef-411d-969a-7e3b157299ab" TYPE="ext4" PTTYPE="dos" LABEL="mirror_boot"
/dev/md3: LABEL="mirror_root" UUID="c25c891a-10f4-403b-8449-66a75c1d1530" TYPE="ext4"
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Sun Feb 02, 2014 12:30 pm    Post subject: Reply with quote

No idea what's the root= param from lilo, maybe you can remove it. However you should change
Code:
root=mirror_root=rootfs

to
Code:
root=LABEL=mirror_root


Also, check readme, sshd_interface was changed to binit_net_if etc, I need to update examples in readme too. ;-) If you dont change it however it will still work, there is backword compatibility layer. ;-)
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Sun Feb 02, 2014 8:22 pm    Post subject: Reply with quote

SlashBeast wrote:
No idea what's the root= param from lilo, maybe you can remove it. However you should change
Code:
root=mirror_root=rootfs

to
Code:
root=LABEL=mirror_root


Also, check readme, sshd_interface was changed to binit_net_if etc, I need to update examples in readme too. ;-) If you dont change it however it will still work, there is backword compatibility layer. ;-)


Thanks all, that really helps. I've never needed initramfs before but I figure it is wise to move along and stop using auto assembly. I'm gonna give it a shot next Tuesday with this:

Code:

boot=/dev/md1
raid-extra-boot=mbr-only
prompt
timeout=600
default=current
lba32

image=/boot/kernel-current
        label=current
        read-only
        root=/dev/md3
        vga=791

image=/boot/kernel-recovery
        label=recovery
        read-only
        append="softraid root=LABEL=root sshd sshd_wait=60 binit_net_if=eth0 binit_net_addr=192.168.10.10/24"
        initrd=/boot/initramfs-recovery
        vga=791


I've created a keypair and put the authorized_keys in the sourceroot folder before making the image. I've copied it to /boot and did not forget to run /sbin/lilo :wink:

Here's /boot:

Code:

total 7.5M
-rw-r--r-- 1 root root  512 Mar 21  2013 boot.0800
-rw-r--r-- 1 root root  512 Mar 21  2013 boot.0810
-rw-r--r-- 1 root root  512 Mar 19  2013 boot.0860
-rw-r--r-- 1 root root  512 Mar 19  2013 boot.0870
-rw-r--r-- 1 root root 1.8M Feb  2 15:30 initramfs.cpio.gz
lrwxrwxrwx 1 root root   17 Feb  1 21:25 initramfs-recovery -> initramfs.cpio.gz
lrwxrwxrwx 1 root root   28 Jan  6 09:06 kernel-current -> kernel-x86_64-3.10.25-gentoo
lrwxrwxrwx 1 root root   35 Feb  1 21:23 kernel-recovery -> kernel-x86_64-3.10.25-gentoo-noraid
-rw-r--r-- 1 root root 2.8M Jan 31 11:01 kernel-x86_64-3.10.25-gentoo
-rw-r--r-- 1 root root 2.8M Feb  1 21:27 kernel-x86_64-3.10.25-gentoo-noraid
drwx------ 2 root root  12K Feb  7  2012 lost+found
-rw------- 1 root root  90K Feb  2 16:20 map


So the 'kernel-x86_64-3.10.25-gentoo-noraid' has auto assembly disabled. 'root' is the label for my rootfs:

Code:

gentoo boot # blkid | grep root
/dev/md3: LABEL="root" UUID="c25c891a-10f4-403b-8449-66a75c1d1530" TYPE="ext4"


This initramfs doesn't mess up my preferred minors? Because when I boot an Ubuntu live USB stick to do some repair/partition stuff is messes up my preferred minors. I now have md1, md2 and m3 but after just booting a Ubuntu stick they're changed to md127, md128 and md129 permanently and I have to change them back before I can boot again.

Ooh, and to uncompress the ramdisk during boot are these options sufficient?

Code:

gentoo linux # cat .config | grep GZIP
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_KERNEL_GZIP=y
CONFIG_RD_GZIP=y
CONFIG_DECOMPRESS_GZIP=y
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Mon Feb 03, 2014 7:35 am    Post subject: Reply with quote

Options are okey. You can use nbd and 'share' your disk over local network, attach it to local /dev/nbd0 and nbd1 and try it in qemu - thats how I usualy boot remote servers for debuging early boot process (bootloader, kernel, initramfs).

nbd-server -> ssh port forward ->
Code:
nbd-client 127.0.0.1 2222 /dev/nbd1

Code:
qemu-kvm -machine accel=kvm -drive if=ide,file=/dev/nbd1,cache=none -sdl


Just a tip, when you use nbd-server you may want to drop caches or you won't see 'changes' on local end or (better) use qemu-nbd on remote end with --nocache option to omit the issue.

better-initramfs will assamble arrays starting from 0 unless mdadm.conf is provided with different names. The mdX names are meaningless when you use LABEL or UUID as initramfs use 'findfs' to get right device node path.

Code:
sabre ~ # busybox findfs LABEL=rootfs
/dev/dm-1
sabre ~ # busybox blkid /dev/dm-1
/dev/dm-1: LABEL="rootfs" UUID="7e055c79-ca6c-44dd-99d5-665f85cde4c2" TYPE="ext4"
sabre ~ #
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Mon Feb 03, 2014 2:07 pm    Post subject: Reply with quote

SlashBeast wrote:
Options are okey. You can use nbd and 'share' your disk over local network, attach it to local /dev/nbd0 and nbd1 and try it in qemu - thats how I usualy boot remote servers for debuging early boot process (bootloader, kernel, initramfs).

nbd-server -> ssh port forward ->
Code:
nbd-client 127.0.0.1 2222 /dev/nbd1

Code:
qemu-kvm -machine accel=kvm -drive if=ide,file=/dev/nbd1,cache=none -sdl


Just a tip, when you use nbd-server you may want to drop caches or you won't see 'changes' on local end or (better) use qemu-nbd on remote end with --nocache option to omit the issue.

better-initramfs will assamble arrays starting from 0 unless mdadm.conf is provided with different names. The mdX names are meaningless when you use LABEL or UUID as initramfs use 'findfs' to get right device node path.

Code:
sabre ~ # busybox findfs LABEL=rootfs
/dev/dm-1
sabre ~ # busybox blkid /dev/dm-1
/dev/dm-1: LABEL="rootfs" UUID="7e055c79-ca6c-44dd-99d5-665f85cde4c2" TYPE="ext4"
sabre ~ #


Yes you're correct in your last statement, however..

My first boot entry uses the root=/dev/md3 statement to tell the kernel that the root filesystem resides on /dev/md3. The MBR gets kicked off, LILO reads the kernel and passes /dev/md3 for root. The kernel does autoassembly and creates /dev/md3 which matches the root parameter.

If I boot the initramfs and it permanently changes my preferred minors from "1,2 and 3" to "0,1 and 2" my root filesystem is no longer on /dev/md3 but on /dev/md2 when I boot my primary entry and thus fail because the kernel is told that /dev/md3 is the root.

See the caveit? So when I boot the initramfs will it permanently change the preferred minor on my arrays like Ubuntu does or will it only create those devices on /dev/md0, /dev/md1 and /dev/md2 without updating the preferred minor?

This would be quite handy to know as I don't have a graphics card at the moment :roll:

It seems that this option

Code:
--update=super-minor


Writes the preferred minor into the metadata, I wouldn't expect your initramfs to do this but Ubuntu does it. Not sure why, maybe UDEV in Ubuntu is renaming the devices into their standard numbering and updating the metadata just for the sake of it? Just booting ubuntu causes this already.

Hence me asking so I don't end up with an unbootable box without a graphics card. :?

Ah, I see what you're doing there:

Code:

InitializeSoftwareRaid() {
        einfo "Scaning for software raid arrays."
        if ! [ -f '/etc/mdadm.conf' ]; then
                run mdadm --examine --scan > /etc/mdadm.conf
        fi
        run mdadm --assemble --scan
        run mdadm --auto-detect
}


So it just reads my config from the disks and uses that.. Shouldn't update the preferred minor's then? It seems it'll just use whatever it finds configured.
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Mon Feb 03, 2014 4:03 pm    Post subject: Reply with quote

I *think* that lilo reads the kernel from whatever device it's runned, so when it reads its config with root= its already after bootloader is there. I can't say for sure as I use extlinux on mdadm-raid enabled boxes.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Mon Feb 03, 2014 4:13 pm    Post subject: Reply with quote

pakjebakmeel wrote:
If I boot the initramfs and it permanently changes my preferred minors from "1,2 and 3" to "0,1 and 2"


You need a mdadm.conf where they are listed as md1, md2 and md3.

Code:
--update=super-minor


This option exists but is really bothersome to use; hence most people use mdadm.conf and never mind the metadata.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54028
Location: 56N 3W

PostPosted: Mon Feb 03, 2014 7:25 pm    Post subject: Reply with quote

pakjebakmeel,

You can use the raid UIDs to assemble the raid sets and the filefystem UUIDs to mount the filesystems.
Then you don't need need /dev entries - until something goes wrong.

Mounting root by UUID adds complexity to the initrd but its well documented on the Wiki and if you are going to have an initrd anyway.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Tue Feb 04, 2014 8:59 am    Post subject: Reply with quote

NeddySeagoon wrote:
pakjebakmeel,

You can use the raid UIDs to assemble the raid sets and the filefystem UUIDs to mount the filesystems.
Then you don't need need /dev entries - until something goes wrong.

Mounting root by UUID adds complexity to the initrd but its well documented on the Wiki and if you are going to have an initrd anyway.


Yes, I can remember now why I used the device nodes in the fstab and lilo config. It seems that you need an initramfs to find the corresponding devices nodes when specifying UUID or LABEL. As I never used an initramfs I just stuck to the device nodes. I can definately see the advantage in using UUID or LABEL, this adds a lot of flexibility. Once I get the initramfs working properly I would like to change the lilo entries and fstab to use UUID or LABEL.


frostschutz wrote:
pakjebakmeel wrote:
If I boot the initramfs and it permanently changes my preferred minors from "1,2 and 3" to "0,1 and 2"


You need a mdadm.conf where they are listed as md1, md2 and md3.

Code:
--update=super-minor


This option exists but is really bothersome to use; hence most people use mdadm.conf and never mind the metadata.


My point is; the init script in better-initramfs checks if /etc/mdadm.conf exists. If it doesn't it creates it using a --scan like so:

Code:
InitializeSoftwareRaid() {
        einfo "Scaning for software raid arrays."
        if ! [ -f '/etc/mdadm.conf' ]; then
                run mdadm --examine --scan > /etc/mdadm.conf
        fi
        run mdadm --assemble --scan
        run mdadm --auto-detect
}


That would just read the config from the metadata and creates an mdadm.conf on the fly wouldn't it? As long as better-initramfs doesn't use the --update=super-minor command it shouldn't permanently change my preferred minor's like Ubuntu does?

Currently an mdadm --examine --scan produces this:

Code:
gentoo initramfs # mdadm --examine --scan
ARRAY /dev/md1 UUID=3bd9364a:f7d81ce4:a65ff077:61ffda2d
ARRAY /dev/md2 UUID=b2494a3c:3a6a30e4:a65ff077:61ffda2d
ARRAY /dev/md3 UUID=e7c1fab3:a11c0038:a65ff077:61ffda2d


All that information is pulled from the metadata isn't it? So if better-initramfs produces the mdadm.conf it will contain exactly this information wouldn't it? So in effect, I would be using an mdadm.conf but it's just created on-the-fly.


Couple of questions left;

1. I think better-initramfs will use findfs to locate the device node so that should work. Where does it get the findfs command from? Or is it just part of busybox hence I cant find the binary file?

2. How does the the kernel (I presume this is what mounts root) find the corresponding device node when a UUID is specified in the fstab? If it simply uses /dev/disk/by-uuid it will fail because those nodes are not created for some reason. Or does it read the UUID/LABEL from all the nodes and mount the corresponding one? Is there something wrong with my setup as I don't have these entries? Will UUID/LABEL work in my fstab despite the missing /dev/disk/by-uuid and /dev/disk/by-label?

Code:
gentoo / # ls -l /dev/disk/
total 0
drwxr-xr-x 2 root root 1160 Jan 31 13:53 by-id
drwxr-xr-x 2 root root   60 Jan 31 13:53 by-partlabel
drwxr-xr-x 2 root root  280 Jan 31 13:53 by-partuuid
drwxr-xr-x 2 root root  400 Jan 31 13:53 by-path
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Tue Feb 04, 2014 4:51 pm    Post subject: Reply with quote

1. findfs is a busybox applet in initramfs and external binary in coreutils on real system.
2. Kernel does not care about /etc/fstab at all, mount/umount uses it and if you specify UUID= or LABEL there it dereference it to /dev/disk/by-label/<PROVIDED-LABEL>. Kernel itself rely on root= variable. If initramfs is used then kernel's rootfs is initramfs and whats next is up to userspace, better-initramfs mounts /newroot and run switch_root there to run /sbin/init.

The mdadm.conf creation on the fly is there to ensure that unless specified otherwise, arrays will be md1, md2, md3 etc. not 127, 128 and 129. It does not update any metadata so its just hidden feature or a solution to get <127 mdX names.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Mon Feb 10, 2014 8:34 am    Post subject: Reply with quote

Code:
boot=/dev/md1
raid-extra-boot=mbr-only
prompt
timeout=600
default=current
lba32

image=/boot/kernel-current
        label=current
        read-only
        append="softraid root=UUID=c25c891a-10f4-403b-8449-66a75c1d1530 sshd sshd_wait=60 binit_net_if=eth0 binit_net_addr=192.168.10.10/24"
        initrd=/boot/initramfs-current
        vga=791

image=/boot/kernel-recovery
        label=recovery
        read-only
        root=/dev/md3
        vga=791


This boots and I can get to the SSH shell too where I can see it has properly assembled the arrays. The thing I don't understand is I went in to /usr/src/linux, changed the option for auto assembly:

Code:
gentoo linux # cat .config | grep MD_AUTO
# CONFIG_MD_AUTODETECT is not set


We're sure it has been unset now and we do:

Code:
gentoo linux # make && make modules_install


Then copy it to /boot:

Code:
gentoo linux # cp arch/x86_64/boot/bzImage /boot/kernel-current
/sbin/lilo -v


So I've disabled the kernel option for auto assembly, compiled it with the option disabled. Copied it to /boot and ran lilo. The dmesg is still showing me:

Code:
gentoo boot # dmesg | grep md
[    0.297960] md: raid1 personality registered for level 1
[   61.269083] md: md1 stopped.
[   61.269574] md: bind<sdb1>
[   61.269696] md: bind<sda1>
[   61.269994] md/raid1:md1: active with 2 out of 2 mirrors
[   61.270020] md1: detected capacity change from 0 to 127860736
[   61.270277]  md1:
[   61.302167] md: md2 stopped.
[   61.302571] md: bind<sdb2>
[   61.302661] md: bind<sda2>
[   61.302885] md/raid1:md2: active with 2 out of 2 mirrors
[   61.302899] md2: detected capacity change from 0 to 4095672320
[   61.303106]  md2: unknown partition table
[   61.306814] md: md3 stopped.
[   61.307289] md: bind<sdb3>
[   61.307395] md: bind<sda3>
[   61.307631] md/raid1:md3: active with 2 out of 2 mirrors
[   61.307648] md3: detected capacity change from 0 to 100000530432
[   61.308636] md: Autodetecting RAID arrays.
[   61.308643] md: could not open unknown-block(8,1).
[   61.308648] md: could not open unknown-block(8,2).
[   61.308651] md: could not open unknown-block(8,3).
[   61.308732] md: invalid raid superblock magic on sda4
[   61.308735] md: sda4 does not have a valid v0.90 superblock, not importing!
[   61.308743] md: could not open unknown-block(8,17).
[   61.308747] md: could not open unknown-block(8,18).
[   61.308751] md: could not open unknown-block(8,19).
[   61.308823] md: invalid raid superblock magic on sdb4
[   61.308826] md: sdb4 does not have a valid v0.90 superblock, not importing!
[   61.308831] md: Scanned 8 and added 0 devices.
[   61.308832] md: autorun ...
[   61.308834] md: ... autorun DONE.
[   61.310843]  md3: unknown partition table
[   61.333426] EXT4-fs (md3): couldn't mount as ext3 due to feature incompatibilities
[   61.333599] EXT4-fs (md3): couldn't mount as ext2 due to feature incompatibilities
[   61.375083] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: (null)


wuhh? Why is it still doing

Code:
[   61.308636] md: Autodetecting RAID arrays.


Is this still kernel auto assembly?

After booting the initramfs and SSH'ing into it it shows me:

Code:
remote rescueshell ~ # dmesg | grep md
[    0.297966] md: raid1 personality registered for level 1


No kernel assembly being done there. That looks good. The arrays look assembled:

Code:
remote rescueshell ~ # mdadm --detail /dev/md3
/dev/md3:
        Version : 0.90
  Creation Time : Mon Jan  7 19:16:15 2013
     Raid Level : raid1
     Array Size : 97656768 (93.13 GiB 100.00 GB)
  Used Dev Size : 97656768 (93.13 GiB 100.00 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 3
    Persistence : Superblock is persistent

    Update Time : Mon Feb 10 08:38:30 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : e7c1fab3:a11c0038:a65ff077:61ffda2d
         Events : 0.10695

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3


But then after doing a resume-boot and logging on it appears the kernel is still trying auto assembly?

Code:
[   60.964876] md: Autodetecting RAID arrays.


Apparently one does not need the /dev/disk/by-uuid nodes to use UUID's in the fstab? I've done this now and it boots fine:

Code:
UUID=9a54f416-e6ef-411d-969a-7e3b157299ab       /boot                    ext4            defaults,noatime        1       2
UUID=ecb1e0e4-81c3-4d81-a96b-48cd39b2fffe       none                     swap            sw                      0       0
UUID=c25c891a-10f4-403b-8449-66a75c1d1530       /                        ext4            noatime                 0       1

proc                                            /proc                    proc            defaults                0       0
shm                                             /dev/shm                 tmpfs           nodev,nosuid,noexec     0       0



waaaah! even if I pass raid=noautodetect:

Code:
[    0.806116] md: Autodetecting RAID arrays.
[    0.806121] md: could not open unknown-block(8,17).
[    0.806124] md: could not open unknown-block(8,18).
[    0.806127] md: could not open unknown-block(8,19).
[    0.806211] md: invalid raid superblock magic on sdb4
[    0.806214] md: sdb4 does not have a valid v0.90 superblock, not importing!
[    0.806220] md: could not open unknown-block(8,1).
[    0.806222] md: could not open unknown-block(8,2).
[    0.806225] md: could not open unknown-block(8,3).
[    0.806296] md: invalid raid superblock magic on sda4
[    0.806298] md: sda4 does not have a valid v0.90 superblock, not importing!
[    0.806301] md: Scanned 8 and added 0 devices.
[    0.806302] md: autorun ...
[    0.806303] md: ... autorun DONE.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Mon Feb 10, 2014 10:49 pm    Post subject: Reply with quote

I guess something in your initramfs (or init system) is running raidautorun (busybox') for some reason.

For me that is the only way to produce this message despite autodetect being disabled.

Code:

[43423.349877] md: Autodetecting RAID arrays.
[43423.349879] md: Scanned 0 and added 0 devices.
[43423.349880] md: autorun ...
[43423.349880] md: ... autorun DONE.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Tue Feb 11, 2014 7:08 am    Post subject: Reply with quote

frostschutz wrote:
I guess something in your initramfs (or init system) is running raidautorun (busybox') for some reason.

For me that is the only way to produce this message despite autodetect being disabled.

Code:

[43423.349877] md: Autodetecting RAID arrays.
[43423.349879] md: Scanned 0 and added 0 devices.
[43423.349880] md: autorun ...
[43423.349880] md: ... autorun DONE.


Hiya, I'm almost there. I've now got:

a) Working initramfs that mounts the mdadm arrays and root by UUID.
b) UUID's in the fstab and lilo.conf
c) Disabled kernel auto detection (yaaay, future proof now)
d) SSH enable recovery boot option. VERY handy if your server is headless.

I stuck in a PCI graphics card but couldn't get it to work. I'm now in doubt whether 'modern' mainboards even support traditional PCI graphics. Anyhow;

I have both init.d scripts mdadm and mdraid enabled. My guess is (by looking at them) is that mdadm only monitors whatever it can find by scanning, I'd like to keep this one as I've mail enabled it to notify me of any failures.

Code:
start() {
        ebegin "Starting mdadm monitor"
        mdadm --monitor --scan \
                --syslog \
                --daemonise \
                --pid-file /var/run/mdadm.pid \
                ${MDADM_OPTS}
        eend $?
}


mdraid however appears to trigger the assembly?

Code:
start() {
        local output

        ebegin "Starting up RAID devices"
        output=$(mdadm -As 2>&1)
        eend $? "${output}"

        local pat="/dev/md_d*"
        set -- ${pat}
        if [ "$*" != "${pat}" ] ; then
                ebegin "Creating RAID device partitions"
                blockdev "$@"
                eend $?
                # wait because vgscan runs next, and we want udev to fire
                sleep 1
        fi

        return 0
}


Would an mdadm assemble/scan cause the dmesg entry I'm seeing? I was expecting that an mdadm -As would assemble in userspace and not cause kernel messages about auto assembly. Obviously I can take out the mdraid service from the runlevels as all the arrays are assembled before mdraid but it also takes care of gracefully stopping the raid devices:

Code:
stop() {
        local output

        # XXX: Maybe drop this check ?
        [ ! -e /etc/mdadm/mdadm.conf ] && [ ! -e /etc/mdadm.conf ] && return 0

        ebegin "Shutting down RAID devices (mdadm)"
        output=$(mdadm -Ss 2>&1)
        eend $? "${output}"
}


If I would be to take out the mdraid nothing would be stopping the arrays upon shutdown/reboot? Do I need to gracefully stop the arrays to prevent corruption? What's the alternative? That, and the fact my /dev/disk/by-uuid does not exist. I think UDEV should be responsible for creating these device nodes? I've not found any relevant information so far. The only place I'm still using the static device node is in my lilo.conf:

Code:
boot=/dev/md1
raid-extra-boot=mbr-only
prompt
timeout=300
default=current
lba32


I would like to do something like:

Code:
boot=/dev/disk/by-uuid/xxxxxxxx-xxxxxxxx-xxxxxxxx-xxxxxxxx
raid-extra-boot=mbr-only
prompt
timeout=300
default=current
lba32


I have taken mdraid out of my runlevels now but still get:

Code:
gentoo smeltekw # dmesg | grep md:
[    0.298009] md: raid1 personality registered for level 1
[    0.787167] md: md1 stopped.
[    0.787597] md: bind<sdb1>
[    0.787695] md: bind<sda1>
[    0.820401] md: md2 stopped.
[    0.820840] md: bind<sdb2>
[    0.820937] md: bind<sda2>
[    0.824660] md: md3 stopped.
[    0.825108] md: bind<sdb3>
[    0.825205] md: bind<sda3>
[    0.826161] md: Autodetecting RAID arrays.
[    0.826166] md: could not open unknown-block(8,17).
[    0.826170] md: could not open unknown-block(8,18).
[    0.826172] md: could not open unknown-block(8,19).
[    0.826250] md: invalid raid superblock magic on sdb4
[    0.826252] md: sdb4 does not have a valid v0.90 superblock, not importing!
[    0.826257] md: could not open unknown-block(8,1).
[    0.826260] md: could not open unknown-block(8,2).
[    0.826262] md: could not open unknown-block(8,3).
[    0.826329] md: invalid raid superblock magic on sda4
[    0.826331] md: sda4 does not have a valid v0.90 superblock, not importing!
[    0.826334] md: Scanned 8 and added 0 devices.
[    0.826335] md: autorun ...
[    0.826336] md: ... autorun DONE.


YESS, succes:

Code:
gentoo smeltekw # dmesg | grep md:
[    0.297969] md: raid1 personality registered for level 1
[    0.784165] md: md1 stopped.
[    0.784676] md: bind<sdb1>
[    0.784797] md: bind<sda1>
[    0.817293] md: md2 stopped.
[    0.817705] md: bind<sdb2>
[    0.817798] md: bind<sda2>
[    0.821578] md: md3 stopped.
[    0.821993] md: bind<sdb3>
[    0.822084] md: bind<sda3>


betterinitramfs function for softraid includes:

Code:
InitializeSoftwareRaid() {
        einfo "Scaning for software raid arrays."
        if ! [ -f '/etc/mdadm.conf' ]; then
                run mdadm --examine --scan > /etc/mdadm.conf
        fi
        run mdadm --assemble --scan
        run mdadm --auto-detect
}


I've changed it and repackaged the initramfs:

Code:
InitializeSoftwareRaid() {
        einfo "Scaning for software raid arrays."
        if ! [ -f '/etc/mdadm.conf' ]; then
                run mdadm --examine --scan > /etc/mdadm.conf
        fi
        run mdadm --assemble --scan
        # run mdadm --auto-detect
}


Commented out the mdadm --auto-detect and all is well now. SlashBeast, any reason to do an auto-detect? Shouldn't mdadm --assemble --scan have picked up anything that can be assembled?

Regarding the /dev/disk/by-uuid, I believe that /lib/udev/rules.d/60-persistent-storage.rules is responsible for for creating the by-uuid nodes?

Code:
# by-label/by-uuid links (filesystem metadata)
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"


I'm not a hero with udev rules but I read that it will create the by-uuid symlinks for anything that is a filesystem, other or crypto type. The MD devices are apparently classified as 'disk' and thus the links are not created?

Code:
gentoo rules.d # udevadm info /dev/md3
P: /devices/virtual/block/md3
N: md3
S: root
E: DEVLINKS=/dev/root
E: DEVNAME=/dev/md3
E: DEVPATH=/devices/virtual/block/md3
E: DEVTYPE=disk
E: MAJOR=9
E: MINOR=3
E: SUBSYSTEM=block
E: USEC_INITIALIZED=91733


okok, long post now but last update:

Code:
gentoo / # rm /etc/udev/rules/d/64-md-raid.rules
gentoo / # udevadm trigger
gentoo / # ll /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 9 Feb 11 10:49 9a54f416-e6ef-411d-969a-7e3b157299ab -> ../../md1
lrwxrwxrwx 1 root root 9 Feb 11 10:49 c25c891a-10f4-403b-8449-66a75c1d1530 -> ../../md3
lrwxrwxrwx 1 root root 9 Feb 11 10:49 ecb1e0e4-81c3-4d81-a96b-48cd39b2fffe -> ../../md2


In a nutshell, there was a symlink pointing to /dev/null to prevent udev from assembling the arrays. I've removed the symlink so the 64-md-raid.rules in /lib/udev/rules.d is parsed which created the by-uuid nodes for MD devices. I've commented out the ARRAY lines in mdadm.conf as I shouldn't need them anymore now the initramfs assembles them.

Last finishing touch:

Code:
gentoo / # cat /etc/lilo.conf
boot=/dev/disk/by-uuid/9a54f416-e6ef-411d-969a-7e3b157299ab
raid-extra-boot=mbr-only
prompt
timeout=300
default=current
lba32

image=/boot/kernel-current
        label=current
        read-only
        append="softraid root=UUID=c25c891a-10f4-403b-8449-66a75c1d1530"
        initrd=/boot/initramfs-current
        vga=791

image=/boot/kernel-recovery
        label=recovery
        read-only
        append="softraid root=UUID=c25c891a-10f4-403b-8449-66a75c1d1530 sshd sshd_wait=120 binit_net_if=eth0 binit_net_addr=192.168.10.10/24"
        initrd=/boot/initramfs-recovery
        vga=791


VERY VERY NICE. Thank you all very much for your help and assistance! I think this topic now has quite some info for anyone who is looking for some info on initramfs, UUID's and mdadm.

8) 8) 8)


Aaaaaand I just found out that LILO does not work with metadata v1.20 but that's a different story.

Code:
gentoo boot # /sbin/lilo
Fatal: Incompatible Raid version information on /dev/md1   (RV=0.90 GAI=1.2)


So re-created /boot with --metadata=0.90 to maintain compatibility with LILO. Any major drawbacks of using meta 0.90 instead of 1.20 apart from the number of devices and the 2TiB limit? I'll have to stick to 0.90 until LILO gets patched I guess. Already updated to LILO v24.0 but still the same problem..

:roll:
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Tue Feb 11, 2014 8:19 pm    Post subject: Reply with quote

the mdadm --auto-detect may not be the best idea ever - I just never had issue with it so it may be just leftover from when I added the feature.
Code:
Auto-detect
    This mode does not act on a specific device or array, but rather it requests the Linux Kernel to activate any auto-detected arrays.


I've just commited a fix to drop this all together to devel branch of better-initramfs, thanks for the report.

When you assemble arrays at initramfs level you don't need userspace support for it on system level (the init scripts) unless you want monitor the arrays etc. Same goes for LVM in this very cause.

About the by-uuid symlnks and udev - I am not using udev so I can't help you much here, all my gentoo deployments run mdev(-like-a-boss) and md* devices trigger code to create /dev/disk/by-uuid/ symlinks whatsoever.

Its not common for bootloader to support many different metadata types, grub2 is the exception but its insane kind of software that in some cases may fail to be installed as it cannot fit into mbr! I do use extlinux as bootloader and put /boot on metadata-1.0 (the only supported metadata version by extlinux), then when kernel and initramfs is mapped into memory, all other devices like rootfs, swap etc are metadata-1.2.

Glad you got it working.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54028
Location: 56N 3W

PostPosted: Tue Feb 11, 2014 10:57 pm    Post subject: Reply with quote

pakjebakmeel,

/dev/disk/by-uuid is not required to use UUIDs

/dev/disk/by-uuid is certainly missing on my static /dev set up but UUIDs work fine.

Static /dev means what is says, no udev, no eudev, no mdev, no DEVTMPFS, no auto anything.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pakjebakmeel
n00b
n00b


Joined: 13 Feb 2012
Posts: 48

PostPosted: Wed Feb 12, 2014 9:47 am    Post subject: Reply with quote

SlashBeast wrote:
the mdadm --auto-detect may not be the best idea ever - I just never had issue with it so it may be just leftover from when I added the feature.
Code:
Auto-detect
    This mode does not act on a specific device or array, but rather it requests the Linux Kernel to activate any auto-detected arrays.


I've just commited a fix to drop this all together to devel branch of better-initramfs, thanks for the report.

When you assemble arrays at initramfs level you don't need userspace support for it on system level (the init scripts) unless you want monitor the arrays etc. Same goes for LVM in this very cause.

About the by-uuid symlnks and udev - I am not using udev so I can't help you much here, all my gentoo deployments run mdev(-like-a-boss) and md* devices trigger code to create /dev/disk/by-uuid/ symlinks whatsoever.

Its not common for bootloader to support many different metadata types, grub2 is the exception but its insane kind of software that in some cases may fail to be installed as it cannot fit into mbr! I do use extlinux as bootloader and put /boot on metadata-1.0 (the only supported metadata version by extlinux), then when kernel and initramfs is mapped into memory, all other devices like rootfs, swap etc are metadata-1.2.

Glad you got it working.


Yes that was what was triggering the kernel messages making it look like kernel auto detection (which it is in effect but triggered by mdadm from userspace).

The only reason I thought I would need that mdraid daemon is because it gracefully stops the arrays when rebooting/shutting down. It's fine without this daemon so probably mounting them read-only is good enough.

/dev/disk/by-uuid nodes are now being created for my md devices, that's solved too.

NeddySeagoon wrote:
pakjebakmeel,

/dev/disk/by-uuid is not required to use UUIDs

/dev/disk/by-uuid is certainly missing on my static /dev set up but UUIDs work fine.

Static /dev means what is says, no udev, no eudev, no mdev, no DEVTMPFS, no auto anything.


I've figured that out now but I wasn't sure before (never used it).. I did need the by-uuid to be able to put this into lilo.conf (which is working now):

Code:
gentoo / # cat /etc/lilo.conf | grep boot=/
boot=/dev/disk/by-uuid/035afafb-2e9b-4a57-9439-7ece158f04d9
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum