LVM børked; how do I rebuild? [SOLVED]

ExecutorElassus · Posted: Thu Apr 26, 2012 5:49 pm Post subject:

openrc is 0.9.9.2.

So, since the RAID5 array isn't loading right now, is it safe to assume that this is because mdadm and lvm aren't built static? If that's the case, I should be able to fix it by booting a liveCD, mounting everything and chrooting in, then simply emerging them as static, yes? Will I need to copy any binaries over to /sbin, or does emerge take care of that?

Cheers,

EE

NeddySeagoon · Posted: Thu Apr 26, 2012 6:19 pm Post subject:

ExecutorElassus,

What version is udev?

If you type

ExecutorElassus · Posted: Thu Apr 26, 2012 9:27 pm Post subject:

Hey Neddy,

I'll have to wait until I get back to my home office Monday evening before I can try mounting everything (I have a music festival this weekend), but I'll report my results once I try.

udev is blocked as per your instructions past 181 (or whatever version starts treating mounting more strictly).

Cheers,

EE

ExecutorElassus · Posted: Mon Apr 30, 2012 11:24 am Post subject:

Hi Neddy,

so, first: 'mount -a' mounts all the drives/partitions without a hitch. No errors in the console, nothing in dmesg. I have udev-171-r5 running.

Now that I have a pager, I can read the dmesg log. I don't see anything too strange. I see complaints about "invalid raid superblock magic on sdX4" which I just take to mean that it isn't v0.90 (and md then says it it consequently not importing the superblock). md126 (the root RAID1 mirror across the three sdX3 partitions) is listed as having an unknown partition table (which makes sense, since it's a logical partition, right?).

Then there is a long gap of info about USB, and udev then starts, before md127 is loaded. md127 appears to load fine (but its included partitions are not mounted).

One more thing: on shutdown ('init 0' or 'init 6') I get an error that md "cannot get exclusive access to md126 [the md array holding / ]," and that the array fails to stop. Could that be causing issues?

Anyway, should I now try to rebuild mdadm, lvm, and busybox as static, and make an initrd?

Thanks,

EE
PS, I notice that if I drop to runlevel 1 - thus shutting down the md arrays - and then reboot, sda4 gets punted off into a single-drive array - md125 - and md127 is stopped (meaning that I have to stop md125, restart md127 degraded, then add /dev/sda4 to it and sit back for a six-hour re-sync). Is there a reason for this? Is there a mismatched UUID somewhere that makes mdadm think the drives are in different arrays? If so, how do I fix it?

NeddySeagoon · Posted: Mon Apr 30, 2012 6:06 pm Post subject:

ExecutorElassus,

I've been there - it all works when you do it by hand.
I *think* but its too difficult to prove, that your openrc is no longer tolerant of udev failures, which have existd for a long time but which are no longer retried.
/usr is not mounted when uden starts, lots of udev things fail and the retries to piuck up the pieces are no longer tried.

The only way is forward, since you need an initrd anyway.
Thats an improved howto over the one I posted earlier in thise thread.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Mon Apr 30, 2012 6:21 pm Post subject:

Hi Neddy,

actually, I think most of this started when I tried to use the earlymount script - I linked it earlier in the thread; it attempts to pre-mount RAID arrays without an initrd - and something *went wrong*.

Anyway, heaven help me, I'll start following your guide once the re-sync is done (in about an hour), and let you know what happens.

Cheers,

EE

ExecutorElassus · Posted: Mon Apr 30, 2012 8:41 pm Post subject:

also, I note that my fstab is listing all my RAID partitions as "/dev/vg/[path]" and not "/dev/mapper/vg-[path-shorthand];" mght that be causing issues?

No matter: I'm replacing them all with UUIDs now, as per your guide. I'll report issues on that thread, and anything that appears to be my system acting schizoid on this one.

Cheers,

EE

ExecutorElassus · Posted: Mon Apr 30, 2012 9:58 pm Post subject:

So, as an example of my system acting wonky, here's what I get when I run 'ldd /sbin/fsck.ext2:"

ExecutorElassus · Posted: Mon Apr 30, 2012 10:17 pm Post subject:

And more on the side of strangeness in setting upyour script:

is there a reason why one of the arrays - specifically, the large one holding everything beyond / and /boot - would have a non-hex UUID? Check this out:

NeddySeagoon · Posted: Mon Apr 30, 2012 10:28 pm Post subject:

ExecutorElassus,

ExecutorElassus · Posted: Mon Apr 30, 2012 10:35 pm Post subject:

so, in the example script, where you have:

NeddySeagoon · Posted: Mon Apr 30, 2012 10:47 pm Post subject:

ExecutorElassus,

You need the UUID of the raid set, not the LVM2 it caries when you assemble the raid.
The safest way to get the UUID of the raid set is to ask madam

ExecutorElassus · Posted: Mon Apr 30, 2012 11:00 pm Post subject:

Hi Neddy,

ah, that's what confused me: the fourth partition on each member drive is a member of a large RAID5 array, which is what is made into /dev/md127. However, that array - as it's an LVM array - does not return a UUID to blkid. That might be useful to add to the guide, since I got confused.

Cheers,

EE

ExecutorElassus · Posted: Tue May 01, 2012 12:06 am Post subject:

Hi Neddy,

I posted to the your guide, but moved it here instead because it seems more to do with my system acting bizarre.

So, I failed to boot, and got dumped to a shell. mdraid is not starting, and consequently the root partition - or, for that matter, /boot - are not getting mounted, and everything stops. I'm getting error messages about being unable to find a boot disk.

I should note: my / is on a RAID1 array; mirrored across the three partitions. Does that make a difference for the UUID from your setup? I changed the kernel line in grub.conf to use the UUID for the md array on which / resides, but it doesn't seem to work (nor if I set it to /dev/md126) .

Any suggestions?

Cheers,

EE

NeddySeagoon · Posted: Tue May 01, 2012 7:37 pm Post subject:

ExecutorElassus,

The mdadm --assembe calls need the UUID of the raid, as determined by mdadm -E /dev/<raid_member>
All members of the same raid set carry the same UUID, which is how mdasm finds them

ExecutorElassus · Posted: Tue May 01, 2012 8:24 pm Post subject:

Hi Neddy,

I was feeding the bootloader the UUID for the / md filesystem as returned by blkid (that is, what the UUID for "/dev/md126" was showing), with no luck.

Also, is there a reason why the first three partitions return a differently-formatted output for "mdadm -E" than the fourth? The three other partitions all return a table at the bottom listing all three devices, major and minor number, raid device, and state; but 'mdadm -E /dev/sdX4' returns the lines for Device Role and Array State as the last two lines. That may not mean anything (or maybe it's because they're in an lvm group? Or it's a v1.2 superblock?). The sdX4 are also not showing up on the blkid list while in the busybox shell.

'mdadm -A' seems to require the UUID of the array, not of the constituent devices. Is there a flag I use to specify them? None of the devices for sdX3 (those which would constitute / on /dev/md126) have an array UUID returned by 'mdadm -E' so the only UUID I know for this array is that of the filesystem. Is there another way to get to it?

thanks,

EE
UPDATE: I assembled it just using the device nodes (ie, 'mdadm -A /dev/md126 /dev/sda3 /dev/sdb3 /dev/sdc3'), and now it's active. However, the UUID it shows is exactly the one I have in the bootloader command line. Any guess why it wouldn't be working?
UPDATE 2: okay, I've rebooted, and I can assemble all my md arrays in the busybox shell. here's what happens when I run 'blkid':

NeddySeagoon · Posted: Tue May 01, 2012 8:36 pm Post subject:

ExecutorElassus,

Show me the output of blkid and mdam -E /dev/sda[1234]
Sight of your initscript would also be good.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Tue May 01, 2012 9:46 pm Post subject:

Hi Neddy,

blkid output (from within busybox, after assembling the three md arrays) is in "UPDATE 2" above.

'mdadm -E /dev/sda1' (copied manually, please kill me):

NeddySeagoon · Posted: Wed May 02, 2012 8:32 pm Post subject:

ExecutorElassus,

Distilling what you posred.

ExecutorElassus · Posted: Wed May 02, 2012 8:50 pm Post subject:

'vg' is indeed the name of my volume group.

okay, so I have the UUIDs wrong in my init (and thus likely also in the bootloader). I can fix the former, but how do I fix the latter? Is there a means from within busybox to remake the initrd, or will I have to use a liveCD?

Cheers,

EE

NeddySeagoon · Posted: Wed May 02, 2012 10:57 pm Post subject:

ExecutorElassus,

You need to use the liveCD to fix the initrd.

The root=uuid= needs to be the uuid of the root filesystem, not that of the underlying raid.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Wed May 02, 2012 11:34 pm Post subject:

Hi Neddy,

alas! I was afraid you'd say that. It'll have to wait until tomorrow night, when I can spend the time opening up the box and plugging in the old IDE CD drive I keep around for this sole purpose.

Once I've booted into the live CD, I know how to start up the md arrays and mount everything. Do I just edit the init script I have stored to match the correct UUIDs, and then repeat the '/usr/src/linux/scripts/gen_initramfs_list.sh -o /boot/initrd.cpio.gz /root/initrd/initramfs_list ' command from your guide?

so, to be clear:
1) the UUID for md126 in the bootloader is for the filesystem, and is returned by 'blkid'
but
2) the UUID for that same array in the init script is for the raid array, and will thus not match, and is the UUID returned by mdadm -E /dev/sdXn

Is that correct?

Cheers,

EE

ExecutorElassus · Posted: Thu May 03, 2012 9:56 am Post subject:

Hi Neddy,

so, I updated your script and now it gets past assembling the md arrays, hurrah!

here's my next problem: my mountpoints in the init script are:

NeddySeagoon · Posted: Thu May 03, 2012 7:34 pm Post subject:

ExecutorElassus,

I was about to post to say only mount /usr and /var but went for a beer instead.
When I got back, you had already done it.

As the initrd does not contain anything kernel specific, there is no need to remake it for every kernel.
Indeed, it uses random binaries from your system, thats a good reason not to remake it unless you really need to.

If there are security updates for the packages in the initrd, do you care?
They cannot be exploited remotely as networking is not started until the initrd has done its thing and been discarded.

When you do update it, give the new initrd a new name. You really don't want to overwrite your only working initrd with a broken one, just like your kernel.

All the initrd really does is to appease >=udev-182 by mounting /usr and /var before the real init script starts udev.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Fri May 04, 2012 1:25 am Post subject:

All right, then this will be initrd-1.0. Hurrah for a feature-complete (ie, it boots!) release! Out of Beta and releasing on time, etc etc.

Well, if I could buy you a beer, I would. Thanks for all the (very patient) help you've provided. I'm (mostly) sure the system - at least as far as having a working lvm for recent udev releases - is functional. I'll mark this as solved. (now I just have to figure out why boinc, kdepimlibs, kgpg, and gnome-settings-daemon won't emerge, but that's more a portage/programming question).

Cheers, mate.

EE