LVM børked; how do I rebuild? [SOLVED]

NeddySeagoon · Posted: Sat Apr 14, 2012 9:23 pm Post subject:

ExecutorElassus,

I cannot think of any reason for a drive to drop out of a raid set and not leave a sign in your log.

If the system shut down correctly and something happened to make the next assemble fail, you might miss it as it may not be possibe to log, particulary if the log was on the raid that did not start.
dmesg keeps a ring buffer in RAM, so the dmesg command works for the content of the buffer, even if the log location is not mounted. Of course, you lose that on power down.

I've not been around much today - my system has been mostly failing to boot while I did the udev-182 upgrade.
It only boots now if I skip the fsck in the initrd.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Sat Apr 14, 2012 9:30 pm Post subject:

Hi Neddy,

well, then you can tell me how to get udev-182 to play nice with my RAIDed /usr once I have it working. :wink:

For the assembly problems, would it be useful to make use of the mdadm.config file? Like, actually specify the arrays manually? Right now, it seems to be building them based on their own superblocks (or whatever else mdadm uses when there's no config file), and maybe stating explicitly which partitions go into which array might make things work better.

In any case, once the recovery is done, I'll run dd as per your instructions to check for read errors. If dmesg says nothing, I'll try rebooting later tonight (tomorrow morning) and report back.

Grr... my failures are always the ones that make no sense.

Thanks again for the help, and good luck with udev. Have you tried using the earlymount script posted here?

Cheers,

EE

NeddySeagoon · Posted: Sat Apr 14, 2012 9:56 pm Post subject:

ExecutorElassus,

raid is OK, I'm having problems with separate /usr and /var which are in lvm2 on raid5.
I'm going with the wiki.gentoo.org page but with additions for raid and lvm.

The raid bits work fine - the lvm ones don't ... not yet.
It looks like I don't get any /dev nodes for the logical volumes.

Do you use raid autodetect in the kernel.?
If so, to use an mdadm.conf file you will need to either move to an initrd or not have root on raid, since root on raid needs the raid assembled before root is mounted.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Sat Apr 14, 2012 10:09 pm Post subject:

Hi Neddy,

I … think I have an initrd? But I do for certain have / on a RAID1 (mirrored, yes?) array. So is /boot, for that matter.

That forum seems to have a script to run pre-mount and checking on RAID/LVM at the sysinit level. I don't know for certain if that would work for my setup (which is the one specified in the gentoo RAID/LVM2 quick-install handbook: /boot on RAID1, x swap partitions, / on RAID1, and then a big RAID5 for everything else [in my case, /usr, /opt, /var, /var/tmp, /usr/portage, /usr/portage/distfiles, and /home all get an lv, along with 1.7TB of storage partitions). Since the kernel source directories are under /usr/src/linux, which is on the big RAID5, is that going to cause problems with the kernel loading (with <udev-182)?

The earlymount script seemed to work okay, but then my drives shut down, and I rebooted into a broken system.

Since I have a couple hours to go before I can test the md127 that's rebuilding, lemme plug you with questions. Is it possible to change the role numbers of an active array? I'm still bothered by sdc4 being [3] instead of [1] like all the other sdcX partitions are in their respective arrays, and wonder if that might be causing problems.

So far, all dmesg says is the mdadm message I posted before, and

NeddySeagoon · Posted: Sat Apr 14, 2012 10:37 pm Post subject:

ExecutorElassus,

The role numbers don't matter. The do not need to map to the partitions in any particular order.
What happens in /usr/src has no impact on booting. The kernel you boot is a binary file, normally in /boot
That you have /boot on raid1 rells me that yur /boot raid is a version 0.9 superblock, so could be kernel auto assebled. Grub is not interested in that since raid assembly, however its done, happens after grub has done its stuff and exited.

You may have an initrd to assemble the rest and start your logical volume. You may not too. look in grub.conf. Do you have an initrd entry user tour kernel line ?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Sat Apr 14, 2012 10:44 pm Post subject:

Hi Neddy,

nope, no initrd line. So, I guess that means no initrd.

Both /boot and / are indeed on 0.9 superblocks, as instructed by the install guide. The RAID5 is 1.2.

What I've heard about initrd or initramfs is that both of them add to boot time, and are thus undesirable (or so I gathered from that thread about the earlymounts script). But I'll worry about that part of setup once I've sorted why my RAID5 keeps barfing.

90 minutes to go from this message. I imagine you'll be off to bed by then. Should I then proceed with 'dd', and then try rebooting?

Cheers,

EE

NeddySeagoon · Posted: Sat Apr 14, 2012 10:52 pm Post subject:

ExecutorElassus,

Yes. The dd is harmless. As you have no initrd, you must use kernel auto assembly, so you can't use an mdadm.conf.

For you, your initrd would only assemble your raid and start your lvm, which has to happen anyway.
It would not be a huge bloated full of kernel modules initrd that you rebuild with every kernel. It would just be a space for some userspace tools. It need not increase the boot time.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Sat Apr 14, 2012 10:58 pm Post subject:

Hrm,

and as I understand it, this is what's causing problems with udev-182, yes? So, an initrd would mount the MDs, and start the RAID/LVM, earlier in the boot process, yes?

Maybe I should just suck it up and use one. Is there documentation on it?

But first things first. I'll see if I can get this RAID5 array to survive a reboot, and then proceed wit the rest.

I'll report back as soon as dd is finished (unless there are other things I should do in the meantime?)

Thanks again,

EE

ExecutorElassus · Posted: Sun Apr 15, 2012 9:44 am Post subject:

Okay, how about this:

dd finished with no messages in dmesg. So I reboot. As before, sda4 gets moved into md125 (a bogus array), and that array fails to start. BUT. I reboot, and on the next bootup, everything is in its correct (active array), partitions are mounted, and fsck checks all the partitions.

This has been happening for some time, now that I think of it: the first reboot always results in a disabled RAID5, but rebooting results in a functional one.

Any idea why?

Cheers,

EE

NeddySeagoon · Posted: Sun Apr 15, 2012 12:14 pm Post subject:

ExecutorElassus,

That sounds like a race condition. Maybe one drive is taking longer then the others to come ready, so the kernel gives up waiting for it.
If it were spin up time related, at the reboot, the drives would already be spun up ... so it would just work.

Spin up time is a wearout paramater reported in smartctl.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Sun Apr 15, 2012 12:28 pm Post subject:

Hi Neddy,

but I'm not booting from a full shutdown. I'm using 'init 6' both times. This morning, after dd finished, I hit 'init 6'. The first boot, I got one drive dropped out of the array. The next boot, it's back in place without my doing anything.

smartctl shows 0 for "Spin Up Time" for all three drives. sdc (the suspect one) does show:

NeddySeagoon · Posted: Sun Apr 15, 2012 1:30 pm Post subject:

ExecutorElassus,

Its difficult for me working though a keyhole ... make friends with wgetpaste and post the whole build log.
The failure message at the end of every failed build tells where the log is.

Warning are just that - warnings and thats all thats in your log snippit.

You need tp look on your drive vendors web site to understand what the raw values mean. They are often several bit fields in a 32 bit vlaue.
The normalised values would be a pass but

ExecutorElassus · Posted: Sun Apr 15, 2012 6:18 pm Post subject:

pastebin is my new imaginary internet boyfriend (sorry): build.log

I'll do some digging with the drive vendor, and let you know inn a sec which ebuilld failed this time.

Cheers,

EE

UPDATE: looking at that log, the only lines which return explicit errors are from a file that belongs to kbproto. I've remerged that, and I'll get back to you about xorg-server once the @world update is done.

ExecutorElassus · Posted: Sun Apr 15, 2012 8:30 pm Post subject:

I got a little further in building X. The new build.log (which still fails) is here.

I'm also failing my @world emerge on pango. The log for that is here.

Any ideas what's going wrong with those?

Cheers,

EE
UPDATE:pango is fixed. It was libXft, not Xutil, that was choking.

ExecutorElassus · Posted: Mon Apr 16, 2012 4:44 pm Post subject:

Okay, I've gotten a bit further. Now I'm on kdepimlibs, which fails on the following:

NeddySeagoon · Posted: Mon Apr 16, 2012 6:06 pm Post subject:

ExecutorElassus,

I've never built KDE, so I'm not a lot of help here. Try searching on bugs.gentoo.org to see if its a known issue.
If so, there may be a fix for it there too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

NeddySeagoon · Posted: Mon Apr 16, 2012 6:07 pm Post subject:

ExecutorElassus,

I've never built KDE, so I'm not a lot of help here. Try searching on bugs.gentoo.org to see if its a known issue.
If so, there may be a fix for it there too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

ExecutorElassus · Posted: Mon Apr 16, 2012 8:50 pm Post subject:

In fact, KDE bugs aren't even handled by the gentoo tracker: they have to be filed upstream. I did so, and we'll see what happens.

I have, however, managed to get xorg back up, so I'm back into my wm (hurrah!). I now have just four packages that won't update for various reasons, and then I imagine I'll be finding stray misnamed files for months.

But at least the system is up, and (sorta) stable, so now let's get back to the original issue: mdadm seems to be randomly dropping one of the drives out of the RAID array on bootup, with no errors in the log from shutdown (and, since logging doesn't work on the non-RAID system, nothing but dmesg to tell me what might have gone wrong at boot [but I can't scroll through that, so it isn't helpful]).

So, you're suggesting it's just spool-up times? Is there anything else that might cause it?

Cheers,

EE

NeddySeagoon · Posted: Mon Apr 16, 2012 8:55 pm Post subject:

ExecutorElassus,

ExecutorElassus · Posted: Mon Apr 16, 2012 8:59 pm Post subject:

when the array is inactive, I do not have access to less. Can I pipe it to a regular file? This presumes I can reboot into a working array, since nfs and ssh are also inaccessible without the RAID5. *sadpanda*

ALSO: I just noticed, that 'ld' will, in the middle of some compiles - in this case firefox - chew up close to 40% of my RAM (about 1.6GB). Is that normal? Or is this an unfortunate consequence of my file system being riddled with files identified as directories, and actual files emerged into backups, etc. etc.?

Cheers,

EE

NeddySeagoon · Posted: Mon Apr 16, 2012 9:58 pm Post subject:

ExecutorElassus,

You should be a little more adventuerous. Try

ExecutorElassus · Posted: Mon Apr 16, 2012 10:06 pm Post subject:

Okay.

For the time being, I'm only going to do that if 1) the array boots inactive and separated, and 2) won't boot back normal the next (one or more) reboot. But thank you for the tips. Okay. Now that I have a system I can (mostly) work with, let's talk about that initrd. So, first step is to build mdadm, lvm, and busybox with +static? I'm fine with using package.use. What next?

Cheers,

EE

NeddySeagoon · Posted: Tue Apr 17, 2012 6:20 pm Post subject:

I used a combination of this wiki post which covers the straight forward (relatively) case of root on lvm on raid and this
wiki thread covering the new bit for mounting the separate partitions.
To actually build the initramfs, I used the kenel provided script.
The second wiki article is writen for a 32 bit install. That cause me some grief as I'm 64bit no-multilib.

I like to build my initrd by hand in /root/initrd/ with everything I need there. That way I know it won't change with a

ExecutorElassus · Posted: Wed Apr 25, 2012 10:58 pm Post subject:

Hi Neddy!

so, after 10 days of basically working okay, I rebooted today. Now, when I reboot, the large RAID5 array - the one holding /usr, /var, /home, etc - is active as "auto-read-only" and none of its partitions are mounted.

So, I'm back to where I started. Sorta.

As far as I can tell, the array is fine, and all its drives are active; they just … aren't being mounted by mdadm. Is there a way to re-initialize the system, so that mdadm and lvm re-do mounting and checking everything, and re-load all the stuff that lives on that array?

On a second question, can you think of any reason why that array would always start up auto-read-only?

Thanks,

EE
(and once again, I don't have access to a pager, because I haven't rebuilt busybox, mdadm, or lvm static. Do I need to do anything besides rebuild them with USE="static" to have access to them?)

NeddySeagoon · Posted: Thu Apr 26, 2012 5:41 pm Post subject:

ExecutorElassus,

If busybox, mdam and lvm are not built with USE=static, you need to remake them and rebuild your initrd.
Without the staic USE, they will have never worked in your initrd, never mind being ok for 5 days.

What versions of openrc and udev do you have ?

The is an alternative to the USE=static. You can add the libraries these applications need to your initrd.
I prefer static.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.