cannot boot with systemd with /usr on a separate lvm FS

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

Upgrading gnome to 3.8 required the system to be booted in systemd. I followed the instructions http://wiki.gentoo.org/wiki/Systemd, but could not get the system to boot with /usr/lib/systemd/systemd as init.

I have / and /usr on two different file systems on LVM2. I used genkernel from genkernel-next to generate the kernel image and initramfs. I am still at the stage of using grub:0 to boot. My grub line reads:

dambacher · Posted: Thu Dec 26, 2013 6:50 am Post subject:

You overlooked the one big drawback of systemd: it needs /usr/lib in boot stage and therefore can not run with separate /usr partitions.

To work around this, you need an initramfs to early mount your /usr partitition before starting systemd.
It seems this does not work correctly with your setup.
Check why your initrd does not correctly mount your /usr/ partition before starting systemd.

Look for http://wiki.gentoo.org/wiki/Early_Userspace_Mounting or try
http://wiki.gentoo.org/wiki/Dracut

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

I was actually quite aware that /usr needed to be mounted in the boot stage. I followed the instructions in http://wiki.gentoo.org/wiki/Initramfs/HOWTO which was a link from http://wiki.gentoo.org/wiki/Systemd, to build the initramfs using genkernel (built from genkernel-next). I thought it would mount /usr when it loaded initramfs, but obviously I was wrong.

The page that dambacher quoted, http://wiki.gentoo.org/wiki/Early_Userspace_Mounting, seemed inconsistent with what I had on my system. It used a file /usr/src/initramfs/initramfs_list which did not exist on my system (3.10.17). It seemed like the file /etc/initramfs.mounts was used instead. In that file, I had:

NeddySeagoon · Posted: Sun Dec 29, 2013 3:31 pm Post subject:

hokiau,

The initramfs is a cpio archive of cpio archives. Use cpio to decompose the initramfs anu read its init script.

I refuse to depend on black magic to make my initramfs and rolling your own isn't that difficult.
I recommend that you use lddtree to discover the libs you need as the list under

dambacher · Posted: Sun Dec 29, 2013 6:26 pm Post subject:

hi hokiau

I can not see anything wrong on how you did your your initramfs.
if you want to look at the initramfs - they are compressed cpio archives, you have to rename and unxz them before using cpio -i < $FILE to unpack. the logic you search for is in init and etc/initrd.d/00-rootdev.sh

I know it is nearly impossible to get a boot log to show us when something is failing in early boot - I used my camera in video mode sometime.

I would like to know if you can deduct from the kernel log of your failing boot
1) if your lvm gets inititalized correctly (you see the devices get activated, how are they named?)

2) that udev or mdev is activated and devtmpfs gets mounted so you can actually see the new devices in userspace

3) you have written "unable to mount ../dm-1 on /usr" - why do you use a relative path in mount?
the initramfs is programmed to mount your real root and check your fstab, if you have some strage things in there it won_t work.
look for something like "Unable to mount ${dev} for ${fs}" in your last lines.

4)add "debug" to your kernel command line which gets you a debug shell just before switching roots to check on the live system if everything is correct and maybe manually mount /usr

What about trying it the other way round - at least preliminary:
You can copy the boot relevant /usr/lib/systemd files to your first dm partitition, they get mounted over during the boot process.
this way you could boot without a ram disk and check that your kernel works correctly.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

I did not use a relative path to mount my file systems. the ../dm-1 thing was done by the system script, not mine. I guess the system scripts look at /etc/fstab and derive ../dm-1 from some paths there.

Apparently the system was able to mount some file systems in my volume group, because my / was also a logical volume. Only my /boot partition was on a real disk partition. E.g. my / is /dev/vg/slash and my /usr is /dev/vg/usr. It's able to mount / but not /usr for some reason.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

Like NeddySeagoon suggested, and following https://wiki.gentoo.org/wiki/Early_Userspace_Mounting, I tried to build initramfs manually. I used lddtree to determine the exact list of .so that /sbin/fsck and /sbin/fsck.ext4 needed. I actually did the same for /sbin/lvm as well.

Built the initramfs. Adjusted grub:0's config to boot with this initramfs. Then I got:

Mounting rootfs
mount: mounting /dev/ram0 on /mnt/root failed: Invalid argument
Checking local filesystem : /usr
/init: line 101: /sbin/fsck not found
Filesystem couldn't be fixed

Something went wrong. Dropping you to a shell.
/bin/sh: can't access tty; job control turned off

I could not type any command, but needed to push the power button to reset.

I specified /sbin/fsck and /sbin/fsck.ext4 in the initramfs_list file.

NeddySeagoon · Posted: Mon Dec 30, 2013 12:43 pm Post subject:

hokiau,

The kernel will find the initramfs if one is provided root=/dev/ram0 on the kernel cammand line is no longer required.

Post the list of files you used to make your initramfs
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

Got rid of the root=/dev/ram0 argument. It still would not boot, giving the same error as before:

NeddySeagoon · Posted: Mon Dec 30, 2013 6:38 pm Post subject:

hokiau,

I do it like this See Code Listing 4.9: Content of package.use
and Code Listing 5.2: /root/initrd/initramfs_list.

As you will be using DEVTMPFS you do not need any of the nod entries.

unknown-block(1,0) is /dev/ram0 and that should not be used any more. Put your kernel .config onto a pastebin please
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

I'm sorry, but how do I paste something in the pastebin? I don't seem to find a pastebin.

I have in .config

NeddySeagoon · Posted: Tue Dec 31, 2013 12:29 pm Post subject:

hokiau,

The easiest way is with wgetpaste. It can paste whole files or the output of commands.
You will need to emerge it unless your liveCD provides it.

When wgetpaste is successful, it returns a URL for your paste.
You post the URL on the forum.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

The puzzling thing was that I was able to boot the same kernel, same initramfs (generated by genkernel-next) if I did not have the real_init=/usr/lib/systemd/systemd kernel parameter.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

OK. I pasted my .config to http://bpaste.net/show/163366/

Thanks!

NeddySeagoon · Posted: Tue Dec 31, 2013 12:55 pm Post subject:

hokiau,

While I look over your kernel, what does the script at /usr/lib/systemd/systemd contain?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

It's a binary. I believe it's a package required by gnome 3.8. It was built from the sys-apps/systemd package.

http://wiki.gentoo.org/wiki/GNOME/3.8-upgrade-guide said I needed to install systemd when I upgraded to gnome 3.8.

NeddySeagoon · Posted: Tue Dec 31, 2013 3:19 pm Post subject:

hokiau,

I had forgotten that systemd is binary almost everything as I'm not a systemd user.

Your kernel .config looks good and as it boots without the

nextgen · Posted: Sun Jan 12, 2014 5:59 pm Post subject:

This systemd upgrade nightmare has brought me an unprecedented amount of pain and suffering in 14 years of gentoo experience. I finally found a workaround.

The error

hokiau · n00b Joined: 24 Dec 2013 Posts: 18

Thanks, nextgen! It worked, after I took your advice and mount /dev/dm-1 on /usr. That should have been documented, shouldn't it?

Now I can proceed to the next step of upgrading to gnome 3.8. Thanks again to everybody who had helped!

nextgen · Posted: Mon Jan 13, 2014 3:43 am Post subject:

Glad it worked!

I agree it should have been documented somewhere. It might have saved me days searching the web in vain until I accidentally observed the ../dm-3 symlink when browsing my LVM devices (/dev/<volume group>/*), which gave me the idea to adjust the fstab file.

Works with grub legacy.

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

note that this will break any time you change anything about your setup that makes the dm-x numbers change. They're dynamic (first come first serve).

You should really mount by UUID instead or find a device name that is actually available.

nextgen · Posted: Tue Jan 14, 2014 12:55 am Post subject:

Can you give more detail on how to mount by UUID? Under /dev/disk/by-uuid, I still find relative-path symlinks (such as ../../dm-3). Hence I suppose that setting up those devices in /etc/fstab won't yield a bootable initramfs.

damocles · n00b Joined: 05 Feb 2010 Posts: 4

Hello,

I had this problem with genkernel-next and the culprit seems this code fragment in [src_initramfs]/etc/initrd.d/00-rootdev.sh

depontius · Advocate Joined: 05 May 2004 Posts: 3509

nextgen · Posted: Mon Feb 17, 2014 9:48 pm Post subject: