Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
cannot boot with systemd with /usr on a separate lvm FS
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Desktop Environments
View previous topic :: View next topic  
Author Message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Tue Dec 24, 2013 1:24 pm    Post subject: cannot boot with systemd with /usr on a separate lvm FS Reply with quote

Upgrading gnome to 3.8 required the system to be booted in systemd. I followed the instructions http://wiki.gentoo.org/wiki/Systemd, but could not get the system to boot with /usr/lib/systemd/systemd as init.

I have / and /usr on two different file systems on LVM2. I used genkernel from genkernel-next to generate the kernel image and initramfs. I am still at the stage of using grub:0 to boot. My grub line reads:

Code:

kernel /boot/kernel-genkernel-x86_64-3.10.17-gentoo root=/dev/ram0 real_root=/dev/vg/slash dolvm intel_iommu=igfx_off vga=0x31b rootfstype=ext4 real_init=/usr/lib/systemd/systemd
initrd /boot/initramfs-genkernel-x86_64-3.10.17-gentoo


The system would not boot with real_init=/usr/lib/systemd/systemd. It resulted in kernel panic, as it complained that it was "unable to mount ../dm-1 on /usr".

Could someone tell me what was wrong?
Back to top
View user's profile Send private message
dambacher
Apprentice
Apprentice


Joined: 11 Feb 2003
Posts: 288
Location: Germany

PostPosted: Thu Dec 26, 2013 6:50 am    Post subject: Reply with quote

You overlooked the one big drawback of systemd: it needs /usr/lib in boot stage and therefore can not run with separate /usr partitions.

To work around this, you need an initramfs to early mount your /usr partitition before starting systemd.
It seems this does not work correctly with your setup.
Check why your initrd does not correctly mount your /usr/ partition before starting systemd.

Look for http://wiki.gentoo.org/wiki/Early_Userspace_Mounting or try
http://wiki.gentoo.org/wiki/Dracut
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Sun Dec 29, 2013 3:21 pm    Post subject: Reply with quote

I was actually quite aware that /usr needed to be mounted in the boot stage. I followed the instructions in http://wiki.gentoo.org/wiki/Initramfs/HOWTO which was a link from http://wiki.gentoo.org/wiki/Systemd, to build the initramfs using genkernel (built from genkernel-next). I thought it would mount /usr when it loaded initramfs, but obviously I was wrong.

The page that dambacher quoted, http://wiki.gentoo.org/wiki/Early_Userspace_Mounting, seemed inconsistent with what I had on my system. It used a file /usr/src/initramfs/initramfs_list which did not exist on my system (3.10.17). It seemed like the file /etc/initramfs.mounts was used instead. In that file, I had:

Code:

/usr


That would make genkernel generate an initramfs that would include and mount /usr, wouldn't it? /usr/lib is in the same partition as /usr.

I tried dracut as well. But it also failed equally miserably. I take it back. It failed even earlier than if I used the initramfs from genkernel. It was not able to mount the root fs. The error message was

Quote:

Kernel panic - not synching: VFS: unable to mount root fs on unknown-block(1,0)


Is there a way that I can examine initramfs to make sure it would mount /usr?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Sun Dec 29, 2013 3:31 pm    Post subject: Reply with quote

hokiau,

The initramfs is a cpio archive of cpio archives. Use cpio to decompose the initramfs anu read its init script.

I refuse to depend on black magic to make my initramfs and rolling your own isn't that difficult.
I recommend that you use lddtree to discover the libs you need as the list under
Code:
# libraries required by /sbin/fsck.ext4 and /sbin/fsck
may be out of date
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
dambacher
Apprentice
Apprentice


Joined: 11 Feb 2003
Posts: 288
Location: Germany

PostPosted: Sun Dec 29, 2013 6:26 pm    Post subject: Reply with quote

hi hokiau

I can not see anything wrong on how you did your your initramfs.
if you want to look at the initramfs - they are compressed cpio archives, you have to rename and unxz them before using cpio -i < $FILE to unpack. the logic you search for is in init and etc/initrd.d/00-rootdev.sh

I know it is nearly impossible to get a boot log to show us when something is failing in early boot - I used my camera in video mode sometime.

I would like to know if you can deduct from the kernel log of your failing boot
1) if your lvm gets inititalized correctly (you see the devices get activated, how are they named?)

2) that udev or mdev is activated and devtmpfs gets mounted so you can actually see the new devices in userspace

3) you have written "unable to mount ../dm-1 on /usr" - why do you use a relative path in mount?
the initramfs is programmed to mount your real root and check your fstab, if you have some strage things in there it won_t work.
look for something like "Unable to mount ${dev} for ${fs}" in your last lines.

4)add "debug" to your kernel command line which gets you a debug shell just before switching roots to check on the live system if everything is correct and maybe manually mount /usr

What about trying it the other way round - at least preliminary:
You can copy the boot relevant /usr/lib/systemd files to your first dm partitition, they get mounted over during the boot process.
this way you could boot without a ram disk and check that your kernel works correctly.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Mon Dec 30, 2013 8:08 am    Post subject: Reply with quote

I did not use a relative path to mount my file systems. the ../dm-1 thing was done by the system script, not mine. I guess the system scripts look at /etc/fstab and derive ../dm-1 from some paths there.

Apparently the system was able to mount some file systems in my volume group, because my / was also a logical volume. Only my /boot partition was on a real disk partition. E.g. my / is /dev/vg/slash and my /usr is /dev/vg/usr. It's able to mount / but not /usr for some reason.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Mon Dec 30, 2013 9:00 am    Post subject: Reply with quote

Like NeddySeagoon suggested, and following https://wiki.gentoo.org/wiki/Early_Userspace_Mounting, I tried to build initramfs manually. I used lddtree to determine the exact list of .so that /sbin/fsck and /sbin/fsck.ext4 needed. I actually did the same for /sbin/lvm as well.

Built the initramfs. Adjusted grub:0's config to boot with this initramfs. Then I got:

Mounting rootfs
mount: mounting /dev/ram0 on /mnt/root failed: Invalid argument
Checking local filesystem : /usr
/init: line 101: /sbin/fsck not found
Filesystem couldn't be fixed

Something went wrong. Dropping you to a shell.
/bin/sh: can't access tty; job control turned off

I could not type any command, but needed to push the power button to reset.

I specified /sbin/fsck and /sbin/fsck.ext4 in the initramfs_list file.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Mon Dec 30, 2013 12:43 pm    Post subject: Reply with quote

hokiau,

The kernel will find the initramfs if one is provided root=/dev/ram0 on the kernel cammand line is no longer required.

Post the list of files you used to make your initramfs
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Mon Dec 30, 2013 3:12 pm    Post subject: Reply with quote

Got rid of the root=/dev/ram0 argument. It still would not boot, giving the same error as before:
Quote:

Kernel panic - not synching: VFS: unable to mount root fs on unknown-block(1,0)


My /usr/src/initramfs_list looks like:
Code:

# directory structure
dir /proc       755 0 0
dir /usr        755 0 0
dir /bin        755 0 0
dir /sys        755 0 0
dir /var        755 0 0
dir /lib        755 0 0
dir /sbin       755 0 0
dir /mnt        755 0 0
dir /mnt/root   755 0 0
dir /etc        755 0 0
dir /root       700 0 0
dir /dev        755 0 0

# busybox
file /bin/busybox /bin/busybox 755 0 0

# libraries required by /sbin/fsck.ext4 and /sbin/fsck
file    /lib/ld-linux.so.2      /lib/ld-linux.so.2                  755 0 0
file    /lib/libext2fs.so.2     /lib/libext2fs.so.2                 755 0 0
file    /lib/libcom_err.so.2    /lib/libcom_err.so.2                755 0 0
file    /lib/libpthread.so.0    /lib/libpthread.so.0                755 0 0
file    /lib/libblkid.so.1      /lib/libblkid.so.1                  755 0 0
file    /lib/libuuid.so.1       /lib/libuuid.so.1                   755 0 0
file    /lib/libe2p.so.2        /lib/libe2p.so.2                    755 0 0
file    /lib/libc.so.6          /lib/libc.so.6                      755 0 0
file   /lib/libmount.so.1   /lib/libmount.so.1          755 0 0
file   /lib/libuuid.so.1   /lib/libuuid.so.1          755 0 0
file   /lib/libc.so.6      /lib/libc.so.6             755 0 0
file   /usr/lib/libudev.so.1   /usr/lib/libudev.so.1          755 0 0
file   /lib/libdl.so.2      /lib/libdl.so.2             755 0 0
file   /lib/libdevmapper-event.so.1.02 /lib/libdevmapper-event.so.1.02  755 0 0
file   /lib/libdevmapper.so.1.02 /lib/libdevmapper.so.1.02       755 0 0
file   /lib/libreadline.so.6   /lib/libreadline.so.6          755 0 0
file   /lib/libncurses.so.5   /lib/libncurses.so.5          755 0 0

file    /sbin/fsck              /sbin/fsck                          755 0 0
file    /sbin/fsck.ext4         /sbin/fsck.ext4                     755 0 0
file   /sbin/lvm      /sbin/lvm             755 0 0

# our init script
file    /init                   /usr/src/initramfs/init             755 0 0


Thanks for your help. I appreciate it.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Mon Dec 30, 2013 6:38 pm    Post subject: Reply with quote

hokiau,

I do it like this See Code Listing 4.9: Content of package.use
and Code Listing 5.2: /root/initrd/initramfs_list.

As you will be using DEVTMPFS you do not need any of the nod entries.

unknown-block(1,0) is /dev/ram0 and that should not be used any more. Put your kernel .config onto a pastebin please
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Tue Dec 31, 2013 12:21 pm    Post subject: Reply with quote

I'm sorry, but how do I paste something in the pastebin? I don't seem to find a pastebin.

I have in .config
Code:

CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Tue Dec 31, 2013 12:29 pm    Post subject: Reply with quote

hokiau,

The easiest way is with wgetpaste. It can paste whole files or the output of commands.
You will need to emerge it unless your liveCD provides it.

When wgetpaste is successful, it returns a URL for your paste.
You post the URL on the forum.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Tue Dec 31, 2013 12:44 pm    Post subject: Reply with quote

The puzzling thing was that I was able to boot the same kernel, same initramfs (generated by genkernel-next) if I did not have the real_init=/usr/lib/systemd/systemd kernel parameter.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Tue Dec 31, 2013 12:47 pm    Post subject: Reply with quote

OK. I pasted my .config to http://bpaste.net/show/163366/

Thanks!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Tue Dec 31, 2013 12:55 pm    Post subject: Reply with quote

hokiau,

While I look over your kernel, what does the script at /usr/lib/systemd/systemd contain?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Tue Dec 31, 2013 12:58 pm    Post subject: Reply with quote

It's a binary. I believe it's a package required by gnome 3.8. It was built from the sys-apps/systemd package.

http://wiki.gentoo.org/wiki/GNOME/3.8-upgrade-guide said I needed to install systemd when I upgraded to gnome 3.8.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 42569
Location: 56N 3W

PostPosted: Tue Dec 31, 2013 3:19 pm    Post subject: Reply with quote

hokiau,

I had forgotten that systemd is binary almost everything as I'm not a systemd user.

Your kernel .config looks good and as it boots without the
Code:
real_init=/usr/lib/systemd/systemd
then that binary is implicated in the problem.
Unfortunately, I have no idea how to unravel the binary to see what it does.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
nextgen
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jun 2004
Posts: 107
Location: Montreal, Quebec, Canada

PostPosted: Sun Jan 12, 2014 5:59 pm    Post subject: Reply with quote

This systemd upgrade nightmare has brought me an unprecedented amount of pain and suffering in 14 years of gentoo experience. I finally found a workaround.

The error
Code:
unable to mount ../dm-3 on /usr

is due to the fact that genkernel looks up the /usr device from /etc/fstab. In my case, the fstab entry pointed to /dev/sysraid/usr, which itself is a symlink to ../dm-3. I changed the fstab entry to explicitly point to /dev/dm-3 and regenerated the initramfs with genkernel. That took care of the above error when booting. (I wasn't quite done with systemd issues after that, but that's yet another story...)

In your case, you should probably point your fstab entry to /dev/dm-1.

I hope this helps.
Back to top
View user's profile Send private message
hokiau
n00b
n00b


Joined: 24 Dec 2013
Posts: 16

PostPosted: Mon Jan 13, 2014 12:40 am    Post subject: Reply with quote

Thanks, nextgen! It worked, after I took your advice and mount /dev/dm-1 on /usr. That should have been documented, shouldn't it?

Now I can proceed to the next step of upgrading to gnome 3.8. Thanks again to everybody who had helped!
Back to top
View user's profile Send private message
nextgen
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jun 2004
Posts: 107
Location: Montreal, Quebec, Canada

PostPosted: Mon Jan 13, 2014 3:43 am    Post subject: Reply with quote

Glad it worked!

I agree it should have been documented somewhere. It might have saved me days searching the web in vain until I accidentally observed the ../dm-3 symlink when browsing my LVM devices (/dev/<volume group>/*), which gave me the idea to adjust the fstab file.

Works with grub legacy.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2968
Location: Germany

PostPosted: Mon Jan 13, 2014 1:01 pm    Post subject: Reply with quote

note that this will break any time you change anything about your setup that makes the dm-x numbers change. They're dynamic (first come first serve).

You should really mount by UUID instead or find a device name that is actually available.
Back to top
View user's profile Send private message
nextgen
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jun 2004
Posts: 107
Location: Montreal, Quebec, Canada

PostPosted: Tue Jan 14, 2014 12:55 am    Post subject: Reply with quote

Can you give more detail on how to mount by UUID? Under /dev/disk/by-uuid, I still find relative-path symlinks (such as ../../dm-3). Hence I suppose that setting up those devices in /etc/fstab won't yield a bootable initramfs.
Back to top
View user's profile Send private message
damocles
n00b
n00b


Joined: 05 Feb 2010
Posts: 4

PostPosted: Tue Jan 14, 2014 8:31 pm    Post subject: Reply with quote

Hello,

I had this problem with genkernel-next and the culprit seems this code fragment in [src_initramfs]/etc/initrd.d/00-rootdev.sh

Code:
   

[...]
        # Resolve it like util-linux mount does
        [ -L "${dev}" ] && dev=$(readlink "${dev}")

[...]



as you can see it replaces you usr volume with the relative target device and by doing this mount gets confused and is unable to mount it.

Removing this line makes my system boot correctly but I wonder if there will be a better way...
Back to top
View user's profile Send private message
depontius
Advocate
Advocate


Joined: 05 May 2004
Posts: 3374

PostPosted: Tue Jan 14, 2014 8:55 pm    Post subject: Reply with quote

nextgen wrote:
Can you give more detail on how to mount by UUID? Under /dev/disk/by-uuid, I still find relative-path symlinks (such as ../../dm-3). Hence I suppose that setting up those devices in /etc/fstab won't yield a bootable initramfs.


Filesystems will have a UUID assigned when created. You can find them by using "blkid /dev/sdXN". I know I said it's assigned when the filesystem is created, but it's assigned to the device, not the filesystem. You can use "UUID="<big long string>"" in place of "/dev/sdXN" in /etc/fstab.

There is also a partUUID, but not for MSDOS partitions. If you partition your disks with GPT the partitions will be assigned a partUUID, even before any filesystem is created. I'm currently using this to assemble my RAID. I have a situation where one drive is eSATA, and of course happens to be /dev/sda. So when it comes and goes it upsets the entire disk naming convention. Conventional UUIDs work for the filesystems, but I'm using partUUID to get the RAID right. I'm under the impression that I don't need to, that RAID is smart enough to go searching and look for fingerprints, but I'm taking no chances. For RAID the drives are specified as "/dev/disk/by-partuuid/<designator>".
_________________
.sigs waste space and bandwidth
Back to top
View user's profile Send private message
nextgen
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jun 2004
Posts: 107
Location: Montreal, Quebec, Canada

PostPosted: Mon Feb 17, 2014 9:48 pm    Post subject: Reply with quote

depontius wrote:
Filesystems will have a UUID assigned when created. You can find them by using "blkid /dev/sdXN". I know I said it's assigned when the filesystem is created, but it's assigned to the device, not the filesystem. You can use "UUID="<big long string>"" in place of "/dev/sdXN" in /etc/fstab.

Thanks for the details. I've tried using UUID="<big long string>" in place of /dev/dm-3 in /etc/fstab. I then regenerated the initramfs using genkernel. I'm not sure what I might have forgotten, but this yielded an unbootable system: kernel panic, unable to mount UUID="440a1...". Fortunately, I had a good old openrc-based kernel, to which I've had to edit the kernel parameter root=/dev/md125 to make it bootable. This allowed me to boot into the old kernel, revert the /etc/fstab change and regenerate the initramfs. Now I can boot into systemd again.

However, the /dev/dm-3 workaround may not be stable as frostschutz notes.
frostschutz wrote:
note that this will break any time you change anything about your setup that makes the dm-x numbers change. They're dynamic (first come first serve).

Yet I'm stuck with my workaround. This will do for now. With this systemd painful adventure after 15 years of gentoo, I think that on my next system I'll be ready for a distro switch with a less complex setup.

Curiously, in my various attempts to boot trying to fix this, I've had a graphics card issue: no image on power up. As I initially connected this symptom to my systemd issues, I thought I had fried my system and that my adventure with gentoo had finally come to a abrupt end. I took out the graphics card, cleaned it with compressed air and could finally revive my system. I guess it was just a coincidence that the graphics card failed as I tried fixing my systemd setup. After all, I don't reboot very often.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Desktop Environments All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum