Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
LVM: Systemd deletes /dev/mapper nodes
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
yarjie
n00b
n00b


Joined: 20 Dec 2013
Posts: 25

PostPosted: Sun Dec 29, 2013 1:50 am    Post subject: LVM: Systemd deletes /dev/mapper nodes Reply with quote

So far had a configuration of system with OpenRC on three lvm logical volumes. It booted via initramfs with init script activating lvm volumes and thus creating the nodes:
Code:

/dev/mapper/_vg-ROOT
/dev/mapper/_vg-home
/dev/mapper/_vg-tmp

(then to mount /, /home, /tmp to these devices).
And with OpenRC it worked very well.

Recently the Systemd replaced OpenRC, everything done according to tutorials, and now lvm doesn't work anymore.
Seems like after changing root in init script:
Code:
 
exec switch_root /mnt/root  /usr/lib/systemd/systemd

the all nodes from /dev/mapper/ disappear, so no mount of the logical volumes seems possible.

Why?
Back to top
View user's profile Send private message
Kompi
Apprentice
Apprentice


Joined: 05 Oct 2002
Posts: 252
Location: Germany

PostPosted: Sun Dec 29, 2013 12:53 pm    Post subject: Reply with quote

What kind of initrd are you using? Generated via genkernel, dracut or custom built?

If you auto-create it, don't use genkernel, switch to dracut with the "systemd" USE flag, as this will give you an initrd with systemd that can automatically hand over some info and udev stuff from the initrd to you main systemd process. I had a lot of trouble with genkernel+systemd.

If you custom built your initrd you may have to mount devtmpfs before creating the LVM nodes in /dev/mapper and probably need to start udevd. I'm not sure if that is enough. If not, you may try to add another unit that calls vgmknodes --refresh when systemd starts up, something like this:

Code:
[Unit]
Description=Linux Volume Manager (LVM)

After=systemd-udevd.service
Before=local-fs-pre.target

DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/lvm vgmknodes --refresh

[Install]
WantedBy=local-fs.target
Back to top
View user's profile Send private message
yarjie
n00b
n00b


Joined: 20 Dec 2013
Posts: 25

PostPosted: Tue Dec 31, 2013 12:47 am    Post subject: Reply with quote

Just for now answering your caring question: yes, custom built, even if something was generated automatically I would try to understand what came out and why, and then to tailor to my specific needs anyway...
So now going to follow your nice hints... and report afterwards.

Would you please give me a clue how in iniramfs init script code fragment looked like to mount devtmpfs before creating the LVM nodes in /dev/mapper and to start udevd? Probably first busybox needs some special care compilation options...

And how to name the unit you propose, as a service or targed?
Back to top
View user's profile Send private message
Kompi
Apprentice
Apprentice


Joined: 05 Oct 2002
Posts: 252
Location: Germany

PostPosted: Tue Dec 31, 2013 1:15 am    Post subject: Reply with quote

Mounting devtmpfs should be straight forward:

Code:
mount -t devtmpfs devtmpfs /dev


As with udev, I guess I would first try to start it the way it is started by systemd as well. In the /usr/lib/systemd/system/systemd-udevd.service file it is just executing the binary. So if you're lucky it will work just by:

Code:
/usr/lib/systemd/systemd-udevd &
(or the path you store that binary in)

I'm not sure though if that will start without systemd running or any other dependencies. Also there might be a problem with starting it in the initramfs and then systemd trying to start it again ... If it doesn't work, I would first try if it also works without udev in the initramfs, as vgmknodes --refresh may trigger all the udev events anyway. (if you try that, have a look at the --noudevsync or --sysinit options of vgchange).

If systemd complains at boot with something like "dev-mapper-xy.device timed out", then it most likely is a udev related problem, as udev needs to apply some tags to make the device available for systemd.


The unit I posted is a .service file. Something like "lvm-refresh.service" or similar should do.
Back to top
View user's profile Send private message
yarjie
n00b
n00b


Joined: 20 Dec 2013
Posts: 25

PostPosted: Tue Dec 31, 2013 6:04 am    Post subject: Reply with quote

At this point must confirm vgmknodes --refresh works, i.e. in the meaning of refreshing the nodes, but during the boot system complains about udev not possible to start.
After all, do you mean to start udev (as well as yet in such an early stage as from initramfs (having copied /usr/lib/systemd/systemd-udevd there)?
Back to top
View user's profile Send private message
Kompi
Apprentice
Apprentice


Joined: 05 Oct 2002
Posts: 252
Location: Germany

PostPosted: Tue Dec 31, 2013 7:55 am    Post subject: Reply with quote

I thought it might have been necessary to have udev running when calling lvm the first time (in the initramfs) in order to have the /dev/mapper/* nodes managed by udev correctly, so that udev can handle events generated by lvm. Because systemd uses the udev database to query devices and might not be able to use the /dev/mapper devices properly, even if they are there as device nodes. This would include copying the udev daemon to the initramfs, yes.

I thought so because the dracut initramfs does that, but when I was trying using a genkernel image that does not, I had issues with systemd (systemd would wait for the .device units until time out, even though the nodes are there).

But I'm not sure if that is really needed. Maybe it is enough to create them in the initramfs (using vgchange with --noudevsync) and then later call the vgmknodes --refresh when udev is actually running. That would at least be much easier to do - I would try whether this is doable first.

If you have to include the udevd binary, make sure it has all linked libraries in the initramfs as well. Get a list with:
Code:
ldd  /usr/lib/systemd/systemd-udevd

If that is not enough it's probably a good idea to create a dracut initramfs and have a look at how it is done there.
Back to top
View user's profile Send private message
yarjie
n00b
n00b


Joined: 20 Dec 2013
Posts: 25

PostPosted: Thu Jan 02, 2014 12:33 pm    Post subject: Reply with quote

Of course tried to make dracut's initramfs but there is no customizable init script but binary executable instead, so we can not see what init exactly does...


Yes, but what about the other "legacy" proc and sysfs usual init script lines?
Please take a look inside the following fragment a say what's wrong:

Code:

mount -t proc none /proc
mount -t sysfs none /sys

sh devmap_mknod.sh

mount -t devtmpfs none /dev
/usr/lib/systemd/systemd-udevd &
       
lvm vgscan
vgchange -a y

mount -o ro /dev/mapper/_vg-ROOT  /mnt/root

umount /proc /sys

exec switch_root $ROOTNEW /usr/lib/systemd/systemd



In this case there're boot complaints about udev not started but systemctl status systemd-udevd.service says "active"...
Back to top
View user's profile Send private message
Kompi
Apprentice
Apprentice


Joined: 05 Oct 2002
Posts: 252
Location: Germany

PostPosted: Thu Jan 02, 2014 1:09 pm    Post subject: Reply with quote

That's what I expected that could be a problem. Udev probably fails to start in systemd because it is already running (left over from the initrd).

I quickly searched the web and found the initramfs example of the LFS project here: http://www.linuxfromscratch.org/blfs/view/svn/postlfs/initramfs.html

There you can see in the script which files are copied in order to use udev. For example, they copy the udev config files from /etc/udev/rules.d and /etc/udev/udev.conf to the initramfs. Most certainly a good idea. They also call udevadm trigger and settle after udevd started up. This may be neccessary to get all block devices detected etc.

But most interestingly, they kill the initramfs udevd at the end before switch_root:
Code:
killall -w systemd-udevd

should do it. That should make it possible for a clean startup of the root's udevd via systemd later.

The mounting of sysfs and procfs is correct and should be included. Udevd will most certainly need them, maybe lvm as well.
Back to top
View user's profile Send private message
yarjie
n00b
n00b


Joined: 20 Dec 2013
Posts: 25

PostPosted: Thu Jan 02, 2014 1:21 pm    Post subject: ExecStart=/sbin/lvm vgmknodes --refresh Reply with quote

First must apologize, in this case I can't see udev failed to start on boot, as previously. (I have written this without testing, just remembering previous attempts)
-----------------------

Thank you very much for the link!
So, if such tailored new init script executes then the command:
Code:
killall -w /usr/lib/systemd/systemd-udevd

doesn't executes neatly and instead generates the error message:
Code:

 sh: cannot set terminal process group (-1); Inapropriate ioctl for device


However the more important seems the question whether we still need the unit with "ExecStart=/sbin/lvm vgmknodes --refresh"?

If this service not enabled the system can not mount lvm logical volumes despite this time their nodes sit inside /dev/mapper!

The boot sequence gets stuck on such messages:
Code:

 [ TIME ] Timed out waititng for device dev-mapper_vg\x2dtlvhone.device.       
 [DEPEND] Dependency failed for /home 
 [DEPEND]   Dependency failed for Local File Systems.


(Have to "mount -a" manually in a system where service with " vgmknodes --refresh" not enabled.)
Back to top
View user's profile Send private message
ClericLeech
n00b
n00b


Joined: 06 Jul 2006
Posts: 11

PostPosted: Tue May 20, 2014 11:47 am    Post subject: Reply with quote

I am currently facing a similar problem.

Custom initramfs does what is should do but systemd can't find the decrypted devices afterwards.
The workaround with a custom service brings no improvement. The device nodes are there but
they are missing from /dev/disk/by-uuid.

I managed to narrow the problem down to systemd-udevd not recognizing the decrypted devices.
So I tried to include systemd-udevd into the initramfs which works up until the point when I try to
decrypt my devices. Then I get the (in)famous "udev cookie waiting for zero" error.
So I included the lvm binarys and links to vgchange etc and all from /lib/udev/, /etc/udev, /etc/lvm.
Still the same problem.

I don't want to use dracut or genkernel, since I need a script to decrypt my gpg encrypted keys and pipe them to cryptsetup.
Since I have many encrypted volumes with different keys and I don't want to type my passphrase more than one time, so I cannot use dracut.

Any help would be appreciated.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Tue May 20, 2014 12:08 pm    Post subject: Reply with quote

If LVM device nodes are missing within the initramfs, there are some workarounds here: https://wiki.gentoo.org/wiki/Custom_Initramfs#LVM

If the device nodes are present in initramfs, but get lost once switching to the real init system - there is a bug in that system somewhere and it's doubtful whether you can fix it from the initramfs side of things. Of course if it's just LVM you can set the lvm.conf on the real system to handle device nodes itself instead of relying on udev (which is one of the workarounds listed in the link above).

Unfortunately I do not have any experience with systemd.

If even the UUID are missing, then something is really weird, because usually even when there are no /dev/mapper/... nodes, you can still fall back to /dev/dm-1, /dev/dm-2, etc.

Are the devices still listed in dmsetup ls, dmsetup table?
Back to top
View user's profile Send private message
ClericLeech
n00b
n00b


Joined: 06 Jul 2006
Posts: 11

PostPosted: Tue May 20, 2014 12:26 pm    Post subject: Reply with quote

The dm-0 to dm-x nodes are present. I can also do a udevadm trigger and the mapper nodes are there. A mount -a is possible and I can mount the filesystems.
So the system seems fine, since I can use it normally. Everything is working, but the automatic boot.
And always entering systemd rescue shell doesn't strike me as an optimal boot with gentoo. 8)

The encrypted devices are in /dev/disk/by-uuid. But not the decrypted ones. So fsck from systemd tries to check /dev/mapper/home for example.
But it can't find the device in /dev/disk/by-uuid and times out. That is the actual problem.
So the culprit is LVM.
Back to top
View user's profile Send private message
ClericLeech
n00b
n00b


Joined: 06 Jul 2006
Posts: 11

PostPosted: Wed May 21, 2014 7:02 pm    Post subject: Reply with quote

Okay.

Here is a little update:

I have successfully changed my custom initramfs so that it includes systemd-udevd.
The boot now works, but systemd still complains about dev-mapper-home.device timing out.
Afterwards it mounts /dev/mapper/home to /home without complains and the system boots up normally.

So this is not yet solved but if anyone faces a similar problem, feel free to contact me.

But now what did I do:
I copied systemd-udevd with all dependencies to the initramfs and launched it by "systemd-udevd --daemon --resolve-names=never" instead of mdev
after mounting devtmpfs to /dev.
After that decrypted my gpg encrypted keys and decrypted the devices. All show up under /dev/mapper/ but not as links to /dev/dm-x.
Before switching to the new root I killed systemd-udevd and mounted /dev to /newroot/dev via the move option.
In systemd I disabled lvm-etad and set lvm-etad = 0 in /etc/lvm.conf

Hope this helps anyone with a similar problem. I will file a bug about the obviously false behaviour of lvm2 in ~amd64.

An Admin or the original creator of the topic can now set the title to [SOLVED] please.

Thanks for all the help! Long live GENTOO!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum