| View previous topic :: View next topic |
| Author |
Message |
steveL
Veteran
Joined: 13 Sep 2006
Posts: 1709
Location: The Peanut Gallery
|
 Posted: Sun Nov 25, 2012 10:29 pm Post subject: localmount in openrc-0.11.5 |
 |
|
I only noticed this after a couple of days on the new openrc, but /etc/init.d/localmount now unconditionally skips unmounting /usr in openrc-0.11.5 and later. The line:
| Code: | | no_umounts_r="$no_umounts_r|/proc|/proc/.*|/run|/sys|/sys/.*" |
was changed to:
| Code: | | no_umounts_r="$no_umounts_r|/proc|/proc/.*|/run|/sys|/sys/.*|/usr" |
This showed as an lvm error here, since it did not want to deactivate an active volume.
I've modified the patches, so initramfs is now a variable in /etc/rc.conf so we can change what localmount does when it stops. As usual, if unset or the default, none of these patches do anything.
However I would like to draw your attention to a couple of things I found, while I was digging around to see what had files open in /usr. It simply never occurred to me that localmount was no longer unmounting /usr, so I didn't even look at that part of rc.log til the following didn't fix it. Nevertheless I think what I changed here was useful, and might be useful to you, especially if you find that there are processes still open in /usr which are causing an issue when shutting down or restarting your machine.
The first thing I found which concerned me, is that agetty has /usr/lib64/locale/locale-archive open. I do remember Frysinger mentioning on the dev ML that a /lib/locale directory might be needed. I think as an example of how /usr is getting polluted and it's going to be more and more difficult to keep things on rootfs, but it doesn't really bother me if I need to tweak a few things. As there's only that one file in there, I had no compunction about doing the following.
NB I did this in a console login, after I had run /etc/init.d/xdm stop.
| Code: | mkdir /lib64/locale
cp -p /usr/lib64/locale/locale-archive /lib64/locale
sync; sync
cmp /lib64/locale/locale-archive /usr/lib64/locale/locale-archive || echo Oops
unlink /usr/lib64/locale/locale-archive
|
(The cmp line just checks to make sure the file is the same.)
However rmdir /usr/lib64/locale fails since there's a hidden .keep_sys-libs_glibc-2.2 file (that's the version here) in the directory. The following got that taken care of, then I could link the /usr directory to the rootfs one:
| Code: | cp -p /usr/lib64/locale/.keep_* /lib64/locale && rm /usr/lib64/locale/.keep_*
rmdir /usr/lib64/locale
ln -s /lib64/locale /usr/lib64/locale
|
Note that agetty is started by PID 1, and its linkage is very tight (see ldd /sbin/agetty): it only links to libc.so.6 (so only needs that, the dynamic loader and the linux vdso gate). Obviously with a libc directory on /usr that goes out of the window; I'm not interested in patching ebuilds unless I really have to, especially if a simple symlink will suffice.
This works thanks to the decades-old Unix tradition of only truly deleting a file once the last descriptor is closed, and allowing unlink to alter the directory hierarchy in the meantime. So processes with the existing file open continue to have its data available. (This is the same reason upgrades of our desktops don't bring down the running machine.)
However there is another glibc directory, /usr/lib64/gconv which holds shared libs for character conversion. This causes bash to have /usr/lib64/gconv/gconv-modules.cache open while it's running, which is bad if /bin/sh is a symlink to /bin/bash (the default on Linux): you end up with runscript having that file open at shutdown time. Again, ldd /bin/bash shows that there's no linkage going on outside /lib64. Clearly the conversion .so must be dlopen'ed by libc from that path, so again, a simple symlink on the directory, and 6.5 MB of rootfs space, means we can get our root dependency back.
| Code: | mkdir /lib64/gconv
cp -p /usr/lib64/gconv/* /lib64/gconv
sync; sync |
Now a check to make sure files have been copied correctly:
| Code: | | for f in /lib64/gconv/*; do cmp "$f" "/usr$f" || echo "$f"; done |
If that produces any output, there's an issue with the filename/s mentioned.
| Code: | rm -f /usr/lib64/gconv/*
rmdir /usr/lib64/gconv
ln -s /lib64/gconv /usr/lib64/gconv |
The rmdir had no issue here, since there were no hidden dotfiles. Obviously if it doesn't work, check what files are in there, with ls -A.
With the above, I no longer have any files in /usr open at shutdown time.
Note that there are lib32 variants of the above directories, but I don't consider those an issue since they're not used by system processes as part of boot or shutdown. In fact there isn't even a .cache file in /usr/lib32/gconv here, so it's not been used yet. On a 64-bit machine, any such apps are very unlikely to run as part of system init, but you should be aware of the possibility. (And if you're on a 32-bit install, you'll only have lib to worry about.)
If you think the above is painful, I'd love to be told what I'm missing, or to see a simple patch for glibc that could be put into /etc/portage. I just wanted to get my machine working right: I actually thought the problem was because of me switching to a *-kit free desktop so spent a lot longer on checking out exactly what files were open, when in fact it was the change in localmount shutdown.
However, by the time I got round to sorting out the localmount initscript, I had already done the above, so I'm writing it up here in case someone using these patches finds that they have processes mysteriously open in /usr at shutdown. As ever, comments, feedback, your experiences, and especially improvements most welcome.
Note also that the second part should not be an issue if you change /bin/sh to point to /bin/bb (which should also give you performance improvements across the board, including in system startup times.) I couldn't as yet see how to do that with runscript without changing the symlink, and I didn't get too far with reading the code. Perhaps setting SHELL=/bin/bb in /etc/rc.conf will work, not sure as yet: I wanted to have the problem solved for people using bash. |
|
| Back to top |
|
 |
steveL
Veteran
Joined: 13 Sep 2006
Posts: 1709
Location: The Peanut Gallery
|
| Posted: Sun Nov 25, 2012 10:40 pm Post subject: |
|
|
In case you're wondering, I used the following lines added to /etc/init.d/lvm to tell me what files were open at shutdown:
| Code: | stop() {
if yesno "${rc_lvm_debug:-NO}"; then
local f=/etc/log/lvm_lsof
ebegin "Listing open files to: $f"
lsof >"$f" 2>&1
eend $?
sleep 3
fi
run_addon lvm-stop
} |
Originally the command was: lsof /usr but then I found it useful to see everything.
This actually got me to:
| Code: | cp -p /usr/bin/lsof /bin/lsof
cmp /usr/bin/lsof /bin/lsof || echo oops
rm /usr/bin/lsof
ln -s /bin/lsof /usr/bin/lsof |
..as well, once I realised that lsof was on /usr and ldd showed me it didn't have to be. No doubt it'll get overwritten on next upgrade, but it was only for while I ensured that there was no issue with the file being open.
Note that I'm using /etc/log so that I can monitor early boot and late shutdown, when /var might be unmounted: I use this more for | Code: | | rc_log_path="/etc/log/rc.log" |
in /etc/rc.conf for when I turn logging on with rc_logger="YES". To set this up I added the following to /etc/logrotate.conf:
| Code: | # system-specific logs may be also be configured here.
# early-boot or late in shutdown
/etc/log/* {
olddir /etc/log/old
size 64k
create 0640 root wheel
}
|
(see man logrotate.)
and ran:
| Code: | mkdir -p /etc/log/old
chgrp -R wheel /etc/log
chmod -R g+w /etc/log
|
|
|
| Back to top |
|
|
ryao
Developer
Joined: 27 Feb 2012
Posts: 96
|
| Posted: Wed Nov 28, 2012 1:20 pm Post subject: |
|
|
| Please file a bug about this issue for the OpenRC team. |
|
| Back to top |
|
|
ryao
Developer
Joined: 27 Feb 2012
Posts: 96
|
| Posted: Thu Nov 29, 2012 4:05 pm Post subject: |
|
|
steveL, I talked to the OpenRC developers about this. The following patch should fix your issue:
| Code: | diff --git a/init.d/localmount.in b/init.d/localmount.in
index 8e67c38..ab0b1b5 100644
--- a/init.d/localmount.in
+++ b/init.d/localmount.in
@@ -22,6 +22,9 @@ start()
if [ "$RC_UNAME" = Linux ]; then
no_netdev="-O no_netdev"
+ if [ mountinfo -q /usr ]; then
+ touch $rc_svcdir/usr_premounted
+ fi
fi
ebegin "Mounting local filesystems"
mount -at "$types" $no_netdev
@@ -48,7 +51,10 @@ stop()
done
if [ "$RC_UNAME" = Linux ]; then
- no_umounts_r="$no_umounts_r|/proc|/proc/.*|/run|/sys|/sys/.*|/usr"
+ no_umounts_r="$no_umounts_r|/proc|/proc/.*|/run|/sys|/sys/.*"
+ if [ -e $rc_svcdir/usr_premounted ]; then
+ no_umounts_r="$no_umounts_r|/usr"
+ fi
fi
no_umounts_r="^($no_umounts_r)$"
|
It will likely be merged to OpenRC head, but you can apply it locally. It would have been helpful had you filed a bug report. |
|
| Back to top |
|
|
steveL
Veteran
Joined: 13 Sep 2006
Posts: 1709
Location: The Peanut Gallery
|
| Posted: Thu Nov 29, 2012 5:44 pm Post subject: |
|
|
Ah nice one ryao: I was wondering about picking up that /usr was mounted at startup, since it's an obvious fix. Good to know that I can
shove crap in "$rc_svcdir" for the future ;) You really should quote that expansion btw: touch "$rc_svcdir"/usr_premounted at minimum (though I usually quote each parameter separately so: touch "$rc_svcdir/usr_premounted". I'm not interested in hearing how it won't ever have spaces in yadda yadda.)
As for the bug report, sorry, but I wanted my machine booting cleanly again, and like I said I was actually switching to a *-kit free KDE, so tracked the problem down to the root cause as I wanted to know what was up. And I did bring it immediately to your attention on IRC, since it will affect forked udevs too ;)
As it is, I'm happy that I've switched the locale and gconv directories to rootfs: as you can see both are needed in normal operation (for agetty and bash for a start) so personally I do not want them in /usr at all come what may. Next step is to move pci and hw dbs to rootfs; I'd actually prefer it if Gentoo kept up with their old ebuild rather than use the udev parts, unless somehow systemd are taking over maintenance of both databases and their web interfaces etc?
Still, thanks for the fix to localmount: I'll add it to front-post when I'm more awake, so we can go back to keeping initramfs in udev.conf alone. |
|
| Back to top |
|
|
ryao
Developer
Joined: 27 Feb 2012
Posts: 96
|
| Posted: Thu Nov 29, 2012 8:31 pm Post subject: |
|
|
steveL, WilliamH wrote the fix. I just posted it for convenience. Anyway, it looks to me like we could merge your modifications into OpenRC if we moved this check into its own script in the sysinit runlevel and made them depend on it.
By the way, with regard to your comment about udev upstream requiring an initramfs to mount /usr, we have a udev fork in development that has the goal of restoring support for this, among other things:
http://www.gentoo.org/proj/en/eudev/index.xml
Support for a separate /usr is currently broken (because we forked off systemd 195), but we plan to restore it before our first release. I would encourage you to try it once we have it in the main tree. |
|
| Back to top |
|
|
steveL
Veteran
Joined: 13 Sep 2006
Posts: 1709
Location: The Peanut Gallery
|
| Posted: Sat Dec 01, 2012 6:32 pm Post subject: |
|
|
| ryao wrote: | | Anyway, it looks to me like we could merge your modifications into OpenRC if we moved this check into its own script in the sysinit runlevel and made them depend on it. |
Thanks, that sounds encouraging.
It took me a while to see what you were getting at, but you're right: if /usr is pre-mounted at startup, we don't need to start udev after localmount, and if it isn't then we do, and additionally localmount should umount it at shutdown. The thing is, we still need to configure udev via a variable to change its need and depend settings, and it has to be in boot, not sysinit, which while we can check at runtime is consistent, we still need the user to change it so that openrc can do the dependency calculation correctly, afaict.
If we had dynamic dependencies (and I don't know that we don't, or that they're not coming, I've just never heard of them) then udev would check $RC_RUNLEVEL, and if usr_premounted is unset, and it is starting in sysinit, tell openrc to start it in boot instead. In either case, the runlevel would determines its need and provide settings. So if we could delay its start from sysinit to boot when /usr is not premounted, the whole thing would be automatic.
But udev-mount has to provide dev when udev is starting late, and I don't think that decision should be based on whether /usr is pre-mounted or not, but on whether the admin has configured it to start late, since they know that the kernel and builtin module device nodes are sufficient. Better to complain noisily if things are inconsistent, which is only an issue for people not using an initramfs wrt /usr.
So as it is, we still need to move udev's runlevel, and we still need the variable so that udev-mount and udev can tell openrc the correct dependencies. And even if we didn't I don't think anyone would be happy with patches that made udev-mount provide dev without some sort of configuration opt-in.
Given that, it made sense to use the same variable for the localmount decision, since it's much lighter-weight (no filesystem access: I did consider checking whether /usr was mounted when localmount started, but already had the variable.) However that's not robust for localmount in all configurations: it should umount what it mounted, and not assume that it never mounted /usr.
I will make the current patches more robust by checking run-level for consistency with initramfs, and udev-mount can need sysfs when the variable is set, so that the user only has to move udev, and set the variable, once the scripts are patched. Thanks for the discussion so far, then, it's already leading to improvements :)
Moving the check to something in sysinit, then, isn't needed for these patches in the current state of things. But having that info would be useful for checking consistency, and might come in useful elsewhere. Until it does though, I don't think it merits a new service: why not just do it as part of openrc startup, and provide a runscript/environment variable, not a file? That way once it does become needed elsewhere, it's already in place and efficient, and in the meantime we can use it for localmount, and udev warnings. Or not, up to you. I see it as a small amount of code for something that is always going to be needed by localmount, and will be useful elsewhere, but until it's required elsewhere, localmount can continue as above.
Anyone else who reads this and wonders what other variables are around, check out man runscript. I realise some of you will think that's obvious, but it took me ages to find that for some reason.
| Quote: | By the way, with regard to your comment about udev upstream requiring an initramfs to mount /usr, we have a udev fork in development that has the goal of restoring support for this, among other things.
Support for a separate /usr is currently broken (because we forked off systemd 195), but we plan to restore it before our first release. I would encourage you to try it once we have it in the main tree. |
Cool, I'll try it out on another machine or a VM when you've put a working release out.
Until it replaces upstream udev in Gentoo, though, there's going to be a need for these patches, so my desktop won't switch for a while. Also, I have to say I'm very conservative about what runs on my machine, even more so with system stuff. I'm much happier delaying udev startup to after localmount, since I know I set my machine up to boot without needing udev to mount drives (nor indeed for network) than changing udev for another piece of software.
Having said that, I'm getting rid of as much software from that development team as I can. So I look forward to seeing what you come out with. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Unsupported Software
