Postmortem: world update resulted in a lockout (pam issue)

[nmel]

This issue is resolved already, however I wanted to share learnings, in case somebody faces the same problem, as well as have a discussion on how to avoid it in the future.

What happened: After reviewing a "emerge -avuUD @world" output, which looked fine, I let it run and went away for a hour. When I came back I could not login by any means (UI, terminal, ssh) and from the cooler sounds I understood that system is not under load, so the update is likely interrupted. I was locked out without any hope for "auto-resolution".

The root cause: My update contained 646 packages. #158 was pam (pam-1.3.1_p20200128-r1 -> pam-1.5.1), which removed some modules as stated in the news, and it went through. Package #243 has failed to build (unrelated to pam) and the update was interrupted. The pambase package upgrade (pambase-20200304 -> pambase-20201103) that supposed to update the configs was scheduled to be #462. Thus all logins have failed with the messages

Code: Select all

login[16993]: PAM unable to dlopen(/lib64/security/pam_tally2.so): /lib64/security/pam_tally2.so: cannot open shared object file: No such file or directory
login[16993]: PAM adding faulty module: /lib64/security/pam_tally2.so
login[16993]: PAM unable to dlopen(/lib64/security/pam_cracklib.so): /lib64/security/pam_cracklib.so: cannot open shared object file: No such file or directory
login[16993]: PAM adding faulty module: /lib64/security/pam_cracklib.so
login[16993]: FAILED LOGIN (1) on '/dev/tty1' FOR '***', Module is unknown

Mitigation: Without ability to log in, I have to boot from LiveUSB and chroot into the system. I found the cause and finished update under the chroot. I coudn't simply emerge pambase due to python conflicts, so I had to wait for 200+ packages to merge. I could have modified pam configs directly for a quick fix, however I was reluctant to do this as it might break other things.

How to avoid: I don't think it's fully possible at this point. Pam and pambase should go together, ideally in one package, otherwise the lock out in cases like this is imminent. I found other threads with the same problem (1, 2) and I somewhat understand maintenance challenges, however I think it should be designed to avoid failure as much as possible.

What's your opinion? How would you avoid this problem in the future?

[eccerr0r]

Yeah looks like lots of people have ran into this issue. I ran into this twice mostly due to other packages failing the emerge that would have eventually gotten pambase merged but didn't because things like spidermonkey failing (it may well have been!). If only pambase had fewer dependencies to build...

[Hu]

If I knew this was coming, I would manually reconfigure PAM to drop the use of the to-be-removed modules before starting the emerge @world. If you're asking how the Gentoo maintainers could have avoided this, that is harder. My first answer was to solve this with RDEPEND blocks, but I'm not sure if that would work since you need not only to install the updated configuration package, but also to have the user run etc-update or equivalent, and complete both those steps before merging the update that deletes the modules. An RDEPEND block could probably force Portage to merge the configuration package first, but if the user doesn't run etc-update, then the problem will still apply.

[nmel]

In my case configs merged automatically in pam.d as I haven't modified any of those files. However, in general case, you're right, simple RDEPEND won't help.

What about the pre-compile checks at the setup stage? Current config file could be checked for potential problems and fail with instructions describing how to fix it (similar to kernel option checks). This will require some config parsing logic integrated into the ebuild though. It's not smooth but at least it prevents a lockout.

I also just thought about a setup with an intermediate pam and pambase updates. The v1.4.0 pam release haven't removed the modules yet, just disabled building them by default. If we turn these modules on and make pambase-20201103 (or whatever version changed the configs) depend on pam-1.4*, while pam-1.5* depend on =pambase-20201103. I guess without pam slots it will introduce circular deps, right?

[pjp]

If I knew this was coming, I would manually reconfigure PAM to drop the use of the to-be-removed modules before starting the emerge @world. If you're asking how the Gentoo maintainers could have avoided this, that is harder. My first answer was to solve this with RDEPEND blocks, but I'm not sure if that would work since you need not only to install the updated configuration package, but also to have the user run etc-update or equivalent, and complete both those steps before merging the update that deletes the modules. An RDEPEND block could probably force Portage to merge the configuration package first, but if the user doesn't run etc-update, then the problem will still apply.

This almost sounds like an instance where manual upgrading should occur as part of the news announcement. That is, either upgrade pam & pambase at the same time with nothing else, or update everything else first, and only then update pam and pambase? Or am I misunderstanding the underlying problem?

[eccerr0r]

Because it depends on both pam and pambase building, I would hope that there's some way that will "prove" that pambase will succeed - as in, all of its dependencies will build - before pam builds. This really sounds like a classical NP-hard computer problem...

Again I'm really worried that spidermonkey was a dependency:

pambase - elogind - polkit - spidermonkey ... and spidermonkey failed for me.

Really spidermonkey needs to be removed from the dependency list somehow. Not saying that it's easy because it's an upstream decision but the fact that this is so far deep in the chain really hurts. Note that pam itself does not require polkit, so it tends to be moved up on the build order, again because lots of stuff depends on it.

[Hu]

pjp: I believe that is an accurate summary. The failure conditions require:

• Certain PAM modules are not installed. Recent versions of the relevant package follow upstream's default of not building or installing these modules.
• PAM configuration files refer to the missing modules. The current version of Gentoo's PAM configuration package has this fixed, but this thread exists because there are straightforward ways for users not to get the change soon enough.

There are several ways to get here:

• Upgrade the package that provides those modules to a version that does not build them. Suffer a build failure before the configuration file package is updated.
• Upgrade both, but fail to run etc-update or equivalent. According to the most recent post from nmel, users with no local modifications to the relevant file might get an automatic merge if they manage to install both packages, which would save them from ever needing manual intervention and could explain how the problem was not caught earlier.

I think it is not possible to guarantee that the configuration package will merge successfully, because even if you cover all the software faults, there remains the possibility that the system crashes due to power loss or the user cancels out before Portage finishes with the configuration package. On the Portage side, it is possible, with sufficiently paranoid checks, to refuse to install the new PAM module package on a system that would break without the corresponding configuration change. I think it would also be possible, as someone alluded to earlier in this thread, to have a transition phase that encourages the user along the right path. However, even with encouragement, guaranteeing a safe path would still require Portage to have a hard block to kill the build if it reaches the point of building toward a bad configuration because the user ignored the required configuration updates.

[eccerr0r]

The warning and actions requested in eselect news are insufficient - you actually need to take action immediately after pam upgrade completes to ensure you don't get locked out. I think this needs to be reflected in the news as the default config links in cracklib and tally so everyone could potentially be affected.

I would agree that it seems to be that the best course of action is that the prechecks for pam will check for later illegal configuration, alas, this will require the user to do this twice because the new config is not legal for the old pam and thus a possibly insecure compromise solution needs to be placed. Then once pam completes, the new config can be used.

Really seems no way out for even the common case where users did not modify their pam config to insure against lockouts if pambase could not build in time for the next login. Those who modified their configs are even worse off, but at least they know about these files and should be familiar on how to fix them.

(I've been locked out twice already on two machines. One due to an induced power outage because I could not login, other due to build failure. The build failure I at least had a login session still open so I was able to fix it without being local, the other machine I was not logged in and had no way to check status or do repair. And neither machine had custom pam configuration.)

[hujuice]

I didn't read everything, so forgive me if I repeat.
I had the same, after an interrupted @world update.
I resolved starting with a live and continuing the @world update in chroot.

Regards,
HUjuice

[eccerr0r]

viewtopic-t-1092476-start-0.html

Dangit they should make it top priority to make polkit use duktape. Core authentication systems should not be using a firefox component, even if it's built with rust.

[eccerr0r]

Not sure if there was a stealth change or not, but this next machine I updated, pam got updated *after* rust and spidermonkey.

Code: Select all

2021-01-22T01:43:34 >>> dev-lang/spidermonkey-78.6.0: 40â²18â³
2021-01-22T02:30:25 >>> dev-util/cbindgen-0.15.0: 19â²16â³
2021-01-22T02:49:41 >>> sys-libs/pam-1.5.1: 4â²27â³
2021-01-22T02:54:08 >>> sys-auth/elogind-243.7: 3â²24â³
2021-01-22T02:57:33 >>> sys-auth/pambase-20201103: 28s

This is much better, but I don't know what changed in portage, if anything. The window of lockout would have been 8 minutes, which is better.

BTW this machine had problems getting spidermonkey to build so it would have had problems as well if pam built early and pambase did not get installed due to spidermonkey failure.

[curmudgeon]

How would you avoid this problem in the future?

Code: Select all

# USE=-pam emerge -e world

Actually, it appears slightly safer to emerge shadow with USE=-pam (and restart openssh after configuring it not to use pam) before proceeding with the above.

Seriously, unless a system requires unusual authentication methods (such as smartcards), it doesn't need pam (meaning for the vast majority of home systems, pam does little more than provide additional layers of complexity). I have seen and read about people getting locked out of a system because of a pam mis-configuration or change of configuration for at least 25 years. I have never used pam (and the option to avoid it ranks high on my list of reasons why I use gentoo).

[eccerr0r]

Note that elogind session management requires pam, so if one says "regular" authentication doesn't need pam, then they also need to start using su/sudo/suid mount to access their usb sticks (versus automount/seamless desktop integration) and other esoteric chown/chmod to access console devices.

But of course curmudgeons don't do this, it's all newfangled conveniences that we should lay off before they scream to get off their lawns :)

[pa4wdh]

What's your opinion? How would you avoid this problem in the future?

There already was a news item about the pam update, and i think it should have contained more warnings/information about the update. Maybe a procedure to exclude pam/pambase from the update first, and have a separate run to update those with minimal time in between them.

An other way would be if pam could be (temporarily) configured to not fail if a configured module does not exist, because that's was causes the lockout in the first place. Such option would likely cause security issues when enabled permanently.

[eccerr0r]

The scary thing is that this is not consistent.

I'm 3 for 4 so far, three machines of the four that I've updated recently that either did have/would have/preemptively avoided (by sacrificing security) lockout problems, but one only had a small window that is "acceptable" for the "eselect news read" warning.

I still have at least 3 more machines to update.

What's alarming is that the 17.1 update had more warnings about it than the potential for losing control of your own box due to pam. This is not right...

[Tony0945]

But of course curmudgeons don't do this, it's all newfangled conveniences that we should lay off before they scream to get off their lawns

GET OFF MY LAWN!


Seriously, I'm very glad I ditched pam last last year and never installed elogind.
I'll accept the C word label. And hold no grudge, eccerr0r.

[eccerr0r]

When I first heard of pam I thought it was a good thing. Or at least I was hoping everything would go through pam and not DIY authentication as I've had programs do their own thing that were not compatible with each other, making changes in one program would completely break another. When PAM came along and everyone used PAM, the problem went away ... until this happened.

This is the first PAM related incident, though extremely inconvenient, that I've ever had that I can recall. In the old days without it I do recall all sorts of random programs stopped authenticating properly when another program changed their algorithm.

(BTW for those in favor of ignoring pam modules if they stop working - that's a security issue. Imagine a hacker having the ability to corrupt one byte in any file, while hitting /etc/shadow might not produce easy results if they don't know the initial contents, hitting a pam module making it no longer recognizable as ELF could disable a whole bunch of security checks.)

But yes I've gotten lazy and having to go switch windows to mount and umount removable media has always been annoying - though this is more related to having seats than specifically to pam.

[curmudgeon]

Note that elogind session management requires pam, so if one says "regular" authentication doesn't need pam, then they also need to start using su/sudo/suid mount to access their usb sticks (versus automount/seamless desktop integration) and other esoteric chown/chmod to access console devices.

Of course, elogind represents a fairly recent development (no doubt someone attempting to justify the "need" for ordinary users to have pam :) ).

You have significantly misstated the mounting issue. The udisks (polkit) option (which I used to use) has existed for years. That never required pam in the past (though I can't attest that it still does not). Autofs (which I use now) will allow ordinary users to mount USB devices without "su/sudo/suid mount."

I still maintain that (in the vast majority of cases) pam does nothing but add unnecessary complexity. So many other examples of that in Linux exist now.

I will offer as the first example one of my favorite targets for this criticism - grub. Again, I have no issue with people that actually reqiure the provided functionality, but I virtually always want to boot the latest kernel with no unusual options.

Take a look at the at what grub-mkconfig spits out (with no changes to the default).

Code: Select all

# grub-mkconfig 
Generating grub configuration file ...
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if loadfont unicode ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
done

That looks like complete insanity to me (how many people use video_bochs?).

My actual (yes, grub 2) /boot/grub/grub.cfg:

Code: Select all

set default=0
set timeout=30
set gfxpayload=1280x1024x8
set root=(hd0,5)

menuentry 'Gentoo Linux 5.4.80-r1' {
    linux /bzImage-5.4.80-r1-gentoo root=/dev/sda6 fbcon=rotate:3
}

menuentry 'Gentoo Linux 5.4.66' {
    linux /bzImage-5.4.66-gentoo root=/dev/sda6 fbcon=rotate:3
}

I can do even better than that on my EFI machines - I don't need grub at all. :) Once again, however, the suggested guidelines for setting up the EFI partition again add unnecessary complexity. I can use efibootmgr to select the kernel to boot, and thus can omit all of the EFI directory structure nonsense. I have a small (just 32MB) EFI partition, which contains only the following files:

Code: Select all

$ ls -al /boot/
total 23656
drwxr-xr-x 2 root root    8192 1970-01-01 00:00:00 ./
drwxr-xr-x 1 root root     134 2020-12-07 08:33:28 ../
-rwxr-xr-x 1 root root 3499839 2020-10-17 05:43:28 System.map-5.4.66-gentoo*
-rwxr-xr-x 1 root root 3528805 2020-12-04 23:33:10 System.map-5.4.80-r1-gentoo*
-rwxr-xr-x 1 root root 8442528 2020-10-17 05:43:38 bzImage-5.4.66-gentoo*
-rwxr-xr-x 1 root root 8499872 2020-12-04 23:33:20 bzImage-5.4.80-r1-gentoo*
-rwxr-xr-x 1 root root  109154 2020-10-17 05:22:10 config-5.4.66-gentoo*
-rwxr-xr-x 1 root root  109334 2020-12-04 23:24:28 config-5.4.80-r1-gentoo*

[eccerr0r]

Actually that is also severely lacking in details: elogind (and systemd) uses pam to handle sessions. Pam is the best place to deal with this data as any login mechanism should go through the same code sequence to handle all login mechanisms. Now udisks (using polkit) can use the data pam hands down to it and know whether a particular login should have access to the media. I don't know if people recall a "broken" consolekit setup where consolekit requires ck-launch-session. Pam now deals with creating the session for you when you login, you now get your elogind/systemd-logind without having to launch a session.

Getting off topic again, the key reason consolekit/systemd/elogind is that a console user should have access to USB devices. However a user logging in remotely through xdm or vnc however should not have access to the USB devices.

As for grub-mkconfig, it was meant to generate very elaborate grub configuration files. The main purpose of that is to design a menu system that has submenus. Notice you can select kernels and also select kernels with different options automated by grub-mkconfig. As you found out you can make a much simpler grub2 config file, but it does not have the fancy menu features that grub-mkconfig provides.

Whether you want the menus or not, that's another question. I've slowly warmed up to grub-mkconfig because it's a one-command automated grub detection system that I don't have to worry about making a typo when vi /boot/grub/grub.conf and accidentally typing the filename of a kernel incorrectly, also resulting in an unbootable system.

And EFI? EFI firmware is SO much more complicated than BIOS firmware yet still frequently omits key functionality. My main reason for not using EFI direct boot is mainly dependent on what's in the EFI firmware. Most EFI firmware omit the CLI, efi-shell. Without efi-shell built into firmware, I can't on-the-fly submit kernel commandline parameters, and because of this grub(grub2) remains my go-to bootmanager.

[pjp]

But of course curmudgeons don't do this, it's all newfangled conveniences that we should lay off before they scream to get off their lawns :)

This is the first PAM related incident, though extremely inconvenient, that I've ever had that I can recall.

That second quote seems to explain the point of view expressed in the first comment.

[eccerr0r]

Seems completely disjoint to me as these were in response to different issues.

Don't get me wrong, pam does add complexity. It can break - and did. It caused an inconvenience indeed. But when working it's much more of a convenience.

Uhoh car analogy: do we stop driving cars because a car broke down due to wear and tear? Then again in this case someone tried to repair the car and it caused a different problem, should we stop driving the car because of that? Go back to walking, yes!

Granted if you just needed to walk 10 feet, no sense for the car to begin with... which is also an acceptable solution, just didn't need to get that far I suppose.

[pjp]

Would you fly in a plane designed comparably to PAM? I wouldn't.

I do however put up with PAM. But its design is security problem in and of itself.

[eccerr0r]

Once again we're talking about two questions here:

1. Would we fly planes that are treated like how Gentoo developers dealt with PAM?

Well the answer, unless one is stupid, would obviously be no. But this isn't necessarily the question at hand.

What if was worded like:

2. Would we fly planes whose software architecture was like how pam is written?

Well I'm sorry, a lot of today's software nowadays is like this, with so much complexity to handle all possible solutions. You may well not fly on any airplanes if you only want software written in the 1960s.

Tough call here. I think we're screwed and have to put up with the software, thousands of people have flown with MCAS and it didn't fail, but it did fail... We put up with it to save a few MPG...

[Irre]

I have this line

"c1respawn:/sbin/mingetty --autologin root tty1 linux"

in /etc/inittab

and is automatically logged in as root. I don't know if it works with PAM.

[Tony0945]

Would you fly in a plane designed comparably to PAM? I wouldn't.

Boeing 737MAX. Written by $9 an hour "software engineers". Or maybe high school kids. Not to mention removing hardware redundancy (of sensors!) .
The place to be cheap is not flight controls. The place in Linux to be cavalier is not login software.