dropping to emergency shell upon boot

[EliasJonsson]

I have two Raspberry Pi 4 running Gentoo using SSDs connected to the Pi's USB ports. Both boxes are updated. Today I tried rebooting one of them. The box dropped to emergency shell.
Tried rebooting the other one. It dropped to an emergency shell. Something tells me Gentoo currently have some stability problem. Am I the only one having this issue?

[NeddySeagoon]

EliasJonsson,

I don't use an initrd, so I don't have an emergency shell.

You should have got an error about something failed, dropping you to a shell.
What was the error?

To save me asking later. please post the output of

Code: Select all

ls -l /boot

and the content of both cmdline.txt and config.txt

[EliasJonsson]

Thank you, NeddySeagoon, for your questions and your attention. However, I'm afraid its tool late for that. I switched to Ubuntu Server. Hopefully the slower and less optimized distribution will not come with nasty surprises as Gentoo have given me lately. Good bye Gentoo, it truly was a blast!

[EliasJonsson]

I thought that I would try booting the broken system again, and lo and behold, the system started without me intervening!
The joy was unfortunately short, because the error reappeared after the next boot, I landed in an emergency shell once again.

Everything seems to be well until 5.6s into the boot process. Does anyone have a clue on how to avoid the emergency mode and why emergency mode starts in the first place?

https://pastebin.com/mSSsVdqi

[EliasJonsson]

EliasJonsson,

I don't use an initrd, so I don't have an emergency shell.

You should have got an error about something failed, dropping you to a shell.
What was the error?

To save me asking later. please post the output of

Code: Select all

ls -l /boot

and the content of both cmdline.txt and config.txt

Here comes a ship loaded. Hope it is not too late

Code: Select all

ls -l /boot

https://pastebin.com/TUXegkRP

cmdline.txt:

Code: Select all

dwc_otg.lpm_enable=0 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

config.txt:

Code: Select all

# set 64 bit mode
arm_64bit=1

# have a properly sized image
disable_overscan=1
# for sound over HDMI
hdmi_drive=2
# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# lets have the VC4 hardware accelerated video
#dtoverlay=vc4-fkms-v3d
dtoverlay=vc4-kms-v3d-pi4.dtbo

gpu_mem=256

# for sound over HDMI
hdmi_drive=2

# Enable audio (loads snd_bcm2835)
dtparam=audio=on


#some overclocking of the RPi4
#over_voltage=6
#arm_freq=2000

#enables ffmpeg hardware accellerated video transcoding
dtoverlay=rpivid-v4l2

[NeddySeagoon]

EliasJonsson,

I know that config.txt. I wrote it. :)

You are using the default kernel8.img for 64 bit Pi without an initrd.
The initrd is not listed in /boot nor in config.txt.

cmdline.txt has all the signs of being good. In particular rootwait.

I don't understand where the 'emergency shell' comes from.

From you dmesg pastebin,

Code: Select all

[    5.285605] cfg80211: Loading compiled-in X.509 certificates for regulatory database

at about 5 sec, it stars going wrong.

Code: Select all

[    5.296056] cfg80211: Problem loading in-kernel X.509 certificate (-22)
...
[    5.303875] cfg80211: loaded regulatory.db is malformed or signature is missing/invalid

That collection of files is provided by linux-firmware.

systemd is not happy. Its not something I use, so I don't know how it handles errors.

Code: Select all

[    5.607628] vc4-drm gpu: [drm] The core clock cannot reach frequencies high enough to support 4k @ 60Hz.
[    5.607660] vc4-drm gpu: [drm] Please change your config.txt file to add hdmi_enable_4kp60.

40k 60Hz support is off by default. The GPU gets hot.

[pingtoo]

@neddy,

I think @EliasJosson is refer to systemd's Emergency.target. I first also think it is refer to initrd drop into shell but it is like you said, no inited involved.

@Eliasjosson,

I don't use systemd, so I am just using Internet search for an answer. May be you can try systemctl --failed in the emergency shell to gather more information on what happen

[EliasJonsson]

@neddy,
@Eliasjosson,
I don't use systemd, so I am just using Internet search for an answer. May be you can try systemctl --failed in the emergency shell to gather more information on what happen

I wish that I could. After an update I now lost visuals. There is no terminal to be seen. Display doesn't even get an HDMI signal from the Raspberry Pi, apart from the first ~3 seconds into the boot process. System does not respond on pinging.

Perhaps that is the last nail in the coffin. I will rebuild the system from the ground and up a day I get more time to do so.
It shall be interesting to see if the same systemd error will reappear on that build. Meanwhile; thank you for you much appreciated help in trying to straight this question mark out.

[EliasJonsson]

EliasJonsson,
I know that config.txt. I wrote it.

I am glad you did! It has been working terrifically well indeed!

EliasJonsson,

Code: Select all

[    5.607628] vc4-drm gpu: [drm] The core clock cannot reach frequencies high enough to support 4k @ 60Hz.
[    5.607660] vc4-drm gpu: [drm] Please change your config.txt file to add hdmi_enable_4kp60.

40k 60Hz support is off by default. The GPU gets hot.

The resolution was good enough, no 4k was needed in my opinion.

I will post back in this thread if the system drops to the emergency mode shell on new new build I will make when I have more time to do so.
Thank you, NeddySeagoon for you time and patience.

[EliasJonsson]

I solved the mystery with the black screen upon boot. All that was needed was to insert `hdmi_hotplug=1` in config.txt and the screen wouldn't go blank. No need to reinstall Gentoo, I hope.
However, I managed to boot once without first needing to go through systemd `emergency mode`, but that was only once.

I have updated both kernel and userland but without much success in terms of getting rid of emergency mode upon boot.

Here comes output from systemctl --failed:

Code: Select all

gen ~ # systemctl --failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                        
● klk_status_watcher.service    loaded failed failed closet light status tracker
● systemd-fsck-root.service     loaded failed failed File System Check on Root Device
● systemd-journal-flush.service loaded failed failed Flush Journal to Persistent Storage

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
3 loaded units listed.

[pingtoo]

Here comes output from systemctl --failed:

Code: Select all

gen ~ # systemctl --failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION                        
● klk_status_watcher.service    loaded failed failed closet light status tracker
● systemd-fsck-root.service     loaded failed failed File System Check on Root Device
● systemd-journal-flush.service loaded failed failed Flush Journal to Persistent Storage

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
3 loaded units listed.

I wonder once your system is in emergency mode, can you check if your root file system is mount read-only? i.e. at shell prompt type "mount<enter>" and check if your /dev/sda2 is mount on / and with "ro" option.

I recall your previously posted console output, the messages in there lead me thinking there are problem connect journal and seems to missing something that leads to many services failure.

Also from your posted console output, it seems you were able to recover system to a state it become functional, am I right about this? If I am right what did you do to recover?

[NeddySeagoon]

EliasJonsson,

How did you do an update?

Code: Select all

systemd-fsck-root.service     loaded failed failed File System Check on Root Device

If the root filesystem check fails, root stays read only.
That's game over its not possible to update from there.

Its possible that you updated the kernel in /boot, somehow else but were unable to add its modules to /lib/modules/ because the filesystem would not mount read/write.

The console driver on the Pi is built as a module, so no modules, no console.