Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Assistance Other Things Gentoo
  • Search

systemd-216:0/2 always boots to recovery / maintenance mode

Still need help with Gentoo, and your question doesn't fit in the above forums? Here is your last bastion of hope.
Post Reply
Advanced search
10 posts • Page 1 of 1
Author
Message
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

systemd-216:0/2 always boots to recovery / maintenance mode

  • Quote

Post by splurben » Sat Sep 13, 2014 7:24 am

Howdy y'all,

Well, I migrated my bench testing system to systemd about a month ago. A few weeks back a BTRFS RAID element failed and I replaced it. Ever since then, and after a full backup and restore of the entire RAID, I still always boot into 'maintenance mode' (press Control-D or give root password).

I've issued:

Code: Select all

sudo systemctl default
and it hasn't changed anything.

I've been poring over my logs and I just don't see anything that explains it. The boot messages continue to scroll very quickly after the initial 'press Control-D or give root password' so I can never see which messages immediately precede the message.

The system boots fully into 'graphical' target after Control-D.

I really gave it a shot but I'd love it if someone would look over these messages and see if they can identify my problem. It sucks because I can't reboot the system remotely anymore.

emerge --info - http://pastebin.com/6LUPsw8q

grub2 cfg - http://pastebin.com/rrT8BV2q

kernel 3.15.10 / genkernel-next config - http://pastebin.com/KbsnpXjq

Let me know if I've excluded anything one might need to troubleshoot this,

Kirk

output of journalctl -xb - http://pastebin.com/DQL2bGfs
--=>Like... Goodness had nothing to do with it.<=--
Top
eccerr0r
Watchman
Watchman
Posts: 10239
Joined: Thu Jul 01, 2004 6:51 pm
Location: almost Mile High in the USA
Contact:
Contact eccerr0r
Website

  • Quote

Post by eccerr0r » Sat Sep 13, 2014 1:46 pm

I'm just giving this a shot, don't know enough about systemd yet, but learning:

It says modemmanager failed, I'm not sure if this is sufficient to cause it to fail startup.

What do you mean you can't reboot remotely anymore? Does running reboot fail or something?
Intel Core i7 2700K/Radeon Firepro W2100/24GB DDR3/800GB SSD
What am I supposed watching?
Top
jburns
Veteran
Veteran
Posts: 1255
Joined: Thu Jan 18, 2007 4:44 am
Location: Massachusetts USA

  • Quote

Post by jburns » Sat Sep 13, 2014 9:24 pm

  • Sep 13 14:47:48 bench6 systemd[1]: Stopping File System Check on /dev/disk/by-label/bench8raid...
    -- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
    --
    -- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.

Does it do a File System Check on every boot and why is systemd stopping it. Later in the log it starts the File System Check and it finishes.
Top
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

  • Quote

Post by splurben » Sun Sep 14, 2014 12:32 am

eccerr0r wrote:I'm just giving this a shot, don't know enough about systemd yet, but learning:

It says modemmanager failed, I'm not sure if this is sufficient to cause it to fail startup.

What do you mean you can't reboot remotely anymore? Does running reboot fail or something?
Well, I'm not using modemmanager so I'll look into either removing or disabling that, most likely in NetworkManager.

I can't reboot remotely because after it reboots it waits for 'Control-D or Give Root Password' instead of continuing to boot.
jburns wrote: Sep 13 14:47:48 bench6 systemd[1]: Stopping File System Check on /dev/disk/by-label/bench8raid...
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li ... temd-devel
--
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.



Does it do a File System Check on every boot and why is systemd stopping it. Later in the log it starts the File System Check and it finishes.
I reckon if fsck (btrfsck I suppose) is failing through from boot to shutdown (i.e. never gets marked completed) it may leave a 'marker' to boot into maintenance mode. I have already tried booting single user and running

Code: Select all

 ># btrfsck --repair LABEL="bench8raid"
which completes successfully. But that doesn't mean that systemd has acknowledged that fact.

I have noticed that systemd does seem to initialise a filesystem integrity scheme using fsck in some manner for all filesystems of all types. When I first started with systemd it always reported that fsck failed on BTRFS filesystems, presumedly because the invocation wasn't BTRFS aware, which I believe it is now.

I will post my findings after some research. Thanks guys.
--=>Like... Goodness had nothing to do with it.<=--
Top
eccerr0r
Watchman
Watchman
Posts: 10239
Joined: Thu Jul 01, 2004 6:51 pm
Location: almost Mile High in the USA
Contact:
Contact eccerr0r
Website

  • Quote

Post by eccerr0r » Sun Sep 14, 2014 1:25 am

Are all of the partitions in your /etc/fstab clean and working properly?

Currently I'd think more towards some critical filesystem to stop the boot process however though it's worth to check everything.

If you just type "systemctl --failed" do you see anything other than modemmanager?
Intel Core i7 2700K/Radeon Firepro W2100/24GB DDR3/800GB SSD
What am I supposed watching?
Top
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

ongoing info on this problem

  • Quote

Post by splurben » Sun Sep 14, 2014 3:46 am

ModemManager is installed, has a systemd unit, and the unit was disabled.

I have enabled the unit so we'll see if that gets rid of that error. The ModemManager is a required dependency of Gnome 3.

I am looking at this fsck set of messages:

Code: Select all

~ $ sudo journalctl -xb | grep -i fsck
Sep 14 09:28:48 bench6 systemd[1]: Starting system-systemd\x2dfsck.slice.
Sep 14 09:28:48 bench6 systemd[1]: Created slice system-systemd\x2dfsck.slice.
-- Subject: Unit systemd-fsck-root.service has finished start-up
-- Unit systemd-fsck-root.service has finished starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has begun starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1692]: /sbin/fsck.btrfs: BTRFS file system.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1693]: benchboot8: clean, 325/80320 files, 38628/321268 blocks
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has finished starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished shutting down
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished shutting down.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1791]: /sbin/fsck.btrfs: BTRFS file system.
It looks like btrfs.fsck is being invoked by systemd twice, and is left on? Can anyone confirm this from these logs? On pastebin.

But I have also found this:

Code: Select all

Sep 14 09:28:48 bench6 mount[1708]: mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
Sep 14 09:28:48 bench6 mount[1708]: missing codepage or helper program, or other error
Sep 14 09:28:48 bench6 mount[1708]: In some cases useful info is found in syslog - try
Sep 14 09:28:48 bench6 mount[1708]: dmesg | tail or so.
Sep 14 09:28:48 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
Sep 14 09:28:48 bench6 systemd[1]: Failed to mount /mnt/backup.
-- Subject: Unit mnt-backup.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mnt-backup.mount has failed.
-- 
This is the mount point /mnt/backup for the bench8raid which is mounted after the system boots where I have to type CONTROL-D.

If its mounting correctly after it's filesystem check, why is it failing here and apparently triggering 'maintenance mode'? Here's the output of blkid and the mount line from fstab.

Code: Select all

# blkid | grep bench8raid
/dev/sdb1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="7fc2c41e-ec19-4cee-8d3c-73964dd1c954" TYPE="btrfs" PARTLABEL="bench8_sdb1" PARTUUID="b5178a75-fa17-40c5-bd14-a881cef5a86e" 
/dev/sdc1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="2399e27b-1f2f-4b99-bdc2-6acf3ebb9220" TYPE="btrfs" PARTLABEL="bench8_sdc1" PARTUUID="f29c9f08-064d-43e8-8745-63d495cd4a8a" 
/dev/sdd1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="8220677b-8389-4633-96f8-ca14800fd2a8" TYPE="btrfs" PARTLABEL="bench8_sdd1" PARTUUID="b3c37ae9-be1e-4e7d-870a-70e8073a919f" 
/dev/sde1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="0230d173-c1ab-45b5-9097-f8aeb99344f0" TYPE="btrfs" PARTLABEL="bench8_sde1" PARTUUID="3a7a1b14-66e4-4c94-b4f6-f5b3e1aec0dd"

Code: Select all

# grep bench8raid /etc/fstab
LABEL="bench8raid"	/mnt/backup		btrfs		defaults,noatime,compress=zlib		1 2
I seriously wonder what's going on here.
I dropped to single user and used fstab based instructions to unmount and remount this array, and it works fine.

Code: Select all

# unmount /mnt/backup
# mount /mnt/backup
with no errors.

Does anyone know if there are more than one default units that attempt to mount fstab filesystems? It's doing it twice, failing the first time, and (I'm guessing) triggering 'maintenance mode' because of the first failure.

Code: Select all

-- Subject: Unit mnt-backup.mount has begun with start-up
-- Unit mnt-backup.mount has begun starting up.
Sep 14 09:28:48 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
-- Subject: Unit mnt-backup.mount has failed
-- Unit mnt-backup.mount has failed.
Sep 14 09:28:48 bench6 systemd[1]: Unit mnt-backup.mount entered failed state.
-- Subject: Unit mnt-backup.mount has begun with start-up
-- Unit mnt-backup.mount has begun starting up.
-- Subject: Unit mnt-backup.mount has finished start-up
-- Unit mnt-backup.mount has finished starting up.
I guess I'll try commenting out /mnt/backup in my fstab and see if that changes this scenario. I'm also searching for multiple units ending in '.mount'.

Well there's smoke, but that damned old lady is still standing on her balcony threatening to jump:

Code: Select all

# systemctl list-units | grep .mount
  proc-sys-fs-binfmt_misc.automount            loaded active waiting   Arbitrary Executable File Formats File System Automount Point
  -.mount                                      loaded active mounted   /
  boot.mount                                   loaded active mounted   /boot
  dev-mqueue.mount                             loaded active mounted   POSIX Message Queue File System
  mnt-backup.mount                             loaded active mounted   /mnt/backup
  run-user-1000.mount                          loaded active mounted   /run/user/1000
  sys-fs-fuse-connections.mount                loaded active mounted   FUSE Control File System
  sys-kernel-config.mount                      loaded active mounted   Configuration File System
  sys-kernel-debug.mount                       loaded active mounted   Debug File System
  tmp.mount                                    loaded active mounted   Temporary Directory
  var-lib-nfs-rpc_pipefs.mount                 loaded active mounted   RPC Pipe File System
  var-tmp-portage.mount                        loaded active mounted   /var/tmp/portage
  systemd-remount-fs.service                   loaded active exited    Remount Root and Kernel File Systems
--=>Like... Goodness had nothing to do with it.<=--
Top
poncho
Tux's lil' helper
Tux's lil' helper
Posts: 92
Joined: Sun Mar 06, 2011 11:21 am

  • Quote

Post by poncho » Sun Sep 14, 2014 12:03 pm

try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.
Top
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

  • Quote

Post by splurben » Mon Sep 15, 2014 9:18 am

poncho wrote:try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.
Cheers! Trying that now...
--=>Like... Goodness had nothing to do with it.<=--
Top
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

  • Quote

Post by splurben » Tue Sep 16, 2014 12:44 am

splurben wrote:
poncho wrote:try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.
Cheers! Trying that now...
Changed fsck flag in fstab to zero and still get double-mount fail/success on boot.

Still can't find a reason for systemd trying to mount, failing, and then successfully mounting the BTRFS RAID. FSCK and SCRUB find no problems on the RAID.

Since this happened after the previous RAID failed, I reckon it's a holdover, but I relabelled and rebuilt the raid so it doesn't have the same UUIDs or labels.
--=>Like... Goodness had nothing to do with it.<=--
Top
splurben
Apprentice
Apprentice
User avatar
Posts: 197
Joined: Tue Feb 03, 2004 10:56 am
Location: Augusta, Southwest Western Australia

still happening

  • Quote

Post by splurben » Thu Oct 09, 2014 1:50 am

the error is still happening, I'm finding new stuff in the journal which is still pointing to a strange problem with the BTRFS RAID.

The RAID is healthy, and is mounted when I'm fully booted, but I always have to type CTRL-D to finish the boot. These are the errors:

Code: Select all

-- The start-up result is done.
Oct 07 16:59:31 bench6 systemd-udevd[1700]: starting version 216
Oct 07 16:59:31 bench6 kernel: BTRFS: device label bench8raid devid 3 transid 3849 /dev/sdd1
Oct 07 16:59:31 bench6 kernel: BTRFS info (device sdd1): disk space caching is enabled
Oct 07 16:59:31 bench6 kernel: BTRFS: failed to read the system array on sdd1
Oct 07 16:59:31 bench6 systemd[1]: Started udev Kernel Device Manager.
-- Subject: Unit systemd-udevd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit systemd-udevd.service has finished starting up.
-- 
-- The start-up result is done.
Oct 07 16:59:31 bench6 mount[1703]: mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
Oct 07 16:59:31 bench6 mount[1703]: missing codepage or helper program, or other error
Oct 07 16:59:31 bench6 mount[1703]: In some cases useful info is found in syslog - try
Oct 07 16:59:31 bench6 mount[1703]: dmesg | tail or so.
Oct 07 16:59:31 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
Oct 07 16:59:31 bench6 systemd[1]: Failed to mount /mnt/backup.
-- Subject: Unit mnt-backup.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mnt-backup.mount has failed.
-- 
-- The result is failed.
Oct 07 16:59:31 bench6 systemd[1]: Dependency failed for Local File Systems.
-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit local-fs.target has failed.
-- 
-- The result is dependency.
Oct 07 16:59:31 bench6 systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Oct 07 16:59:31 bench6 systemd[1]: Unit mnt-backup.mount entered failed state.
Oct 07 16:59:31 bench6 systemd[1]: Closed CUPS Printing Service Sockets.
-- Subject: Unit cups.socket has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--=>Like... Goodness had nothing to do with it.<=--
Top
Post Reply

10 posts • Page 1 of 1

Return to “Other Things Gentoo”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy

 

 

magic