Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
systemd-216:0/2 always boots to recovery / maintenance mode
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sat Sep 13, 2014 7:24 am    Post subject: systemd-216:0/2 always boots to recovery / maintenance mode Reply with quote

Howdy y'all,

Well, I migrated my bench testing system to systemd about a month ago. A few weeks back a BTRFS RAID element failed and I replaced it. Ever since then, and after a full backup and restore of the entire RAID, I still always boot into 'maintenance mode' (press Control-D or give root password).

I've issued:
Code:
sudo systemctl default

and it hasn't changed anything.

I've been poring over my logs and I just don't see anything that explains it. The boot messages continue to scroll very quickly after the initial 'press Control-D or give root password' so I can never see which messages immediately precede the message.

The system boots fully into 'graphical' target after Control-D.

I really gave it a shot but I'd love it if someone would look over these messages and see if they can identify my problem. It sucks because I can't reboot the system remotely anymore.

emerge --info - http://pastebin.com/6LUPsw8q

grub2 cfg - http://pastebin.com/rrT8BV2q

kernel 3.15.10 / genkernel-next config - http://pastebin.com/KbsnpXjq

Let me know if I've excluded anything one might need to troubleshoot this,

Kirk

output of journalctl -xb - http://pastebin.com/DQL2bGfs
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Sat Sep 13, 2014 1:46 pm    Post subject: Reply with quote

I'm just giving this a shot, don't know enough about systemd yet, but learning:

It says modemmanager failed, I'm not sure if this is sufficient to cause it to fail startup.

What do you mean you can't reboot remotely anymore? Does running reboot fail or something?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
jburns
Veteran
Veteran


Joined: 18 Jan 2007
Posts: 1213
Location: Massachusetts USA

PostPosted: Sat Sep 13, 2014 9:24 pm    Post subject: Reply with quote

    Sep 13 14:47:48 bench6 systemd[1]: Stopping File System Check on /dev/disk/by-label/bench8raid...
    -- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.


Does it do a File System Check on every boot and why is systemd stopping it. Later in the log it starts the File System Check and it finishes.
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sun Sep 14, 2014 12:32 am    Post subject: Reply with quote

eccerr0r wrote:
I'm just giving this a shot, don't know enough about systemd yet, but learning:

It says modemmanager failed, I'm not sure if this is sufficient to cause it to fail startup.

What do you mean you can't reboot remotely anymore? Does running reboot fail or something?


Well, I'm not using modemmanager so I'll look into either removing or disabling that, most likely in NetworkManager.

I can't reboot remotely because after it reboots it waits for 'Control-D or Give Root Password' instead of continuing to boot.

jburns wrote:
Sep 13 14:47:48 bench6 systemd[1]: Stopping File System Check on /dev/disk/by-label/bench8raid...
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.



Does it do a File System Check on every boot and why is systemd stopping it. Later in the log it starts the File System Check and it finishes.


I reckon if fsck (btrfsck I suppose) is failing through from boot to shutdown (i.e. never gets marked completed) it may leave a 'marker' to boot into maintenance mode. I have already tried booting single user and running
Code:
 ># btrfsck --repair LABEL="bench8raid"
which completes successfully. But that doesn't mean that systemd has acknowledged that fact.

I have noticed that systemd does seem to initialise a filesystem integrity scheme using fsck in some manner for all filesystems of all types. When I first started with systemd it always reported that fsck failed on BTRFS filesystems, presumedly because the invocation wasn't BTRFS aware, which I believe it is now.

I will post my findings after some research. Thanks guys.
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Sun Sep 14, 2014 1:25 am    Post subject: Reply with quote

Are all of the partitions in your /etc/fstab clean and working properly?

Currently I'd think more towards some critical filesystem to stop the boot process however though it's worth to check everything.

If you just type "systemctl --failed" do you see anything other than modemmanager?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Sun Sep 14, 2014 3:46 am    Post subject: ongoing info on this problem Reply with quote

ModemManager is installed, has a systemd unit, and the unit was disabled.

I have enabled the unit so we'll see if that gets rid of that error. The ModemManager is a required dependency of Gnome 3.

I am looking at this fsck set of messages:
Code:
~ $ sudo journalctl -xb | grep -i fsck
Sep 14 09:28:48 bench6 systemd[1]: Starting system-systemd\x2dfsck.slice.
Sep 14 09:28:48 bench6 systemd[1]: Created slice system-systemd\x2dfsck.slice.
-- Subject: Unit systemd-fsck-root.service has finished start-up
-- Unit systemd-fsck-root.service has finished starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has begun starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1692]: /sbin/fsck.btrfs: BTRFS file system.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1693]: benchboot8: clean, 325/80320 files, 38628/321268 blocks
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-benchboot8.service has finished starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun shutting down.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished shutting down
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished shutting down.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun with start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has begun starting up.
-- Subject: Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished start-up
-- Unit systemd-fsck@dev-disk-by\x2dlabel-bench8raid.service has finished starting up.
Sep 14 09:28:48 bench6 systemd-fsck[1791]: /sbin/fsck.btrfs: BTRFS file system.


It looks like btrfs.fsck is being invoked by systemd twice, and is left on? Can anyone confirm this from these logs? On pastebin.

But I have also found this:
Code:
Sep 14 09:28:48 bench6 mount[1708]: mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
Sep 14 09:28:48 bench6 mount[1708]: missing codepage or helper program, or other error
Sep 14 09:28:48 bench6 mount[1708]: In some cases useful info is found in syslog - try
Sep 14 09:28:48 bench6 mount[1708]: dmesg | tail or so.
Sep 14 09:28:48 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
Sep 14 09:28:48 bench6 systemd[1]: Failed to mount /mnt/backup.
-- Subject: Unit mnt-backup.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mnt-backup.mount has failed.
--

This is the mount point /mnt/backup for the bench8raid which is mounted after the system boots where I have to type CONTROL-D.

If its mounting correctly after it's filesystem check, why is it failing here and apparently triggering 'maintenance mode'? Here's the output of blkid and the mount line from fstab.
Code:
# blkid | grep bench8raid
/dev/sdb1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="7fc2c41e-ec19-4cee-8d3c-73964dd1c954" TYPE="btrfs" PARTLABEL="bench8_sdb1" PARTUUID="b5178a75-fa17-40c5-bd14-a881cef5a86e"
/dev/sdc1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="2399e27b-1f2f-4b99-bdc2-6acf3ebb9220" TYPE="btrfs" PARTLABEL="bench8_sdc1" PARTUUID="f29c9f08-064d-43e8-8745-63d495cd4a8a"
/dev/sdd1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="8220677b-8389-4633-96f8-ca14800fd2a8" TYPE="btrfs" PARTLABEL="bench8_sdd1" PARTUUID="b3c37ae9-be1e-4e7d-870a-70e8073a919f"
/dev/sde1: LABEL="bench8raid" UUID="b65b2adf-8b40-457d-b59a-8d4c50d46967" UUID_SUB="0230d173-c1ab-45b5-9097-f8aeb99344f0" TYPE="btrfs" PARTLABEL="bench8_sde1" PARTUUID="3a7a1b14-66e4-4c94-b4f6-f5b3e1aec0dd"

Code:
# grep bench8raid /etc/fstab
LABEL="bench8raid"   /mnt/backup      btrfs      defaults,noatime,compress=zlib      1 2

I seriously wonder what's going on here.
I dropped to single user and used fstab based instructions to unmount and remount this array, and it works fine.
Code:
# unmount /mnt/backup
# mount /mnt/backup
with no errors.

Does anyone know if there are more than one default units that attempt to mount fstab filesystems? It's doing it twice, failing the first time, and (I'm guessing) triggering 'maintenance mode' because of the first failure.
Code:
-- Subject: Unit mnt-backup.mount has begun with start-up
-- Unit mnt-backup.mount has begun starting up.
Sep 14 09:28:48 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
-- Subject: Unit mnt-backup.mount has failed
-- Unit mnt-backup.mount has failed.
Sep 14 09:28:48 bench6 systemd[1]: Unit mnt-backup.mount entered failed state.
-- Subject: Unit mnt-backup.mount has begun with start-up
-- Unit mnt-backup.mount has begun starting up.
-- Subject: Unit mnt-backup.mount has finished start-up
-- Unit mnt-backup.mount has finished starting up.


I guess I'll try commenting out /mnt/backup in my fstab and see if that changes this scenario. I'm also searching for multiple units ending in '.mount'.

Well there's smoke, but that damned old lady is still standing on her balcony threatening to jump:
Code:
# systemctl list-units | grep .mount
  proc-sys-fs-binfmt_misc.automount            loaded active waiting   Arbitrary Executable File Formats File System Automount Point
  -.mount                                      loaded active mounted   /
  boot.mount                                   loaded active mounted   /boot
  dev-mqueue.mount                             loaded active mounted   POSIX Message Queue File System
  mnt-backup.mount                             loaded active mounted   /mnt/backup
  run-user-1000.mount                          loaded active mounted   /run/user/1000
  sys-fs-fuse-connections.mount                loaded active mounted   FUSE Control File System
  sys-kernel-config.mount                      loaded active mounted   Configuration File System
  sys-kernel-debug.mount                       loaded active mounted   Debug File System
  tmp.mount                                    loaded active mounted   Temporary Directory
  var-lib-nfs-rpc_pipefs.mount                 loaded active mounted   RPC Pipe File System
  var-tmp-portage.mount                        loaded active mounted   /var/tmp/portage
  systemd-remount-fs.service                   loaded active exited    Remount Root and Kernel File Systems

_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
poncho
Tux's lil' helper
Tux's lil' helper


Joined: 06 Mar 2011
Posts: 92

PostPosted: Sun Sep 14, 2014 12:03 pm    Post subject: Reply with quote

try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
Quote:
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Mon Sep 15, 2014 9:18 am    Post subject: Reply with quote

poncho wrote:
try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
Quote:
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.


Cheers! Trying that now...
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Tue Sep 16, 2014 12:44 am    Post subject: Reply with quote

splurben wrote:
poncho wrote:
try to disable fsck for your btrfs partitions.

from http://wiki.gentoo.org/wiki/Btrfs
Quote:
fstab's last number column should be 0 for all btrfs partitions. fsck.btrfs & btrfsck should not be run.


Cheers! Trying that now...


Changed fsck flag in fstab to zero and still get double-mount fail/success on boot.

Still can't find a reason for systemd trying to mount, failing, and then successfully mounting the BTRFS RAID. FSCK and SCRUB find no problems on the RAID.

Since this happened after the previous RAID failed, I reckon it's a holdover, but I relabelled and rebuilt the raid so it doesn't have the same UUIDs or labels.
_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
splurben
Apprentice
Apprentice


Joined: 03 Feb 2004
Posts: 197
Location: Augusta, Southwest Western Australia

PostPosted: Thu Oct 09, 2014 1:50 am    Post subject: still happening Reply with quote

the error is still happening, I'm finding new stuff in the journal which is still pointing to a strange problem with the BTRFS RAID.

The RAID is healthy, and is mounted when I'm fully booted, but I always have to type CTRL-D to finish the boot. These are the errors:

Code:
-- The start-up result is done.
Oct 07 16:59:31 bench6 systemd-udevd[1700]: starting version 216
Oct 07 16:59:31 bench6 kernel: BTRFS: device label bench8raid devid 3 transid 3849 /dev/sdd1
Oct 07 16:59:31 bench6 kernel: BTRFS info (device sdd1): disk space caching is enabled
Oct 07 16:59:31 bench6 kernel: BTRFS: failed to read the system array on sdd1
Oct 07 16:59:31 bench6 systemd[1]: Started udev Kernel Device Manager.
-- Subject: Unit systemd-udevd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit systemd-udevd.service has finished starting up.
--
-- The start-up result is done.
Oct 07 16:59:31 bench6 mount[1703]: mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
Oct 07 16:59:31 bench6 mount[1703]: missing codepage or helper program, or other error
Oct 07 16:59:31 bench6 mount[1703]: In some cases useful info is found in syslog - try
Oct 07 16:59:31 bench6 mount[1703]: dmesg | tail or so.
Oct 07 16:59:31 bench6 systemd[1]: mnt-backup.mount mount process exited, code=exited status=32
Oct 07 16:59:31 bench6 systemd[1]: Failed to mount /mnt/backup.
-- Subject: Unit mnt-backup.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mnt-backup.mount has failed.
--
-- The result is failed.
Oct 07 16:59:31 bench6 systemd[1]: Dependency failed for Local File Systems.
-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit local-fs.target has failed.
--
-- The result is dependency.
Oct 07 16:59:31 bench6 systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Oct 07 16:59:31 bench6 systemd[1]: Unit mnt-backup.mount entered failed state.
Oct 07 16:59:31 bench6 systemd[1]: Closed CUPS Printing Service Sockets.
-- Subject: Unit cups.socket has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

_________________
--=>Like... Goodness had nothing to do with it.<=--
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum