Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] Some services don't always start automatically
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Sun Apr 27, 2014 9:42 pm    Post subject: [solved] Some services don't always start automatically Reply with quote

Is anyone else finding that some services which should start automatically at boot don't always do so? I just restarted my laptop and my LAN network interface wasn't up along with all of the services which depend on it such as shorewall. I did an 'rc' and it came right up followed by all dependencies. I have a remote system which I sometimes can't get into after rebooting and I suspect it's due to this issue.

Last edited by grant123 on Sat May 10, 2014 9:51 pm; edited 1 time in total
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Wed Apr 30, 2014 4:27 pm    Post subject: Reply with quote

What could cause this?
Back to top
View user's profile Send private message
Chiitoo
l33t
l33t


Joined: 28 Feb 2010
Posts: 983
Location: Here and Away Again

PostPosted: Wed Apr 30, 2014 4:46 pm    Post subject: ><)))°€ Reply with quote

Teegrins, grant123!


I don't really have any guesses to share (yet), but do you actually see them failing during boot?

If you boot straight into a login-manager, or you don't have --noclear set for the terminal in /etc/inittab, it may be difficult to see any of the failures. There is, however, the /var/log/rc.log which can be examined later on.

One needs to have something like this in /etc/rc.conf for the file to appear:

Code:
# rc_logger launches a logging daemon to log the entire rc process to
# /var/log/rc.log
# NOTE: Linux systems require the devfs service to be started before
# logging can take place and as such cannot log the sysinit runlevel.
rc_logger="YES"

I know this isn't much, but I do hope it helps!
_________________
Kind Regards,
~ The Noob Unlimited ~

Sore wa sore, kore wa kore.
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Thu May 01, 2014 3:01 pm    Post subject: Reply with quote

Thank you, I've just implemented that and I will report back with any findings.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 3636
Location: The Peanut Gallery

PostPosted: Thu May 01, 2014 4:23 pm    Post subject: Reply with quote

grant123 wrote:
What could cause this?

It sounds a lot like udev "persistently-changing" network names. Have you disabled those; if so, you know they changed the filename that disables it? If not, you might want to, unless you have multiple ethernet cards. The kernel parm works, but I currently have both /etc/udev/rules.d/80-net-name-slot.rules and /etc/udev/rules.d/80-net-setup-link.rules (I just hardlinked the latter to the former with ln, and haven't upgraded yet.)
Code:
$ cat /etc/udev/rules.d/80-net-setup-link.rules
# dummy empty file to keep eth* and wlan*

Don't do this unless you're sure, as it means you have to change your net setup scripts back to eth0, and I don't want to be blamed for your remote system. ;) I mention it in case you've already done this in 80-net-name-slot.rules, and as I think it's better, in general; your specific case might differ.
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Sat May 03, 2014 12:45 am    Post subject: Reply with quote

I have this from /var/log/rc.log:

* Bringing up interface enp0s20u2u1
* ERROR: interface enp0s20u2u1 does not exist
* Ensure that you have loaded the correct kernel module for your hardware
* ERROR: net.enp0s20u2u1 failed to start
* Bringing up interface enp0s20u2u2
* ERROR: interface enp0s20u2u2 does not exist
* Ensure that you have loaded the correct kernel module for your hardware
* ERROR: net.enp0s20u2u2 failed to start

Both interfaces are USB connected to my laptop and the drivers are built into the kernel. A subsequent reboot starts both interfaces fine. I checked /var/log/everything/current (metalog) but nothing jumps out at me except this later in the boot sequence:

[/etc/init.d/net.enp0s20u2u1] net.enp0s20u2u1: not allowed to be hotplugged

I'll try rc_hotplug="net.*" but that's not fixing the real problem.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Sat May 03, 2014 1:55 am    Post subject: Reply with quote

i suggest you check out the monit wiki. https://wiki.gentoo.org/wiki/Monit

also post (as root)

Code:

rc-update
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Sun May 04, 2014 3:36 am    Post subject: Reply with quote

rc_hotplug="net.*" seems to start the network interfaces reliably but I'd like to get to the root of the problem and disable hotplug. I found this in the logs:

systemd-udevd[1249]: renamed network interface eth0 to enp0s20u2u1

Could udev be renaming my interfaces after the initscripts are trying to start them? I have net.enp0s20u2u1 and net.enp0s20u2u2 in the default runlevel.
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Sun May 04, 2014 12:01 pm    Post subject: Reply with quote

grant123 wrote:
Could udev be renaming my interfaces after the initscripts are trying to start them? I have net.enp0s20u2u1 and net.enp0s20u2u2 in the default runlevel.

grant123 ... no, udev should be in the boot runlevel and so have re-named the devices prior to them being started. Note that if you re-plug the USB into another slot these "(non-)persistent names" will be changed.

What do you have set for 'rc_depend_strict' (/etc/rc.conf)? Also, rc_parallel="YES" isn't set?

BTW, is there a reason both interfaces are brought up ... are they providing separate subnets?

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Sun May 04, 2014 2:30 pm    Post subject: Reply with quote

Quote:
udev should be in the boot runlevel and so have re-named the devices prior to them being started


In this chunk from the logs, "net.enp0s20u2u1 would not start" comes before "renamed network interface eth0 to enp0s20u2u1":

Code:
[/etc/init.d/shorewall] ERROR: cannot start shorewall as net.enp0s20u2u1 would not start
[sshd] Server listening on 0.0.0.0 port 22.
[lightdm] PAM unable to dlopen(/lib64/security/pam_systemd.so): /lib64/security/pam_systemd.so: cannot open shared object file: No such file or directory
[lightdm] PAM adding faulty module: /lib64/security/pam_systemd.so
[kernel] [    4.139828] ax88179_178a 3-2.1:1.0 eth0: register 'ax88179_178a' at usb-0000:00:14.0-2.1, ASIX AX88179 USB 3.0 Gigabit Ethernet
[dbus] [system] Activating service name='org.freedesktop.Accounts' (using servicehelper)
[mtp-probe] checking bus 3, device 8: "/sys/devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1"_
[mtp-probe] bus: 3, device: 8 was not an MTP device_
[kernel] [    4.154662] systemd-udevd[1395]: renamed network interface eth0 to enp0s20u2u1


I have udev and udev-mount in the sysinit runlevel:

Code:
# rc-update -s
            alsasound | boot                         
             bootmisc | boot                         
                cupsd |      default                 
                 dbus |      default                 
                devfs |                       sysinit
        device-mapper | boot                         
                dmesg |                       sysinit
                 fsck | boot                         
             hostname | boot                         
              hwclock | boot                         
              keymaps | boot                         
            killprocs |              shutdown       
    kmod-static-nodes |                       sysinit
                local |      default                 
           localmount | boot                         
             loopback | boot                         
                  lvm | boot                         
              metalog |      default                 
              modules | boot                                       
             mount-ro |              shutdown       
                 mtab | boot                         
      net.enp0s20u2u1 |      default                 
      net.enp0s20u2u2 |      default                 
               net.lo | boot                         
             netmount |      default                 
                 ntpd |      default                 
               procfs | boot                         
                 root | boot                         
            savecache |              shutdown       
            shorewall |      default                 
                 sshd |      default                 
                 swap | boot                         
            swapfiles | boot                         
               sysctl | boot                         
                sysfs |                       sysinit
         termencoding | boot                         
         tmpfiles.dev |                       sysinit
       tmpfiles.setup | boot                         
                 udev |                       sysinit
           udev-mount |                       sysinit
              unbound |      default                 
              urandom | boot                         
           vixie-cron |      default                 
                  xdm |      default


Quote:
What do you have set for 'rc_depend_strict' (/etc/rc.conf)?


It's unset so it's default.

Quote:
Also, rc_parallel="YES" isn't set?


Unset there too.

Quote:
is there a reason both interfaces are brought up ... are they providing separate subnets?


enp0s20u2u1 is the WAN and enp0s20u2u2 is the LAN. The laptop has no built-in ethernet adapters.
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Sun May 04, 2014 7:05 pm    Post subject: Reply with quote

grant123 wrote:
Code:
[lightdm] PAM unable to dlopen(/lib64/security/pam_systemd.so): /lib64/security/pam_systemd.so: cannot open shared object file: No such file or directory
[lightdm] PAM adding faulty module: /lib64/security/pam_systemd.so

grant123 ... overlooking the fact that obviously this is getting messed up by systemd-udev not completing its tasks before the default runlevel is reached ... why on earth do you have components built with the systemd useflag? Or is this some new "udev needs it" shananigans.

Anyhow, I can't say why systemd-udev is screwing the process up as I don't use udev, perhaps the "aggressive parallelization" of systemd is now default for all components, or the above is first signs that "standalone udev" properly requires systemd for two network devices to be activated before the rest of the system is brought up. Well, as I don't support systemd (or systemd users) I might as well include systemd-udev in the list ... as the old adage goes "when it breaks you get to keep all the peices".

grant123 wrote:
Quote:
What do you have set for 'rc_depend_strict' (/etc/rc.conf)?

It's unset so it's default.

I believe that is rc_depend_strict="NO" (at least as I remember I changed this myself), anyhow, you should read the description of what this does (in /etc/rc.conf).

grant123 wrote:
Quote:
Also, rc_parallel="YES" isn't set?

Unset there too.

So, "NO" ... ok.

grant123 wrote:
Quote:
is there a reason both interfaces are brought up ... are they providing separate subnets?

enp0s20u2u1 is the WAN and enp0s20u2u2 is the LAN. The laptop has no built-in ethernet adapters.

Huh? This laptop is the gateway?

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Mon May 05, 2014 9:30 pm    Post subject: Reply with quote

Quote:
overlooking the fact that obviously this is getting messed up by systemd-udev not completing its tasks before the default runlevel is reached ... why on earth do you have components built with the systemd useflag? Or is this some new "udev needs it" shananigans

I actually don't have anything built with USE=systemd:

# USE=-systemd emerge -pvDuN world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
#

Quote:
Anyhow, I can't say why systemd-udev is screwing the process up as I don't use udev, perhaps the "aggressive parallelization" of systemd is now default for all components, or the above is first signs that "standalone udev" properly requires systemd for two network devices to be activated before the rest of the system is brought up. Well, as I don't support systemd (or systemd users) I might as well include systemd-udev in the list ... as the old adage goes "when it breaks you get to keep all the peices".

Should I try emerging world with USE="systemd"?

Quote:
I believe that is rc_depend_strict="NO" (at least as I remember I changed this myself), anyhow, you should read the description of what this does (in /etc/rc.conf).

Here's what I have in /etc/rc.conf:

# Do we allow any started service in the runlevel to satisfy the dependency
# or do we want all of them regardless of state? For example, if net.eth0
# and net.eth1 are in the default runlevel then with rc_depend_strict="NO"
# both will be started, but services that depend on 'net' will work if either
# one comes up. With rc_depend_strict="YES" we would require them both to
# come up.
#rc_depend_strict="YES"

So I think it defaults to YES. It sounds like this only matters if both interfaces should not come up, but I do want them both up.

It looks like rc_parallel defaults to NO.

Quote:
Huh? This laptop is the gateway?

Yes, it's connected to a modem/router which forwards all ports to the laptop.
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Tue May 06, 2014 12:12 am    Post subject: Reply with quote

grant123 wrote:
I actually don't have anything built with USE=systemd

grant123 ... ok, but then why is lightdm looking for /lib64/security/pam_systemd.so, that doesn't seem normal as it (or something) has to be linked to pam_systemd.so. Sorry, I wasn't suggesting you'd screwed up somehow, but having the useflag enabled seemed the most obvious reason for this.

grant123 wrote:
Should I try emerging world with USE="systemd"?

No, definitely not. Your best to try and debug udev and see why its taking so long. Oh, my memory ... ummm ... should be /etc/udev/udev.conf ... set the debug level and enable 'udev_monitor' and 'udev_debug' ... you should then get debug info in dmesg and syslog ... udev also logs somewhere but I can't remember where right now (perhaps under /var/log or /var/run/udev/).

grant123 wrote:
Quote:
Huh? This laptop is the gateway?

Yes, it's connected to a modem/router which forwards all ports to the laptop.

I see ... that'll eat up quite a lot of power, I'm sure if you went to a saturday market, boot sale, thrift store, or similar you'd find something that could replace it.

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Tue May 06, 2014 10:19 pm    Post subject: Reply with quote

Quote:
why is lightdm looking for /lib64/security/pam_systemd.so, that doesn't seem normal as it (or something) has to be linked to pam_systemd.so.

I think that's because of these lines in these default files:

/etc/pam.d/lightdm:
Code:
session   optional pam_systemd.so

/etc/pam.d/lightdm-greeter:
Code:
session   optional pam_systemd.so


I tried commenting them and rebooting and the errors do go away but the problem remains.

Quote:
Your best to try and debug udev and see why its taking so long. Oh, my memory ... ummm ... should be /etc/udev/udev.conf ... set the debug level and enable 'udev_monitor' and 'udev_debug' ... you should then get debug info in dmesg and syslog ... udev also logs somewhere but I can't remember where right now (perhaps under /var/log or /var/run/udev/)

I tried that but couldn't find anything enlightening. Do we know that udev is taking too long? At this point I don't get any messages in /var/log/everything/current about any services failing to start because the enp* interfaces failed to start, but I do still get this in rc.log:

Code:
 * Bringing up interface enp0s20u2u1
 *   ERROR: interface enp0s20u2u1 does not exist
 *   Ensure that you have loaded the correct kernel module for your hardware
 * ERROR: net.enp0s20u2u1 failed to start
 * Bringing up interface enp0s20u2u2
Cannot find device "enp0s20u2u2"
 *   ERROR: interface enp0s20u2u2 does not exist
 *   Ensure that you have loaded the correct kernel module for your hardware
 * ERROR: net.enp0s20u2u2 failed to start


Once the system is fully booted, 'rc' does nothing so I think hotplug is making sure everything comes up but I'd prefer the interfaces to start via their initscripts so I could disable hotplug.

Quote:
Huh? This laptop is the gateway?

Quote:
Yes, it's connected to a modem/router which forwards all ports to the laptop.

Quote:
I see ... that'll eat up quite a lot of power, I'm sure if you went to a saturday market, boot sale, thrift store, or similar you'd find something that could replace it.

It's a workstation too though. It does double-duty.
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 815

PostPosted: Sat May 10, 2014 9:52 pm    Post subject: Reply with quote

I needed to build xhci_hcd into the kernel instead of as a module. Thank you for all your help. You definitely helped me reach the solution.
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2950

PostPosted: Sun May 11, 2014 12:29 am    Post subject: Reply with quote

grant123 wrote:
I needed to build xhci_hcd into the kernel instead of as a module. Thank you for all your help. You definitely helped me reach the solution.

grant123 ... you're welcome. Occassionally a topic reply notification either doesn't reach me or I overlook it for some reason ... this is one such case, I hadn't noticed your reply until now. Anyhow, pleased to hear your issue was resolved ... I'd look into the reason for the pam_systemd.so is being defined though.

best ... khay
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum