how to automatically restart a crashed init.d service

[Vieri]

Hi,

I'm wondering how others are doing this.

For some critical daemons, I use inittab with the respawn option.

I'd like to do something similar with my init scripts, and automatically restart them if their corresponding services crash (but not necessarily all of them).

What is the best approach?

method 1:
Use a custom cron job or inittab script to parse the output of:

Code: Select all

rc-status -c

Restart the list of services.

The inconvenience of this method is that the restart does not necessarily start right after a crash (depends on the cron periodicity setting).
Also, I am not sure every single service crash case is reflected as "crashed" in rc-status -c. Maybe some "crashes" are sometimes reported as "stopped" services? Finally, will "-c" report a crashed service in any runlevel, including the "manual runlevel"?

method 2:
Use a custom cron job or inittab script to parse the output of:

Code: Select all

rc-status -m -C | grep -v '\[  started  \]' | cut -f 2 -d ' '

Restart the list of services in the manual runlevel.

The inconvenience of this method is that I don't know how to tell rc-status to list only the services in the default runlevel. I might also auto-restart a service that was purposely stopped manually. As in method 1, the restart does not necessarily take place right after a crash.

method 3:
Use supervise-daemon.

The huge inconvenience of this method is that Gentoo wiki states that it is still unstable software, requires manually hacking the init.d files of the services you want to supervise, and make sure the service doesn't use its own PID file or forks.

Any ideas?

Despite its disadvantages, method 1 seems to be the easiest to implement and maintain for now.
The supervise-daemon looks interesting, but it's too tedious to set up and maintain.

[mike155]

Switch to Systemd. Use one of the restart options.

Code: Select all

[Service]
Restart=always
RestartSec=3

[toralf]

Does https://wiki.gentoo.org/wiki/OpenRC#Aut ... d_services help?

[Jaglover]

method 0:

Investigate why they crash and fix it.

I have one process which relies on an unreliable source and terminates sometimes as a result, I use daemontools to keep it running.

[Vieri]

Does https://wiki.gentoo.org/wiki/OpenRC#Aut ... d_services help?

Not quite. OpenRC does not seem to restart crashed services in "Dynamic Runlevel: manual". It will restart crashed services in "Runlevel: default".
Thanks anyway.

Moving to systemd is another option, but that would require a lot of work and testing.

[Vieri]

method 0:

Investigate why they crash and fix it.

Good one, and of course the preferred method. However, sometimes there's simply no time or resources to investigate. For instance, I had 1 crash of the tomcat service every 6 months, nothing worthwhile in the logs. That's the kind of service crash I don't mind simply restarting, but it has to be done right away.

[Vieri]

Does https://wiki.gentoo.org/wiki/OpenRC#Aut ... d_services help?

Not quite. OpenRC does not seem to restart crashed services in "Dynamic Runlevel: manual". It will restart crashed services in "Runlevel: default".

Let me rephrase that.

Code: Select all

openrc -n

will start a crashed service even if it's in the manual dynamic runlevel.
That's fine, but I need to run "openrc -n" if I want to avoid openrc stopping the services that are not in the default runlevel but in "Dynamic Runlevel: manual". That's a manual run of openrc -n.
I haven't found a config option in rc.conf equivalent to "-n". I guess that's why they came up with supervise-daemon.

[GDH-gentoo]

The easiest alternatives, compared to changing the init system or installing additional software, are inittab respawn lines and supervise-daemon, indeed.

inittab respawn lines are the reliable alternative, at the cost of having almost no service management (services start all at the same time when a runlevel is entered, are killed when the runlevel is left, and pretty much that's it).

If you are using OpenRC, supervise-daemon comes for free, at the cost of having imperfect supervision (nobody supervises the supervisors).

method 1:
Use a custom cron job or inittab script to parse the output of:

Code: Select all

rc-status -c

[...]
The inconvenience of this method is that the restart does not necessarily start right after a crash (depends on the cron periodicity setting)
[...]
method 2:
Use a custom cron job or inittab script to parse the output of:

Code: Select all

rc-status -m -C | grep -v '\[  started  \]' | cut -f 2 -d ' '

[...]
As in method 1, the restart does not necessarily take place right after a crash.

Exactly. That is polling. See "notification vs polling"

method 3:
Use supervise-daemon.

The huge inconvenience of this method is that Gentoo wiki states that it is still unstable software,

New OpenRC features are usually labeled "experimental", and seemingly stay experimental forever. It's like the package never reaching version 1.0

requires manually hacking the init.d files of the services you want to supervise,

If the service script does not have a start() and a stop() function, i.e. it relies implicitly on start-stop-daemon, the supervisor=supervise-daemon assignment to turn on supervision for the service, and the assignment to command_args_foreground, if needed, can be added to the corresponding /etc/conf.d file, which is meant to be modified (or created) by the administrator. And if the supervisor=supervise-daemon assignment is present, command_background and command_args_background are ignored in the service script, pidfile is used as the PID file of the supervise-daemon process, and start_stop_daemon_args is interpreted as command line options for supservise-daemon instead (or ignored if supervise_daemon_args is set). So directly modifying the service script might not be needed. But if you have to, what would be the problem?

and make sure the service doesn't use its own PID file or forks.

If this can be configured with command line arguments, command_args_foreground in the /etc/conf.d file can be used to do that.

[Vieri]

So directly modifying the service script might not be needed. But if you have to, what would be the problem?

Maintainability.
I'd have to keep track of file changes between upgrades. I grant you it seldom happens with init scripts, but I'd need to check for that anyway.

Thanks for all the feedback.

[Ionen]

I'd have to keep track of file changes between upgrades. I grant you it seldom happens with init scripts, but I'd need to check for that anyway.

What I like to do in case like this is to use a user patch instead, so packages will install the pre-modified files (I notably have a few for openrc). Also nice to keep track of changes, if the patch breaks it likely mean I should check rather than use my old file as-is which may be incompatible.

[Vieri]

if the patch breaks it likely mean I should check

Thanks, Ionen.
I already do that for other parts of the system, so I was hoping not to have to do this for init scripts too...
Oh, well, openrc -n every couple of minutes isn't too bad if I don't get any major crashes during working hours.

[GDH-gentoo]

Maintainability.

I'd still have a look at the service script of the daemon you want to restart. Depending on what it does, maybe all you need to have it supervised is:
/etc/conf.d/$insert-service-name-here

Code: Select all

supervisor=supervise-daemon
command_args_foreground="some command-line options"

[apurkrt]

You might be interested in sys-process/daemontools

https://cr.yp.to/daemontools.html

https://cr.yp.to/daemontools/faq/create.html#why