Restarting services after updates

Rcomian · Last edited by Rcomian on Sat Feb 25, 2006 11:36 am; edited 2 times in total

You know the problem - after updating a bunch of packages you may need to restart some services in order to pick up any changes or security updates. But unless you were really paying attention, or the service being updated was a key service, it can be hard to find out what's going on.
This also goes if your trying to have a server applying security updates whilst being un-attended - it would need to know when to restart a service.

There's lots of issues involved with this, but the script presented here is a first stab at actually providing at least a partial solution.
The idea is that it gets run after all updates and etc-updates are done (I prefer to use cfg-update because it has an automatic mode, but your tastes may vary).
There's a magical directory (/var/lib/init.d/started) that contains a symlink to all the currently running services. This script keys on the simple fact that if a service has been updated, the modified time of the actual init script will be later than the modified time of this symlink. If this is the case, it's a fair bet that the service needs restarting.

There are several ways to control this script. First there are commanline options:
-p sets the script into PRETEND mode - no services will actually be restarted.
-v sets the script into VERBOSE mode - all running scripts will be listed along with a description of whether they were restarted, and if not, why not.
-d <dir> use a directory other that /var/lib/init.d/started to look for the magic symlinks
-b <file> use this file for the blacklist
-w <file> use this file for the whitelist
-h prints the usage message
-l <file> redirects output to a log file.

You can set a blacklist or a whitelist (or both if your perverse) of services this script can control. These should by default be placed at:
/etc/service_restart_blacklist
/etc/service_restart_whitelist

When restarting, a service is simply restarted with the "/etc/init.d/<service> restart" command.
It won't restart a service if it finds an update for the init script waiting in the wings.

There's a lot more things we could key on to decide if a service needs restarting or not, these may be added in if the basic script proves its worth. It's something I've seen a lot of people ask for, and I'd kind of like it myself - I'm playing with trying to get a machine which performs basic maintenance un-attended.

So, here it is, enjoy!

Update: As far as blacklists go, it's probably wise to blacklist the xdm service - you don't want to be logged out of your desktop session by an overzealous cron job!

Kooky · n00b Joined: 10 Sep 2005 Posts: 23 Location: Mannheim

Hi,
first of all "yes that is what i'm searching for!" but why you use a symlink in /var/lib/iinit.d and not just check "/etc/init.d/<service> status" to see if it is running?

Greets Kooky

Rcomian · Posted: Fri Feb 10, 2006 8:52 pm Post subject:

That's a good question. We need something to key on when deciding whether to restart a service, one of the simplest things I've found is that if the init script for the service was modified after the service was started, then it needs restarting. If you iterate over the init.d directory the scripts will tell you if they're started or not, but they wont tell you when they were started, so I had to dig around to find that information. Also, different scripts have different capabilities, so I didn't want to rely on them too much. The only place I found the information I was after was kind of en-passant - in the creation/modified date of the symlinks created to record which services are considered to be "running". These symlinks are created automatically when a service is started and they're deleted again when the service is stopped or zapped.

Now since we're already in this directory which tells us when the services were started, and it also, by definition says which services are running, there's not really much point in iterating over the init.d directory to speculatively find this information when it's given to us right there.

The location of the directory containing the symlinks is defined in the file /sbin/functions.sh. If you source this file and echo $svcdir you should see it. Eventually I'd like to use this environment variable to find the location in a more accurate way than having it hardcoded. If your svcdir isn't /var/lib/init.d then it may be /var/lib/supervise - so use that directory instead.

There's a lot of room for improving the detection for when a script needs restarting. Ultimately we can use genlop to see if a service or any of its dependencies have been installed since the service started, but knowing when a service was started is really key, so I can't see moving away from iterating this directory until there's a better way to find that information for whatever script we're looking at.

psychomunky · Guru Joined: 02 Nov 2004 Posts: 337 Location: Canada

This be damn cool, but I have a couple of questions/suggestions:

Nonetheless, this is a much better attempt than my hokey shell script that attempts to restart all the services in a particular run-level. My script also actually suffers from the pitfalls of point 3 above as well. Combining this with GLCU (Gentoo Linux Cron Update), revdep-rebuild and cfg-update (in automatic mode), would be a good start to a self maintained Gentoo box....

Rcomian · Posted: Thu Feb 23, 2006 7:55 pm Post subject:

Thanks for your comments psychomunky, I'll look at your points in reverse order, if I may.

3.
I'd actually just been heavily bitten by your 3rd point not 5 minutes before reading your post.

I'll have to look into a reliable way of working out the dependencies. The dependencies are cached in a fairly readable format in /var/lib/init.d/deptree, so that might be a starting point. I'll have a think and when I get a good clean answer I'll have a go at implementing it.

I'm also looking at making a default blacklist if no black or white list is defined at all, as having things like xdm restart in the middle of the night without warning could be devastating and I don't think it's necessarily reasonable to expect everyone who uses this script to think of things like that

2.
Looking for more reasons to restart a service is definitely something I want to move on. I had considered the /etc/conf.d issue. I didn't do it originally because it wasn't necessarily a clean relationship - the /etc/conf.d/net file indicated that not all services would have config entries with the exact corresponding name. Looking at the scripts involved with dependencies etc, it appears that the net config file is an exception that's dealt with explicitly and that most other services do expect the config file to have the same name as the init script. So, that said, I may well implement this as it wouldn't be difficult.

There's always the additional issue of external configurations (which are normally in /etc) - we'd probably want to restart if they changed as well.

I'd also like to look at dependencies of the packages. If a shared library has a critical security fix (well, lets face it, in practice this would be any fix) I want to restart all services that depend on that library, nomatter how far up the tree they appear. This should be fairly simple with judicious use of genlop and equery or their equivalent functionality although it would make the script much much slower to run.

One situation I'm very keen on is restarting apache when php is updated. This is awkward because this is a reverse dependancy - php depends on apache, not the other way round. Although this would be easy to do as a one off exception, I'd like to find some generic pricipal rather than coding in specifics, but so far all I can find is that we need to look for reverse dependencies in addition to looking for normal dependencies. If that's the case then fair enough, it's well defined and easy enough to put a command line switch in to turn this off.
Another possibility for this situation would be the ability to define explicitly linked packages (ie, when we check apache, also check updates to this list of packages). The problem I have with this is setting up sensible & expected default behaviour, and in the end it may well only be reproducing a subset of package dependencies anyway, without the benefit of having those dependencies updated by the people who really know what they are.

1.
As far as syslog logging goes, I'm all for it, although I've never even looked at how it would be done. Logging output to a standard output file would be very simple as well - just a new commandline switch and probably a wrapper to replace the "print" statements in the script. If you want to get your hands dirty with script - that would be an excellent place to start

One thing to bear in mind tho, is that actual failures whilst restarting services wouldn't be logged by this mechanism. The script would only log the outputs from the print statements. Obviously the reason I didn't have a log file in the first cut was because it can be manually redirected

Now as far as GLCU & a self maintaining gentoo goes, this is what I've been working to (very slowly) over the last couple of months. I've not even heard of GLCU before, I'm very happy to see that I'm not alone in thinking this is possible! (if not, perhaps practical, it's just to cover for my own lazyness).

I'll have a look at GLCU and see if the approach it takes matches what I'm trying to do, it should all fit together quite nicely

I'm also kinda keen to have some form of auto retry when a build fails, for example turning down the CFLAGS to see if that helps (or having package specific CFLAGS), but this is another project.

psychomunky · Guru Joined: 02 Nov 2004 Posts: 337 Location: Canada

Rcomian,

All very excellent points you make. It sounds like I am on the same, or at least a very similar page as you.

It is funny that you had been bitten by that, as the reason I mentioned it was, I was also bitten about a half hour before with a cheesy bash script I wrote to restart services.

Anyways, regarding point 2, I hadn't considered the /etc/init.d/net.* vs /etc/conf.d/net issue. I wonder if it is a standard naming thing that if an init script contains a . in its name, it's config file is only named up until the dot?? In this logic, if you see that /etc/conf.d/service has changed, you could check to see if you have an /etc/init.d/service script that can be restarted, and if you cannot find an /etc/init.d/service script is not found, then look of scripts matching the pattern /etc/init.d/service.* that may need to be restarted. the net scripts are the only ones I have seen as well with this sort of "odd" behaviour, but perhaps it was a forward thinking thing on Gentoo's part?

As for GLCU, you can find it at http://glcu.sf.net. The author was busy doing school stuff and hasn't updated for a while, but it almost looks like there is activity on it again. I have already tried your script in conjunction with GLCU by doing the following:

- placed your script in /usr/local/sbin (which is part of the path that GLCU ca use) and called it restart-services.py
- set the "updatetc" option in the glcu config file to be:

Rcomian · Posted: Sat Feb 25, 2006 11:34 am Post subject:

Fantastic patch, psychomunky - thanks for that.

I've integrated your changes, all good stuff and made a version 1.2.
This script includes a first attempt to handle the dependency stuff. At first I expected to be reading the dependency cache in /var/lib/init.d, then I thought I'd at least be parsing the output of "/etc/init.d/<service> needsme" and doing some cool stuff with it.
Then I realised that if you call "pause" and then "start", you restart the service without restarting any dependants. I'm a little dissapointed that the solution could be so simple, but you can't have everything.

Now I'm not sure if the dependants SHOULD be restarted in this sort of case or not. I suspect they might, but since this way is so simple I'm going to give it a try before doing anything more complex.

As a test to see the sort of effect that might be happening here I configured boa to listen exclusively on my external interface. I then paused the network and noticed that boa couldn't be reached, then started the network again, and boa came back again, all by itself.

Now by default I've stopped restarting the network interfaces (I just feel that's got too much potential for danger, or at least annoyance), but I'm taking the results above as an indication that pausing and starting the services may be enough.

Obviously, an option to restart dependencies as well, with full tracking for this, should be included, but in the interests of release early release often, I'll put this out and test it as it is, just to understand what's going on myself and see if there are any more pressing issues than that.

One issue I have noticed is that restarting services after an update often just doesn't work, I'm not talking about this script, it doesn't work even if you do it manually. I suspect the reason could be that a service is started with one version of the init script, then another version of the init script tries to stop it. If there's been any changes in how this start/stop happens (and after all, why else would an init-script change?) then it looks like the new script can't find the old service to stop and fails. This results in the service still actually running and needing to be manually killed, zapped and started again.

I suspect that the solution may be that if an unapplied update to an init script is found, the service should be stopped (BEFORE the update is applied), then the updates applied and the script restarted again. This means we get a wrapper to the config update scripts (etc-update, dispatch-conf or cfg-update), where we stop potential scripts early, try to apply the updates then ensure the services are started again after the update.
This is a fairly fundamental change to the script and I'm wondering if it would be better implemented as part of the config update scripts.
It could be done in this sort of way:

psychomunky · Guru Joined: 02 Nov 2004 Posts: 337 Location: Canada

Glad you liked the patch....

As you could obviously tell, I haven't done a whack load of python scripting. I am assuming that moving the function headers I had into triple quotes is something similar to javadoc comments in Java (if, of course, you are familiar with Java). Is this correct??

Anyways, I see that you've discovered some more ways for me to try to melt my brain. All interesting points... But I have a couple questions before I'll attempt to help:

1. This business of pause/start....does it work for all services?? I have looked through a lot of init.d scripts and off the top of my head, I cannot recall one that actually defines a pause() method. As well, are there going to be some services that actually do a "true" pause and remain in memory and when a start is invoked actually just continue where they left off?? I definately agree that this is a way to try out to see if it works first....the simpler the better IMHO.

2. Restarting the network interfaces..definately could be trouble there. Unless you are using 3rd party drivers, or really need a new feature in the new baselayout, I cannot think of a really good reason to need to restart the network. Although, I must say, I have restarted the network on a remote box via SSH and was not dropped at all....I think it is a situation similar to your boa experience. (Stopping on the other hand will completely hoop your session

I agree with omitting them for now...perhaps the best long term approach, should this ever get into portage would be to have a default blacklist that includes xdm and net

3. This issue of new init scripts not being able to stop the old ones is also something I have found upon occasion. I have never had it with any of the baselayout scripts (net, localmount, etc), but I have experienced it a few times with things like apache. It doesn't happen often, but it does happen. I think that this might be a bug or a lack of a feature in gentoo's rc system. I know that the start-stop-daemon executable is used to start and stop most services, and that it and the rc system track what is running vs what is not via pid files. Perhaps the rc system should be enhanced to track the pid files along with the service names in order to be able to tell if there is a process still running for that particular service and if the stop() method defined in the init script fails, it should forcefully terminate the process if it exists. This behaviour could be controlled via the /etc/conf.d/rc file.

Your solution on this would also work. But both of our solutions involve modifying other utilities. If there was some way to determine the PID file that is being used by the rc system, then your script could do the manually kill and zap thing before restarting the service.

Although, perhaps we are trying to solve world hunger or invoke world peace here. Maybe the simplest and best solution is to just e-mail the sysadmin if there is a situation like this, or if a service errors and cannot be restarted. Realistically, we cannot expect things to go smoothly all of the time, but if the script could notify a sys admin that something needs to be looked at, well then, at least we don't have to worry about the box until something goes wrong. This is the way I have seen a few things take, like GLCU and a lot of my DBA monitoring tools....they try to do updates and maintain the system like 80-90% of the time, but when they encounter that other 10-20%, they detect it and let the admin know.

I will install the new version of the script tomorrow and take it for a test drive, but for now, I must get some sleep.

Rcomian · Posted: Fri Mar 03, 2006 1:35 pm Post subject:

Pause/start should work for all services. According to the documentation:

Rcomian · Posted: Sat Jun 17, 2006 9:27 pm Post subject:

Ok, I've now got a new script using portage's new hook capabilities.
https://forums.gentoo.org/viewtopic-p-3388303.html#3388303