Server clock falls behind

audiodef · Posted: Sat Jul 27, 2013 12:45 am Post subject: Server clock falls behind

What could cause my server's clock to start falling behind after setting it via ntp-client? Within hours of running ntp-client, I notice that my server's clock is a minute behind and falling further so.

My current server is a dedicated hardware server. It was built from a stage-4 tarball of when it used to be a virtual machine, and one that had known clock skew issues. But the kernel is one I built with a fresh seed when I moved to a dedicated server. I'm guessing there's a kernel setting I should adjust, but don't know which one. I'd appreciate advice.
_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

I have experiences with some HP 8xxx class workstations that were based on an early generation opteron (maybe even original slegehammer) where the mobo and an early 2.6.x kernel had bad clock skew. The only other time I've seen something like there is where line noise was coming in through the power supply and causing some sort of resonance or ringing problems with the clock crystal. That one was due to some microwave frequency interference from radars.

audiodef · Posted: Mon Jul 29, 2013 1:52 pm Post subject:

I don't think I have that processor, and I have a much later kernel. Maybe I should email OVH and see if they know of any clock skew issues.
_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

The kernel can account for clock drift, but to do that it needs continuous NTP updates. Running ntp-client once isn't going to work.

NeddySeagoon · Posted: Mon Jul 29, 2013 5:38 pm Post subject:

audiodef,

ntp-client is a one time set the clock application.
You need ntpd to check and sync the clock too.

PC crystals are very poor timekeepers and they are always slow.
Crystals the run a little fast can be trimmed down. Crystals that run slow are sold cheaply or scrapped.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

audiodef · Posted: Wed Aug 07, 2013 3:40 pm Post subject:

Does it matter whether I use ntpd or run ntp-client as a cron job every hour?
_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

NeddySeagoon · Posted: Wed Aug 07, 2013 7:58 pm Post subject:

audiodef,

Unless your clock is so bad that ntpd cannot calibrate a correction, ntpd is preferred.
Your clock may run slow but it should be a fixed rate slow. Once ntpd has calibrated, it will keep the clock correct. ntp-client will make step changes to system time, which is in general a bad thing.

Steps forward are preferred over steps back. The latter will play havoc with make.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

audiodef · Posted: Wed Aug 07, 2013 9:00 pm Post subject:

Interesting... I just noticed that ntpd is already started and yet I've been having clock problems. Upon restart of the daemon, I got "start-stop-daemon: no matching processes found", so maybe it got killed somehow and the pid got orphaned. It started up, and a subsequent restart had no repeat of this issue.

I've taken ntp-client out of its cron job and will see what ensues.
_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

NeddySeagoon · Posted: Thu Aug 08, 2013 5:25 pm Post subject:

audiodef,

Read man ntpd note the -g option.

If you want to look at the drift its in /var/lib/ntp/ntp.drift
I have not read enough to know if thats an offset or a rate to be applied to timekeeping.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

morpheus2051 · Tux's lil' helper Joined: 07 May 2006 Posts: 95

Hi,

even if you use the -g parameter, ntpd kills itself when the clock skew gets greater than a 1000s during the time ntpd is running. I had this problem on an old kvm machine. Worked around it by calling ntpdate and then restarting ntpd every few hours. With a new kvm version this problem went away.

Akkara · Posted: Fri Aug 09, 2013 10:56 am Post subject:

If the old system had a bad clock, it's possible there's files laying around that has incorrect information for the current system. /etc/adjtime is often the culprit, if you brought the old image directly over and it had one there.

It is OK to remove that file, it'll get re-generated using new measurements of clock skew. (But, it's been a while since I had this problem and things may be different now, so check first.)

audiodef · Posted: Fri Aug 09, 2013 3:10 pm Post subject:

It seems to be working OK. No major skewing in the last few days. /etc/adjtime was 0.0.0.0 but I rm'd it anyway (thanks for that advice).

It seems like -g only really helps once, but I will peruse the man page if I run into any further major skew.

I appreciate all the responses.

_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

CleanTestr · Posted: Sat Aug 10, 2013 9:33 am Post subject:

Back in the 90s we used to get PCs with 'really bad clock skew' (at D+B, Inc, USA).
Using an add-on PCI-bus clock-card would solve the problem: it 'replaces'
the crystal and RTC on the mobo.

(If you don't mind spending the money, I seem to recall that, you can get
'oven-controlled crystal temperature' on some high-end add-on cards)
_________________
Stan: A signal? Why didn't you wake me?
790: It was a distress signal. They only lead to trouble, so I always ignore them. --Lexx

NeddySeagoon · Posted: Sat Aug 10, 2013 11:56 am Post subject:

CleanTestr,

If you don't mind how much you spend, you can get an atomic clock for a PC.
Of course, it doesn't fit inside the PC case :)
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

audiodef · Posted: Sat Aug 10, 2013 2:10 pm Post subject:

Get a bigger case?

_________________
decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN

PaulBredbury · Watchman Joined: 14 Jul 2005 Posts: 7310

Don't use cron for this. And take this opportunity to switch from ntpd to chrony, as Fedora and Mageia have done.

CleanTestr · Posted: Sat Aug 10, 2013 5:32 pm Post subject:

@ NeddySeagoon:

that would be the 'older style' Cesium Atomic Clock, would it not?
_________________
Stan: A signal? Why didn't you wake me?
790: It was a distress signal. They only lead to trouble, so I always ignore them. --Lexx

NeddySeagoon · Posted: Sat Aug 10, 2013 5:35 pm Post subject:

CleanTestr,

Yep.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

audiodef · Posted: Sun Aug 11, 2013 2:58 pm Post subject:

PaulBredbury · Watchman Joined: 14 Jul 2005 Posts: 7310

CleanTestr · Posted: Sun Aug 11, 2013 4:14 pm Post subject:

@ audiodef:

Not to rain on ne1's parade, but, though finding a purely software solution to this
problem might be nice, in Theory, perhaps some consideration ought be given to
the underlying problem: that, once set via ntp, there is clock drift.

Crystals are simple beasts: they're affected by current, and by temperature.
On a larger server, it's possible that the cooling given to the cpu is not also applied
to the part of the mobo housing the crystal.

(not knowing your configuration) In the past I've run AT-style cases with 2 additional
case fans (one front, one rear) in addition to the power supply fan; and in rack-
mount equipment (such as raid), as many as one fan per 3 drives in a cage.

An example that might make sense for large cooling heatsinks on cpu chips is:
when there is side-flow from the heat-sink fan (out horizontally across the surface
of the mobo, for instance). If the crystal is in the path of such outflow, it could be
running at 2x or higher its rated operating temperature.

You might want to take a look at better air-flow within the server, perhaps.
_________________
Stan: A signal? Why didn't you wake me?
790: It was a distress signal. They only lead to trouble, so I always ignore them. --Lexx

NeddySeagoon · Posted: Sun Aug 11, 2013 5:57 pm Post subject:

CleanTestr,

Crystals are manufactured to some random tolerances but the frequency distribution will be gaussian for any reasonable process.

Those that run too fast, can be trimmed down but the trimming has an adverse effect on lots of other parameters, like stability and temperature coeffcient of frequency. These crystals do not find their way into PCs.
Crystals that run too slow are either scrap or sold for use in applications where frequency precision isn't too important, like PC timing circuits.
Here, we are talking about the CPU frequency control, not the real time clock, which is usually pretty good.

After boot up and the kernel has synced its real time to the RTC, the kernel keeps its real time updated from the rather poor low frequency crystal used for the the CPU clock.
No amount of cooling will fix it - its already too slow.

Hence the software calibration and correction approach.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Jaglover · Posted: Sun Aug 11, 2013 6:05 pm Post subject:

I have a M/B that just couldn't keep the clock right with openntpd. It kept fluctuating too fast / too slow. It is a desktop M/B with mobile CPU. I switched to net-misc/ntp and haven't had a clock issue after that. My 2¢.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

CleanTestr · Posted: Sun Aug 11, 2013 9:11 pm Post subject:

@ NeddySeagoon:

Well, * if it were Me *, I'd be in there with a temperature probe and
an oscilloscope...

Not saying that that's the way to go, in this case... ~~P. -- * ~.

Ohhh, well.
_________________
Stan: A signal? Why didn't you wake me?
790: It was a distress signal. They only lead to trouble, so I always ignore them. --Lexx

NeddySeagoon · Posted: Sun Aug 11, 2013 9:27 pm Post subject:

CleanTestr,

Been there done that ... but not on a PC.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.