Techie manipulated time itself to get servers in sync • The Register
On Call Why, look at the time! 7:30 AM on Friday morning, the moment at which The Register regularly runs a fresh instalment of On Call, the reader-contributed column that shares your finest tech support stories.
This week, meet a reader we’ll Regomize as “Kim” who was asked to investigate an air-gapped lab in which time had become unhelpfully inconsistent.
“The problem was that an application server was more than five minutes out of sync with the domain controller that served as the lab’s Network Time Protocol (NTP) server,” Kim told On Call.
His inquiries led to the discovery that all the servers were out of synch with the NTP server, and that while staff had known about the problem for over a year nothing had been done.
Higher-ups eventually got wind of the situation and issued an edict: The application’s timestamps must be within one minute of real time.
Kim dove into the problem again, and found the controller’s NTP service configured correctly, but attempts to manually sync the machines saw the NTP daemon report it could not find any servers.
Which was odd, because the NTP server was clearly broadcasting the time, and other machines in the lab requested and received time information.
Kim eventually remembered that the NTP daemon has a “panic threshold.”
NTP’s job is to keep time and broadcast it so that clocks on all machines connected to a network are in synch. This matters for many reasons, among them consistency of log files – if every machine on a network kept different time it would be very hard to investigate an incident.
The panic threshold exists to stop a client of an NTP server synching if there’s a big discrepancy between time as experienced by client and server.
The discrepancy on the network Kim tried to fix was well beyond the preset panic threshold, so the NTP daemon did as it was told and panicked.
The fix was simple: increase the panic threshold so the daemon stopped panicking.
Kim made the change, and declared to On Call that doing so made him an “Air-gapped time cop!”
Have you messed with time and space to fix tech? If so, hop in your TARDIS and click here to send On Call an email so we can tell your story on a future Friday. ®
