Timing Resilience: The Clock Nobody Was Watching

Why Timing Resilience Matters

Networks are built to distrust things. Packets are protected, certificates are validated, links are monitored, traffic is inspected on the way in and on the way out. Time is the exception. It arrives from a time server, and it is believed.

That is a lot to take on faith, because almost everything else rests on it. A mobile network needs its base stations aligned in phase to within microseconds; lose that and cells begin to interfere with one another. A power grid uses synchronized measurements to locate a fault and isolate it before it spreads. Payments, transport signaling, broadcast and emergency services all depend on distributed clocks agreeing with one another, often to within fractions of a second. When those clocks agree, nobody notices.

Most critical dependencies are managed deliberately. Power has backup generation, connectivity has redundant paths, security has audits and penetration tests. Timing, in many networks, comes from a device that has been doing its job without incident for years. When problems are so rare, it often escapes the scrutiny applied to other critical systems. Reliability becomes its own blind spot.

A dead clock gets noticed. A wrong clock gets believed.

It isn’t only people who take timing on trust. Systems typically have no independent way to check whether the time they have been given is correct.

When a time server dies outright, going silent or unresponsive, systems notice and fail over. That is the failure everyone architects for. The harder problem is a clock that does not appear to fail at all. A device can confirm that it is running. It cannot confirm that it is right. So it keeps running, keeps answering and keeps distributing the wrong time with a straight face. The systems downstream have no reason to doubt it. They synchronize to it. Everything stays perfectly consistent with everything else, and perfectly wrong.

Certificates that are valid today fail validation against a date years in the past. Sessions that authenticated a moment ago no longer make sense. The failure rarely announces itself as a timing problem; it shows up everywhere else, as everything else, which is why timing tends to be diagnosed last, after everything else has been ruled out.

What resilient timing looks like

The defense against this is well understood, though it takes deliberate effort. It means treating time as something to be verified rather than assumed.

• Diverse timing sources: No single reference, GNSS or otherwise, should be a single point of failure. Holdover capability and alternative distribution paths turn a source failure into a manageable event rather than an outage.
• Constraints on what time is allowed to do: A reference that suddenly reports a date years out should not be accepted on its own authority. Limits on step size and rate of change are inexpensive to specify, and they convert a silent failure into one that announces itself.
• Verification before deployment: Timing chains, failover behavior and failure modes can all be tested in the lab before they are trusted in the field. An architecture that has never been made to fail under controlled conditions will eventually fail under uncontrolled ones.
• Monitoring and measurement in the live network: Sync performance measured continuously turns an assumption into a record. Collecting key performance and operational data from network elements is what lets you prevent disruptions, or understand them when they happen. Degradation shows up as data long before it shows up as an outage.

The same discipline applies whether the reference distributes time of day or the phase and frequency alignment the radio network depends on – and it is the second where the tolerances are tightest and the verification hardest.

The industry has already worked most of this out. The ITU-T recommendations that define timing performance for telecom networks set out what verifiable timing looks like (see as an example Recommendations in the G.826x and G.827x series). Solutions with enhanced resiliency are continuously specified (e.g., enhanced Primary Reference Time Clock in G.8272.1 and coherent PRTC in G.8272.2). Work is ongoing to specify new concepts and solutions in ITU-T G.8274 for enhancing synchronization network resiliency.

Regulators internationally have reached similar conclusions about resilient PNT: diverse sources, strong holdover, continuous monitoring and timing treated as critical infrastructure in its own right.

The gap is operational. Standards describe what good looks like. Applying that discipline to what is already installed and running is a different exercise – and it is the one that gets skipped.

The latest reminder

In July, a national mobile network went down for almost five hours after the clocks in its time synchronization nodes jumped back twenty years. Trains stopped running. Traffic lights dropped out. Payment terminals failed across the country. Hundreds of emergency calls didn’t connect.

The operator has said the outage originated in nodes managing time synchronization within some of its data centers, and reporting has attributed it to a known fault that reset server clocks by around twenty years. The official investigations are still ongoing, so the full picture is not yet established.

What is established is that clocks landing two decades in the past is a documented failure mode, not a novelty. GPS receivers count weeks in a fixed-width field. When that counter wraps, a receiver that has not been updated to handle it calculates a date roughly two decades earlier. Nothing about the device looks wrong: it still has satellite lock, its hardware is fine, every check it runs on itself passes. It simply reports the wrong year, and everything downstream believes it.

Whether or not that was the mechanism here, the wider point holds: a fault like this can sit in a network for years without ever drawing attention to itself. Regulators are also beginning to treat timing failures as compliance failures, with investigations and the prospect of significant financial penalties following outages of this kind.

Somewhere in every timing infrastructure there is a device like this: installed years ago, still running, still trusted, rarely examined. The most dangerous piece of equipment in a network is rarely the one that fails often. It is the one that has never failed yet.

Most network or clock failures can be prevented with proper verification and testing practices. When you can’t stop a clock or a network from failing, you can make sure you’re the one who finds it first – while there’s still time.

About the Author Stefano Ruffini is Strategic Technology Manager at Calnex Solutions and plays a central role in global standardization, serving as Rapporteur for ITU-T SG15 Q13 and contributing actively to IEEE 1588 and other synchronization forums. He has been working on synchronization aspects for more than 20 years.