Why Network Synchronization Fails and How to Prevent it

Synchronization is of little concern in a network … until it fails

Continuous compliance has become a growing concern, especially in data-driven industries such as telecommunications, hyperscalers, and high-frequency trading. One area where compliance is critical — and often overlooked — is synchronization.

The rising importance of synchronization in modern applications

Synchronization is vital in an increasing number of applications such as 5G, Broadband, high frequency trading (HFT) and distributed data base management in data centers. Failure to maintain sync in these applications can have serious and expensive consequences. However, too often the strategy appears to be to wait until there is an issue and only then look at sync performance. Dedicated monitoring of synchronization is seldom done. Possibly this is because synchronization is thought to be generally pretty reliable, or it may be that knowledge about how synchronization works is concentrated in a few individuals in the organization. Because of this, the number of sync related failures in a network may not be widely known in an organization.

The business impact of sync problems

What is the impact of sync problems? Depending on the industry, sync failure can impact an organization in different ways. However, it always results in a real cost.

Mobile Networks:

In mobile communications the move from 4G to 5G brought gains in bandwidth and services but carried the requirement that all base stations be synchronized to within ±1.5µs of UTC (Universal Coordinated Time). Failure to maintain this synchronisation can lead to interference between base stations and between providers leading to poor performance and reputational damage.

Broadband Networks:

In broadband networks, growth in bandwidth demands and the number of subscribers, as well as the need to deploy new services, has led to changes in the way broadband networks are architected. As a consequence, synchronisation is now required in the network to avoid a poor customer experience.

Data Centers:

Data Center operators have been able to greatly speed distributed data base management using less processing power, but it requires tight sync in the network. Failure to maintain synch can lead to the distribution of bad data.

Financial Networks:

Perhaps nowhere else is the need for time synchronisation more apparent than in HFT. Besides the direct consequence it has on competitiveness, synchronization is maindated by financial regulators. Failure to comply can lead to fines, penalties and other legal consequences.

What do these companies do to maintain accurate sync?

In some cases, synchronisation is not monitored at all. Companies rely of existing Network Management Software (NMS) which may not detect sync issues or only report the symptoms of synchronisation problems once it has gone seriously wrong. Action is only taken when a problem occurs. Often, significant time will be spent diagnosing the problem before it is even identified as a sync issue. Continuous monitoring, however, can identify issues in the synchronisation before they affect performance.

Periodic Compliance Checking in Regulated Industries

In other cases, especially in regulatory environments, compliance is only checked periodically. When synchronisation issues are detected, a company may have to suspend operations at an exchange until compliance is restored.

The Case for Continuous Synchronization Monitoring

Given the risks associated with failure to maintain accurate synchronisation, the case for continuous compliance monitoring is strong. However, the challenges to implement a continuous compliance system that can detect time error at the edge of the network are daunting. A couple of the biggest issues are scale and the lack of availability of a reference.

Why Measuring Synchronization at Scale Is Hard

In an ideal world, measurements of time synchronisation would be made at every ‘user’. However, this involves the deployment of measurement hardware on a huge scale making this approach impractical. Even if a measurement device is present, it can only measure time error if the local time can be compared to a reference.

The Role of PTP in Network Time Distribution

It is common to distribute time in a network using the Precision Timing Protocol (PTP). This delivers accurate time to the devices on the edge of the network from a central time server (The Grandmaster clock). However, because no timing reference exists at the network edge, it is difficult to verify the accuracy of the time delivered through PTP.

Combining NMS and Sync-Specific Monitoring Tools

Leaders in their industries have developed practical solutions that provide continuous compliance monitoring. This often involves the use of traditional NMS systems in conjunction with dedicated synchronisation measurement tools.

Monitoring tools such as conventional Network Management Software (NMS) are useful as part of the solution. They can be configured to detect alarm conditions in the devices that are transferring PTP information. Should one of these devices (a boundary clock or Transparent clock) issue an alarm, it is an indication that time is not being delivered, and synchronisation has failed. A generic NMS is useful in this application even though it often can not make use of all the sync related information that is available from devices in the network. A dedicated synchronisation monitoring solution would be able to give additional insight into the network, would detect more subtle sync failures and would speed troubleshooting.

Monitoring Sync Alarms and Verifying Grandmaster Accuracy

Monitoring the alarms gives part of the sync picture. This provides some assurance that time is being delivered through the network. Equally important is to verify that accurate time is being supplied by the grandmaster into the network. Grandmasters are often referenced to GNSS and provide an output accurate to within ±100ns of UTC. There are often several GMs in a network so that backup is available. A robust continuous compliance monitoring system would be able to measure the accuracy of the GM output. However, to directly measure the GM output using traditional T&M best practices would require a reference with an accuracy of around ±10ns to UTC. This is impractical in a commercial application. However, there is an alternative that has proven effective in large scale deployments.

A Scalable Solution, Comparing Grandmasters

Instead of directly measuring the output of each GM, it is more effective to compare them to one another. Commercial off the shelf test equipment is available that can compare multiple GMs to a relative accuracy of ±5ns. The failure of one GM will quickly be detected by the divergence of the measured signals. If suitable attention is paid to spatial diversity of GNSS antennas, this technique can detect the failure of a GM or the jamming or spoofing of the GNSS.

If proper attention is paid to reporting of anomalies and the archiving of data, a complete and continuous record of compliance is possible. This has been proven in both hyperscale and HFT applications. These companies can more quickly detect and resolve issues before they become problems. They can easily show compliance to regulations and SLAs. The need for accurate synchronisation will continue to increase and the consequences of failure will become more onerous. Failure to test synchronisation or the use of only periodic testing is not an effective way to manage synchronization.