Measuring the Accuracy of Simplified PTP Time Transfer

A measure of the elegance of a design is how well it stands the test of time and how well it can be adapted to changing needs. By this measure, David L. Mills had the most elegant of designs when he invented Network Timing Protocol (NTP). NTP has been adapted to the need for greater accuracy with Precision Timing Protocol (PTP) which is now enshrined in standard IEEE 1588. PTP, in turn, has been adapted to a number of use cases by using different profiles such as for Telecoms and industrial applications and even for sub-nanosecond synchronization applications with IEEE 1588-2019 Annex M (White Rabbit). While it is robust and versatile, PTP has some overhead associated with things like handshakes, timers and subscriptions in networks that support unicast. The packet exchange sequence is shown below. Meta (formerly known as Facebook) has recently developed a profile for PTP that takes advantage of the fact that in their Data Centers, they have a well-defined and homogeneous sync architecture using unicast packet exchange and transparent clocks in the network. The diagram below shows the packet exchange sequence they use and comes from github at tree/main/ptp/sptp. The protocol is referred to as Simplified Unicast PTP, simple PTP or sPTP. The Client sends a DELAY REQUEST packet to initiate an exchange with the Server. The Client records timestamp t3, while the Server records CorrectionField_2 (CF_2) from the DELAY REQUEST packet and the Rx timestamp t4.  The Server then sends a SYNC packet adding timestamp t4 in the originTimestamp field, and records the Tx timestamp t1. The Server also sends an ANNOUNCE packet with the Tx timestamp t1 and CF_2 from the DELAY REQUEST packet.  The Client records t2 of the received SYNC packet, plus CF_1. It also records data from the ANNOUNCE packet and CF_2.  As a result of this exchange the Client has access to t1, t2, t3, t4, CF_1 and CF_2 to calculate mean path delay and offset metrics.  In addition to the benefits including simple implementation, no Server-Client negotiation, Client-driven exchange, and low resource consumption, sPTP provides a potential bonus for network monitoring. This is the ability to centrally monitor the absolute time error for any remote Client.   To see how to monitor synchronization in a Data Center packet network, let’s take a look at the reference network diagram from the OCP Data Center profile.   Given the reduced resource required, there is the potential for each Server to implement both the sPTP timeReceiver and sPTP timeTransmitter functions. The Clients will issue DELAY REQUESTs to the Grandmaster and process the received SYNC and ANNOUNCE/ FOLLOW_UP messages to discipline their own clocks. In addition, if transparent clocks and unicast packet exchange are used, the Server could also periodically respond to a monitoring device’s DELAY REQUEST with their own SYNC and ANNOUNCE/FOLLOW_UP messages using their own clocks for generating t1 and t4. This holds the potential to solve the problem of how you verify the accuracy of the delivered time when there is no reference located at the timeReceiver other than the network-provided PTP time. Here we see time being delivered to a Client from a Grandmaster via the red path while being monitored via the yellow path. A suitably powerful monitoring instrument could poll through all Clients periodically, comparing time domains and Grandmasters to ensure an accurate source of time and assuring network synchronization accuracy through continuous performance monitoring. Moreover, the monitoring instrument could be directed to specific devices based on indications of sync failure, probing network devices to check whether time synchronization has been maintained at various points in the network. Learn more about the Sentry and Sentinel measurement solutions.

Open Compute Project (OCP) Summit 2022

This year’s OCP summit was held in person in San Jose California 18-20 October. The Open Compute Project (OCP) was initiated in 2011 with a mission to apply open source and open collaboration to increase the pace of innovation in data centers. The Global summit like data centers themselves are bursting at the seams and so the event was extended to a full 3 days of workshops, presentations and solution demonstrations with hundreds of companies participating. Calnex was there with our synchronization test solutions. Time and phase synchronization accuracy is an increasingly important topic in data centers. With the virtualisation of applications, better accuracy is required. Also, more and more data center operators are realising the potential efficiency gains that can come from better sync accuracy. It was not surprising therefore, that, while some folks wanted to learn about PTP, most people in the industry are already familiar with it and are in various stages of deploying it in their networks. There will be a lot to hear about this subject at the Sync focused ITSF show in Dusseldorf the week of 7 November. But what did we think were the really hot topics at the OCP event? Well, the first was, in fact, heat. With the increasing density of the electronics and the growth in processing-intensive applications, liquid cooling is proving to be the necessary solution. With data centers consuming about 1% of the total world demand for electricity, it is no surprise that energy use and energy efficiency were big topics. The global energy situation brings this into sharp focus, but even before the recent issues, we know that the power demands of a single modern data center can put strains on a local grid. We can see this creating challenges in locating new facilities. New data centers are being commissioned all the time and given the scale of these facilities and the huge investment made to build them, it is crucial that they be fitted out and commissioned as quickly as possible so as to start generating revenue. We also know that in addition to large data centers, networks require thousands of small data centers located near the network edge to lower the latency and latency variation to the end user. Because of these two dynamics in the industry, modular hardware systems for data centers was the third big topic we followed at the summit.

Data Center Synchronization Assured

At ITSF 2022 data centers were one key market that was extensively discussed throughout the presentations and a key element highlighted throughout as something that needs to be minimized was latency. Modern applications require low levels of latency to meet user expectations and to support this, many data center operators are looking to move the data closer to the user in edge-located centers. However, the decentralization of these data centers and the distribution of databases across the network is very inefficient without improved synchronization to the µs. Without this level of synchronization accuracy, very time-consuming processes such as commit-wait must be employed to assure that the correct and most current version of data is accessed. As a solution, many data center operators are adopting PTP. Hyperscalers are leading the way (read our META case study here), but it is only a matter of time before tenants start to seek PTP from their CoLo host to ensure synchronization of their data. Recognizing this, Calnex has launched Sentry, a data center native measurement solution providing independent verification that a data center’s synchronization accuracy is being maintained 24/7. As a data center-friendly rack mount solution, Sentry is controlled over VNC or via the API. It is capable of simultaneously monitoring multiple PTP, 1PPS and NTP signals to ensure that the synchronization capability that is relied upon is working as expected and that any trends in performance are spotted before they become an issue. Learn more about Sentry here.

5G OTA Wander Mask: Fact or Myth?

In my previous blogs (Synchronization Monitoring: The fuel gauge for mobile networks and Synchronization is not a single number), I discuss the importance of making time error (TE) measurements on the transmitted mobile signal and visualizing this information as a TE vs time plot. This gives valuable insight into the performance of the network and the likelihood that it will or will not exceed the ±1.5µs limit required by the 3GPP standard. The question now arises as to whether further metrics should be used on Over the Air (OTA) measurements and what these might look like for 5G. 5G deployment is in its early days and network operators’ experience with deployed 5G networks is increasing all the time. The industry is still discovering the challenges of synchronization in a network with many overlapping 4G and 5G cell sites. So, yes, given the ease by which metrics like MTIE can be calculated from TE data, it could only be helpful if we looked at actual network synchronization performance in different ways. MTIE is a good place to start. It is a familiar metric and gives us insight into wander characteristics. The ITU-T recommendation G.8271.1 specifies the required dynamic low frequency time error (dTEL) performance at the edge of the packet network with an MTIE mask. However, what is a good MTIE measurement when measuring over the air? Is there a mask for the MTIE of a 5G transmission? In short, no, but it is possible to derive an MTIE mask to guide in the interpretation of OTA results. Let’s start by looking back at 2G/3G mobile networks. Initially 2G/3G mobile networks were supported by a synchronous core such as TDM. The transmissions were Frequency Division Multiplexed (FDD) and interference was avoided by assuring frequency synchronization to 50ppb at the transmission. The core network had to provide synchronization that was stable enough over the long term to allow the base station to be synchronized to 50ppb. In the short term, the core network did not have to achieve 50ppb. Base stations connected to TDM circuits contained very narrow-band low-pass filters to ensure the 50ppb criteria was met on their transmission. The MTIE mask at the end of a TDM circuit was given in G.823. Fig.1 – G.823 – 2048 kbit/s interface output wander limit Frequency stability at the mobile backhaul is traceable to a PRC in the core network, so the long-term frequency is very good (much less than 50ppb), but over the short term there can be significant variations. Most of the left-hand side of the G.823 mask allows frequency deviations far in excess of 50ppb (as much as 46 parts per million at the extreme left). Therefore, base stations needed ultra-stable filters to be able to guarantee that frequency error was filtered down to 50ppb for the RF carrier frequency. Such filters had double-oven OCXOs (very large, expensive, power-hungry devices) and needed careful thermal handling. When G.8261.1 was developed, its purpose was to replace the TDM circuits connecting the base stations with Ethernet/IP networks. The assumption was that the base station design didn’t change, apart from the interface. In many cases, even that didn’t change, and the Ethernet/IP was terminated by an external cell-site gateway before being handed over to the base station on a TDM link. The G.8261.1 mask shown below is almost identical to the G.823 mask above, with just an extension on the right-hand side to ensure the frequency accuracy remains better than 16ppb. The G.823 mask guarantees 18ppb in the long term, but not 16ppb. The base station filter is expected to do the rest by removing the short-term variations to ensure compliance with 50ppb. Fig.2 – Output wander network limit for case 3 based on [G.823] For FDD systems, the MTIE mask at the RF output should contain a 50ppb limit on the short term, and a G.8261.1 limit on the long term. Fig.3 For illustration, the green line is G.823/G.8261.1, and the orange line is the 50ppb limit. The black line represents a notional MTIE curve for 2G, 3G and 4G FDD systems. As the fronthaul was not expected to add additional long-term noise to the output, the G.8261.1 MTIE mask dominates beyond about 400s. However, as the requirement is only that the transmission be synchronized in frequency to 50ppb, one could argue that the 50ppb (red dotted line) limit alone fully defines the MTIE curve. In 4G and 5G TDD systems, the 50ppb frequency synchronization still applies but there is also the requirement to synchronize phase. The 3GPP standard specifies that overlapping cells be synchronized to within 3µs of each other. However, this is widely implemented as each cell being synchronized to within ±1.5µs of UTC. If we combine these all on one MTIE graph, the curves that dominate are 50ppb and 1.5µs. This is shown in red below. Fig.4 While it is the norm to look at the time error (TE) vs time plot of the OTA signal, looking at the MTIE of this same information may give some additional insight into the network. As a minimum, it will give an indication as to whether short term noise in the timing results in frequency offsets exceeding the 50ppb limit. Bryan Hovey Product Manager Related product: Sentinel Related blogs: Synchronization Monitoring: The fuel gauge for mobile networks and Synchronization is not a single number

Synchronization is not a single number: Can we see what is to come?

In my previous blog, I pointed out that a good test solution can verify not only that the network is operating within performance limits of the ITU-T and 3GPP but also to give insight into just how much margin there is before a problem might occur. Regular monitoring also allows the operator to spot drifting behaviour and address the underlying cause before it affects their customers. Ideally, one would want to look at the performance of the entire network including the Radio Units (RUs) by measuring the RF signal over the air (OTA). While it is tempting to represent the network synchronization as a single number, this is deceiving. It is best to visualise the measurement on a time graph which gives the user an idea of the dynamic characteristics of the synchronization. It is also important to make these measurements periodically and to regularly make measurements for extended periods of time so that slow wandering fluctuations can be seen. Fig.1 shows an example of a short (10 minute) OTA measurement of synchronization on a 5G transmission. It demonstrates excellent performance with synchronization to about 200ns of UTC and a small variation of less than 40ns. Fig.1 – within limits Fig.2 shows the performance of another network with similarly excellent synchronization. Within the measurement period, the sync has a mean of 330ns from UTC and a peak-to-peak change of 35ns. However, in this case, an observer might question the long-term wander of the synchronization. Have we just captured a measurement that is mid cycle in a much larger variation? Fig.2 – potential trend Because OTA measurements can be made without access to the network hardware, it is a straightforward process to survey a number of base stations in your network quickly. You can also look at the synchronization of base stations in other networks since they too can create interference in your network. In my next blog, I look at how wander can be evaluated on an OTA measurement. Bryan Hovey Product Manager Related product: Sentinel Related blog: Synchronization Monitoring: the fuel gauge for mobile networks and 5G OTA Wander Mask: Fact or Myth?

Synchronization Monitoring: The fuel gauge for mobile networks

Extrapolation of ITU-T recommendations on sync to the output of the end user application can be challenging. ITU-T recommendations are intended to deal only with the packet network portion of a communication system. However, being always pragmatic, the ITU-T authors recognise that the packet network ultimately serves an end application. Because of the significance of mobile communication, the ITU-T recommendations include references to and are consistent with 3GPP standards. Fig.1 – Deployment case 1 This deployment case example is widely used. ITU-T recommendations define performance requirements to reference point C in support of various end user applications which each have their specific synchronization requirements at reference point E. ITU-T G8271.1 An example of this is in the ITU-T G8271.1 recommendation on Network limits for time synchronization. The recommendation itself lays down performance limits to the edge of the packet network. This includes limits on the maximum absolute low pass filtered TE network limit, max|TEL| as well as the dynamic low frequency TE, dTEL(t) and dynamic high frequency TE, dTEH(t). In an appendix to this document, the recommendation gives example design options and error budgets for a representative network with an end application. In this case, the end application has a total maximum absolute TE limit of ±1.5µs. While the recommendation never specifically states that this refers to 5G TDD, the ±1.5µs limit conveniently aligns with the 3GPP requirement that adjacent base stations be synchronized to within 3µs. ITU-T recommendations are often used to define the performance requirements of the overall core network in mobile communications as well as the individual components making up the network. Components often have elaborate specifications and performance requirements defined in the ITU-T recommendations to, among other things, assure interoperability in a network. The packet network itself has relatively few performance limits to meet as listed earlier ( max|TEL|, dTEL(t), dTEH(t)). Beyond the edge of the packet network (reference point C), one can refer to the 3GPP standards for performance limits. However, in the case of synchronization, the only synchronization accuracy metric is that the overlapping cell transmissions on the same frequency be synchronized to within 3µs (macro-cells with cell radius less than 3km). No distinction is made about constant or dynamic behaviour of the synchronization. This 3µs is allocated to intercell synchronization as part of the overall sync error budget in 3GPP to assure that there is no interference between transmitters. One could simply make spot measurements to verify that the intercell synchronization is within 3µs and use this as an indicator of the overall health of the network sync. However, the same logic applies to fuel gauges in cars. So long as all is well, the fuel gauge could be replaced by simple ‘out-of-fuel’ light, and all will be well until it illuminates. One of the values that a good test solution can bring to the operators of a mobile network is the ability to not just verify that the network is operating within hard performance limits but also to give insight into just how much margin there is before a problem might occur. Regular monitoring also allows the operator to spot drifting behaviour and address the underlying cause before it affects their customers. In my next blog, I will look at how measuring sync performance over the air gives a full picture of how the network synchronization is behaving and more importantly point at potential problems. Bryan Hovey Product Manager Related product: Sentinel Related blogs: Synchronization is not a single number and 5G OTA Wander Mask: Fact or Myth?

Innovating to Shape the Future of Distributed Edge

Colt delivers ground-breaking private 5G pilot in Paris supported by Cisco and Calnex. Innovation is at the heart of Colt’s strategy and we’re continually testing the latest technologies to meet the future needs of our customers. As part of our 5G and Edge incubation program we’ve been working with partners including Cisco and Calnex on a private 5G pilot in Paris. This explored a range of new concepts, including a distributed Edge platform and network synchronisation, both enabled on the >Colt IQ Network. Our goal was to provide valuable use cases, especially for digital transformation and Smart City concepts, and validate them in a real user environment. As part of Colt’s 5G and Edge incubation program, we have been working with partners including Cisco and Calnex on a private 5G pilot in Paris to trial a range of new concepts, including a distributed Edge platform and network synchronisation, both enabled on the Colt IQ Network. This was intended to provide valuable use cases, especially for digital transformation and Smart City concepts, and validate them in a real user environment. In a world where everything is connected and every user will have the opportunity to enjoy a highly personalised experience from home or at the office with the hybrid model, 5G offers massive potential to revolutionise both business and consumer markets. As a result, many sectors with high bandwidth requirements for business-critical services, including manufacturing, R&D, and broadcasting, are looking at how private 5G can help deliver the promised benefits. The real estate sector is also assessing private 5G, and our pilot examined the performance of indoor 5G connectivity and opportunities related to smart offices and consolidated private building networks. For developers and real estate companies, connectivity presents an ever-present challenge – particularly for their development pipelines, where the expectation is that they will be at ‘state of the art’ upon delivery, which may be several years down the line. The pilot represents an important step forward in further understanding the benefits 5G for business verticals like real estate and how to deliver them. It also highlights the importance of working closely with trusted partners, which we see as a critical element in delivering services in the future. Cisco supported Colt with the distributed Edge platform that underpinned the end-to-end private 5G architecture, by supplying the on-premise compute infrastructure to deliver the disaggregated Open-RAN architecture (DU, CU and security). In addition, it enabled Colt’s Telco Edge NFV infrastructure in Amsterdam, hosting the virtualised private 5G core, security gateway, centralised management and telemetry VNF applications for the overall solution. In addition, Calnex Sentinel delivered time-based test and measurement capabilities to support the network synchronization element of the pilot. This allowed Colt to successfully test the synchronisation between London and Paris, across the Dense Wavelength Division Multiplexing (DWDM) network over a distance of 661km. This long-term test delivered positive results for MAX TE (maximum time error), cTE (constant time error) and dTE (dynamic time error). Asymmetry measured via cTE was also adjusted at the destination, resulting in a maximum time error of just 72nsec, which was very promising and well within standard tolerances. This opens a range of possibilities for customers who are using applications that need high quality time synchronisation to deliver the real benefits of 5G. Cisco’s Vice President, EMEAR Service Provider, Adam MacHale, said: “Colt continues to invest in cutting-edge innovation and technology with Cisco. From Segment routing based 5G backhaul on its IQ Network to 400G OPEN ZR+ Routed Optical Networking the partnership goes from strength to strength. Colt have now further evolved the platform to tie their customers’ cloud and 5G services together with a transport network slice with Service Level Agreements. This proved the perfect platform for Colt’s Private 5G initiative as we explore opportunities across retail, healthcare and manufacturing, alongside enterprise networks, based on Wi-Fi.” Calnex’s Business Development Director, Paul Norris, said: “It has been exciting to be part of this leading-edge Private 5G network project in Paris. As the development of private networks in the business and consumer market grows, the importance of high-quality time synchronisation is ever-more critical for success. The measurement results of Colts IQ network performance captured on Sentinel demonstrated that the required tight sync standards can be achieved, and we are looking forward to the next step to test time error measurements at the network O-RUs.” Colt’s Executive Vice President – Strategy and Transformation, Jaya Deshmukh, said: “Our pilot in Paris, which was part of our 5G and Edge incubation program, was ground-breaking. Not only is private 5G is a new concept, particularly for the real estate sector, but Virtual Functions, Edge and Open RAN are also brand new in the market. In addition, delivering 5G sync from two different countries is completely new and offers really exciting possibilities.” Related product: Sentinel Related literature: Timing and Synchronization Library

Decoding ITU-T standards for optimum APTS network limits

ITU-T Rec. G.8271.2 edition 2 was published in May 2021 and addresses the network limits for deployment scenarios of partial timing support (PTS) in mobile networks. In PTS networks, it is assumed that the core network is not fully time aware. In other words, not every node in the core network implements a transparent clock or boundary clock function. In some regions this is a fairly common deployment model. The network may traditionally have implemented GNSS at every base station such that the core network did not have to be phase synchronised. More recently, there has been an increase in attention focused on the vulnerabilities of GNSS. Also new smaller cell deployments are often in urban canyons or even indoors without clear view of the sky. In either case, operators recognise the need to implement time awareness on the network. However, parts of the network may be leased or otherwise not under their direct control and may not implement timing support. Also, complete implementation of full timing support will take a finite amount of time. Operators, therefore, may have a network with GNSS as the primary timing source at the base station and partial timing support on the core network as backup if the GNSS timing becomes unavailable. This is the deployment scenario described by the recommendation as Assisted Partial Timing Support or APTS. It is assumed that the GNSS based time source at the base station, when operating properly, will have better accuracy than the network provided timing as it is not subject to impairments such as asymmetry. Asymmetry will create a constant time error in network provided timing. In the event of loss of GNSS at the base station, the instantaneous value of phase (time) will be maintained and only changes in the phase on the network provided timing will be tracked. Therefore, the ITU-T G.8271.2 network limit at the reference point C for APTS networks only specifies the peak-to-peak variation of the TE. Eventually, the operator of an APTS network may have full timing support in the core network while still implementing GNSS at the base station. We are then operating in what might be called Assisted Full Timing Support (AFTS). This is a different case from either a fully time aware network as is described in G.8271.1 or an assisted partial timing support network as described in G.8271.2. It raises the question as to which recommendation should be used to set the network limits? In the case of a network designed for APTS, in the event of loss of GNSS, the absolute phase is retained but holdover is extended by using the network provided timing to limit any phase variation. As G.8271.2 assumes PTS on the core network, it specifies that packet selection be implemented and so the network limit at reference point C to be Peak-to-peak pktSelected2wayTE < 1100 ns In the case of AFTS, packet selection is not required, and one might assume the network limit to be Peak-to-peak 2wayTE < 1100 ns If we test to this limit, we assume that the network uses GNSS as the primary reference at the base station and, in the event of loss of the GNSS, it maintains the instantaneous phase and then only tracks changes in phase as indicated by the core provided timing. However, if the network identifies the GNSS timing and the core provided timing as two valid master clock sources, then the constant time error on the core network is important. In this case, you would expect the core network to comply with G.8271.1 network limits. G.8271.1 sets network limits on the maximum and dynamic components of the timing error and so are more stringent than the limit set by G.8271.2. The most pragmatic solution may be to test to G.8271.1. This would assure that the requirements of G.8271.2 are also met. However, it is often best to speak with your vendor and understand how the network manages the event if GNSS is lost.

The Open Compute Project opens the door to big benefits for datacenters

The first presentation at the ITSF 2021 conference in Brighton discussed the implementation of Precision Timing Protocol (PTP) in datacenters and this was the first of many presentations that focused on the benefits and challenges of improved synchronization in this market. Improve Sync in datacenters and the benefits are shared Throughout the conference, it was pointed out that better synchronization in datacenters promises to improve the end user experience in sectors such as gaming and streaming. In the highly regulated Finance section, MIFID II regulations state that clocks be synchronized to within 100us of UTC. Therefore, a highly accurate synchronization protocol is vital for compliance. For broadcasters highly accurate synchronization enables the streamlining of the production process, eliminating the need to bring all equipment and resources to an event. Media feeds from an event can be sent from the venue and then combined with production effects and commentator video/audio created in another country. If time is known accurately at each location, there will be no perceivable artefacts created in the end broadcast. Mobile network operators are already familiar with the need for highly accurate timing and synchronization. As these operators disaggregate and virtualise functionality, as when implementing ORAN in 5G NR, the need for µs timing accuracy moves into the datacenter. Better synchronization in datacenters improves efficiencies. The relatively high timing uncertainty that exists in networks employing legacy protocols like NTP can create reductions in throughput. Servers and database software have become so powerful that dealing with the uncertainty around milliseconds of timing error, which is common when NTP is used, represents a huge overhead. It has actually been reported that, in trials, if the uncertainty in timing in datacenters is removed, throughput can be improved by up to 100 times. We know that our appetite for more and more data shows no sign of abating and this drives the building of more datacentre infrastructure to support the demand. Datacenter operators are, therefore, spending billions of dollars every year just to keep up. Anything they can do to improve the efficient use of existing infrastructure will benefit them and anything they can do to reduce the energy needed to drive their infrastructure will benefit us all. Another benefit of better timing accuracy is the simplification of operations. Accurate timing can simplify the operation of distributed data bases and allow engineers to start treating the datacenter like one big computer. So why don’t all datacenters roll out PTP across all their infrastructure? The challenge is that until you have the infrastructure it can be challenging to prove that the benefits outweigh the cost but it can be a challenge to prove the benefits until the infrastructure is in place. Part of the problem is that widespread deployment of PTP traditionally requires the use of large quantities of expensive off-the-shelf time appliances, but as Meta recently pointed out in their engineering forum, off-the-shelf time appliances come with drawbacks. While they are well proven and generally very well performing and stable devices, their drawbacks include: Technology is often older and vulnerable to software security concerns. Devices come with closed source software, making configuration and monitoring problematic. Proprietary hardware is included that is not user-serviceable. They can become very costly to purchase and operate. Making the business case with Open Compute Project (OCP) and Time Appliance Project (TAP) Several of the ITSF presentations dealt specifically with Meta and the work they have been doing in datacentre timing and synchronization as part of the Open Compute Project (OCP) and Time Appliance Project (TAP). The OCP TAP is making the business case for widespread PTP deployment just a bit easier. Through this project, Meta have developed a versatile and economical time engine enabling µs accurate PTP synchronisation any server with hardware timestamping. The time card, as it is called, can support hundreds of thousands of servers enabling deployment at the scale present in modern datacenters. Meta has made the time card HW and SW open source and available to all. The time card removes much of the cost and other drawbacks of reference clocks currently on the market. Calnex is pleased to also contribute to the OCP TAP via the Instrumentation and Measurement group which aims to provide a path to Open Sourced implementations of Test & Measurement equipment and systems. While the implementation of better timing can vary between datacenter operators and may involve boundary clocks or transparent clocks, PTP, distribution of 1pps or even white rabbit, the need for and benefits of better timing accuracy is well established. While the type of timing appliance traditionally used in other networks may not be ideal at the scale of hyperscalers, new technologies such as the OCP TAP timing card along with good network design and proper monitoring will allow operators to economically implement and reap the benefits of the wide-spread deployment of better synchronization. Related product: Sentinel, Calnex SNE

Elimination of guard bands in 5G NR increases need for sync testing

Only Over-the-Air sync testing fully characterizes network performance. The rollout of 5G networks worldwide is drawing sharp focus on the need for phase/time synchronization. Poor phase synchronization will lead to interference. The specifications laid down by 3GPP WG-4 clearly state that any pair of cells on the same frequency and with overlapping coverage are to be synchronized to within 3us. Although the 3GPP specification is given as a relative 3µs alignment between cells, the measurement is usually interpreted as alignment of each cell to within to ±1.5µs of a standard reference time such as UTC.   In an operator’s 5G TDD network, if two overlapping cells are out of sync, there is the potential for the downlink from one tower to interfere with the uplink from user equipment at a different tower. The problem is compounded because of the elimination of guard bands between the allocated spectrum of different operators. Even though adjacent allocated spectrum is notionally non-interfering, in practice it can still cause interference. It is not possible to completely filter away out-of-band emissions, and, without guard bands, these emissions can be at significant power levels within the band of an adjacent cell. If transmissions between different networks are not synchronized, interference can result, especially because the interfering tower, in this case, can be closer than the user equipment. An additional problem results when the poorly synchronized in-band and out-of-band interfering transmission drives the receiving amplifiers into nonlinear operation. The resulting intermodulation leads to further interference and performance issues. FREE on-demand webinar TDD interference scenarios Inter-operator synchronization Elimination of guard bands Synchronization requirements 5G synchronization signal block (SSB) Frequency positioning of the SSB Measuring 5G NR sync Over-the-Air (OTA) 5G NR OTA measurements     FREE on-demand webinar   TDD Interference scenarios Inter-operator synchronization Elimination of guard bands Synchronization requirements 5G synchronization signal block (SSB) Frequency positioning of the SSB Measuring 5G NR Sync Over-the-Air (OTA) 5G NR OTA measurements TDD systems like 5G NR, therefore, must maintain the specified phase/time synchronization, and operators must design their networks and select network appliances so as to stay within their timing error budget. Appendix V of ITU-T G.8271.1 offers example error budgets for networks using different classes of equipment. From this, we get the familiar threshold of ±1.1µs at reference point C between the core network and the end application. Even if it isn’t always convenient, point C is often the final point in the physical network where we are able to access a 1pps signal or the PTP packet communications from the upstream clock allowing us to measure synchronization. Downstream from this point sync testing gets tricky. We may no longer have traditional signals present that allow synchronization testing.   However, now with the accelerating roll out of 5G which is much more reliant on TDD and with the increasingly complex and disaggregated nature of the CU, DU, RU chain, it is more important than ever that phase synchronization is measured beyond reference point C and take into account the error contribution of the fronthaul components. Ideally, the measurement should be done over the air so as to measure the full synchronization performance of the network. This means more than a quick single number check of time alignment but a full analysis of the static and dynamic behaviour of the synchronization just as we have always done at reference point C.   Historically, it was very problematic making a robust and detailed analysis of the timing after point C. However, this may not have been a big issue as, prior to the implementation of 4G and 5G TDD, frequency synchronization was all that was required. We may have been content until now with the ITU recommendation that beyond reference point C, we can account for further time synchronization degradation with 150ns noise contribution from the end application and 250ns to account for failure events.   Part of the error budget developed by the operator will be an allowance for failure events such as re-arrangements and short-term holdover which will occur during loss of PRTC traceability or short interruption of the GNSS signal. It may only be once an operator starts measuring synchronization performance over the air that the question arises as to how to account for the budget allowances for failure events, and this raises an interesting thought. If you are analyzing a mobile cell transmission for its synchronization behaviour, what is the proper testing threshold for max|TE| at the air interface? The immediate answer is of course ±1.5µs. But wait, what about the sync budget allocated to holdover and rearrangement. If we assume that the network is running without failures when it is being tested, then shouldn’t the error budget for re-arrangements and short-term holdover be removed from your pass/fail threshold?   In Appendix V of ITU-T G.8271.1 several example scenarios are given where there is a failure in some part of the synchronization chain. These different examples have allocated sync budget for holdover and re-arrangement of anything from 250ns to 620ns. In the latter case, one could argue that when the system is operating without failure, the time alignment at the air interface should be ±880ns rather than the familiar ±1.5µs.   Well there is no right or wrong answer. What is important is that the operator constructs their own budget and that they carry out over the air tests to verify that their complete end to end network is operating as designed. Tests should ideally be conducted over an extended period of hours or even days to identify issues that might be related to the time of day. It also seems reasonable that elements of the error budget that relate to fault situations such as re-arrangement and holdover should be excluded from the target synchronization level that is to be attained during normal operation. Related literature: Over the Air technical primer Related product: Calnex Sentinel