An open infrastructure for sub-millisecond internet time
Our vision on the time infrastructure of the future
Microsecond level or more accurate time is crucial for emerging internet services, such as smart grids, high-frequency financial trading, and industrial automation. However, providing sub-millisecond time at internet scale is problematic today because the underlying infrastructure is still in its infancy, with only a few operators of time sources and time distribution networks serving specific regions or target groups. This for instance reduces the resilience level of the time services that internet operators such as ISPs can provide, because they cannot easily get a time signal from multiple time distribution networks. To solve this problem, we propose the establishment of an open sub-millisecond internet time infrastructure that makes it easy for new players to provide, distribute and use sub-millisecond time. To accomplish this, we propose developing (1) a set of best operational practices and open-source software components for new operators of time sources, time distribution networks, and internet services and (2) a database to easily discover and connect to time distribution networks.
What is time on the internet about?
Time is crucial for an increasingly wide range of internet applications and services. For example, internet security protocols like TLS (secures end-to-end communication), DNSSEC (secures domain name to IP address mappings), and RPKI (secures routes across the internet) cannot function without accurate time, nor can a function such as domain name registration. Time also plays a crucial role in critical infrastructures, which increasingly use the internet and its protocols for their communications. Examples include high-speed financial trading systems, power grids, charging stations, remote ship navigation, and telecom networks.
Timing inaccuracies can have serious operational consequences, such as data loss because backup programs kicking in too late, or the cache of a DNS resolver being flushed prematurely resulting in performance or availability problems. It is therefore of paramount importance to build dependable time services that can serve internet applications with varying accuracy requirements and to consider those services a first-class citizen of the internet infrastructure. This is unlike today’s situation, where internet time services are often-overlooked.
Time distribution: the network time protocol
Internet time services such as the widely used NTP Pool are typically based on the Network Time Protocol (NTP). NTP infrastructures distribute time through a hierarchy of time servers (see Figure 1), so theycan serve many internet devices. The top level is called “stratum 0” and consists of reference clocks, which are an NTP infrastructure’s authoritative time sources. Examples include radio-based clocks (e.g. using GPS) and terrestrial clocks (e.g. Caesium or Strontium atomic clocks, quantum clocks or optical clocks). Stratum 1 servers get their time from directly connected reference clocks or through a time-sensitive network (e.g. based on the Precision Time Protocol, see below). Stratum 2 servers get their time from stratum 1 servers, and so on. Time servers in the same stratum may also agree on their time in a peer-to-peer way, as indicated by the horizontal arrows in Figure 1.
Figure 1: Organisation of an NTP infrastructure (based on Wikipedia).
The accuracy of stratum 1 servers depends on that of the reference clocks and on the hardware of the server itself. For example, a GPS-based reference clock can provide the time with an accuracy of 30 nanoseconds or less (95% of the time), whereas a Caesium atomic clock will not even lose a second in 300 million years. A stratum 1 server’s hardware and software typically introduce some imperfections, so in practice the accuracy will be somewhat lower.
Most time consumers use a stratum 2+ server, which typically uses the internet to get time from servers in the strata above it. As a result, NTP packets may be subject to delays and delay variations (jitter), which NTP’s clock synchronisation algorithm can compensate for to provide an accuracy in the order of tens to around a hundred milliseconds.
Microsecond level or more accurate time through the precision time protocol
A popular alternative to NTP is the Precision Time Protocol (PTP, IEEE1588), which provides sub-millisecond accuracy (microsecond level or more). One class of applications that requires such accuracy are critical applications, which increasingly use the internet and its protocols for their communications. For example, smart grids need a microsecond or higher accuracy, highspeed trading systems require microsecond or even nanosecond accuracy, and telecommunication systems such as CDMA require an accuracy of several microseconds. These advanced applications typically also need a high stability, such as 1 nanosecond timing stability, for the navigation of ships into a harbour.
PTP is typically used across layer 2 networks rather than across the internet (layer 3), both in local area networks as well as wide area networks. PTP supports sub-millisecond accuracy because PTP switches are time aware. This means that they can compensate for the delay it takes a protocol message to pass through a switch. Figure 2 shows an example of a PTP packet in which the switch added the delay the packet incurred in the switch, which is 2974 nanoseconds (correctionField).
PTP networks are typically closed, which has the advantage of reducing the risk of attacks (e.g. as with NTP attacks) but the downside is that they are typically unavailable to third parties. Another protocol that can also provide sub-millisecond precision is White Rabbit.
In this blog, we refer to networks that use time-aware protocols like PTP and White Rabbit as time networks.
Sub-millisecond internet time is based on an early-stage infrastructure
While PTP has been a standard since 2002, the infrastructure for sub-millisecond internet time is still in its infancy. It consists of only a few operators of reference clocks and wide-area time networks, each serving a specific region or a specific target group. For example, internet exchange Netnod operates a PTP network in Sweden that provides microsecond precision time across the country. In addition, several time networks are being deployed or being planned for scientific research, such as the National Physical Laboratory’s (NPL)nationwide network for the UK, the sub-nanosecond pilot network of the Dutch research and education network operator SURF, and CLONETS-DS, a pan-European time network commissioned by the EC. Examples of local-area time networks are the microsecond time network that data centre operator Equinix offers its customers, and our network for TimeNL, which uses PTP to synchronize its backup atomic clock with radio-based reference clocks.
To manage the quality of time signals, time networks are increasingly using terrestrial reference clocks because the GPS and other radio-based reference clocks can be jammed, spoofed or otherwise interfered with. For example, security researchers outlined how targeted GPS interference could result in inaccuracies in NTP time stamps by manipulating the “majority vote” procedure that NTP clients use to calculate time on the basis of timestamps they get from multiple upstream NTP servers. There is also anecdotal evidence that NTP servers can crash when an attacker spoofs the GPS signal or jams it. GPS disturbances are a realistic risk because they occur “in the wild”. For example, in January 2022 air traffic around Denver Airport was disrupted for 33 hours because of a rogue GPS radio source.
Examples of time networks that make use of terrestrial clocks are those operated by Netnod and NPL, which provide time through Caesium atomic clocks instead of GPS receivers. Other examples of initiatives aimed at reducing GPS dependencies are the US government’s initiative on Complementary Positioning, Navigation, and Timing (PNT) and GPS Backup, the PNT reference architecture and the industry initiative OpenPNT.
In Europe, reducing the dependency on third-country satellite systems is also important form the perspective of strategic digital autonomy.
The need for an open sub-millisecond internet time infrastructure
The limited number of players in today’s sub-millisecond infrastructure is a problem because it makes it difficult to offer services of that level of accuracy on an internet scale. For example, internet service providers (ISPs) cannot easily get a time signal from multiple providers of time networks. As a result, the sub-millisecond time services they provide might be less robust than those of an NTP infrastructure (see Figure 1) or than their IP connectivity services, for which they typically work with multiple redundant upstream ISPs. In addition, time network operators often also act as providers of reference clocks, which limits the flexibility of the infrastructure because reference clocks can’t be shared across time networks.
We believe an important requirement to solve this problem is the establishment of an open sub-millisecond time infrastructure (“OSub” time infrastructure for short) that makes it easy for new players to provide, distribute and use sub-millisecond time on the internet. To accomplish this, we propose developing (1) a set of best operational practices and open-source software components for operators of time sources, time distribution networks, and internet services that enable them to set up their time services and add them to the OSub time infrastructure and (2) a database called “OSubTime-DB” that enables internet operators such as ISPs to easily discover and connect to time distribution networks, similar to how the well-known PeeringDB helps operators of IP networks deicide how to connect to the internet.
Our vision of an OSub time infrastructure
Figure 3 shows an example of the OSub time infrastructure we envision. It consists of four different roles: reference clock providers (e.g. RCP1 through RCP4 in Figure 3), time network providers (TNP1 and TNP2), PoPs (P1 through P10) and internet service providers (ISP1 and ISP2). Players in an OSub time infrastructure may combine multiple roles, similar to Netnod acting as a time network operator and a reference clock provider for the PTP network in Sweden. While we focus on a public OSub time infrastructure, we also envision infrastructures based on the area they cover (e.g. local, nation-wide, regional) or per target group (e.g. specific sectors such as energy or finance).
Figure 3: An example of an open sub-millisecond internet time infrastructure.
A reference clock provider produces a sub-millisecond-accurate time signal (e.g. accurate to within micro or nanoseconds). Examples are NPL, Equinix, SIDN and the provider of the Netherlands’ official time (VSL). Reference clock providers operate at stratum 0 in the NTP model.
A time network provider runs 1 or more time networks. For example, TNP1 in Figure 3 operates time networks TN1 and TN2. We distinguish tier 1 and tier 2 providers. Tier 1 time network providers (e.g. TNP1 and TNP2) get their time signals from reference clock providers and distribute them to tier 2 time network providers, which serve time consumers. Tier 1 providers typically run wide-area networks, whereas tier 2 providers typically run local area networks, for instance as part of an ISP’s infrastructure.
A point of presence (PoP) is a facility such as a data centre or an internet exchange point that enables reference clock providers and ISPs to connect to time networks. For example, reference clock provider RCP1 connects to time network provider TNP1 at PoP P1, while ISP1 connects to TNP1 at P4.
An internet service provider (ISP) operates IP networks (e.g. IP1 in Figure 3) and connects to the wider internet. In the example in Figure 3, the 2 ISPs (ISP1 and ISP2) also act as tier 2 time network providers and serve distributed applications to remotely managed offshore windfarms. The ISPs connect to tier 1 time providers through demarcation points D1 and D2, which are time-sensitive switches.
A time consumer gets its time from the ISP’s local time-sensitive network through a switch (e.g. D1 in Figure 3) or through NTP stratum 1 time servers (e.g. T1 in Figure 3), depending on the consumer’s time requirements. For example, the remote windfarm operator in Figure 3 uses ISP1’s local area time network (TN3) for sub-millisecond time, while the cell phone uses NTP server T1 or higher-stratum servers to get time with millisecond accuracy.
OSub best operational practices
The best practices we envisage for an OSub time infrastructure pertain to several aspects of engineering and running a time network. They are different from running a typical IP network, because time networks require much tighter control of the network’s delay and jitter.
Three examples of best practices that we have encountered in our work on TimeNL are:
Network design: transporting high-accuracy time signals is more complex than transporting best-effort IP packets and requires special network designs. For example, to transport sub-millisecond time using PTP, all intermediate switches need to be PTP-aware, meaning that they are able to adjust for the delay caused by forwarding the PTP packet in each node’s switching fabric (see the PTP Correction Field in Figure 2). All paths in the time network must be free of “obstacles” that may negatively influence accuracy, such as non-PTP-aware switches or asymmetric paths. Also, PTP is typically deployed over layer-2 connections, rather than over IP (layer 3).
Security design: PTP networks require extra security measures because the protocolwas designed with trusted environments in mind. As a result, the original specification is less suitable for an environment with time consumers that cannot be trusted. For example, if a time consumer makes an operational error, it can unintentionally announce itself as the best clock in the network (the “time transmitter”), through PTP’s best clock selection algorithm, causing all other clocks including the real primary clock to synchronise with it. Protecting against such undesirable scenarios is hard and requires special measures (e.g. low-level filtering of certain PTP packets) or an upgrade of the PTP standard (which is being worked on).
Network monitoring: time networks require tight quality-of-service control, which means they need to be monitored in more detail than typical best-effort IP networks. For example, operators need to monitor for accuracy variations (e.g. using the PTP Trackhound tool from Meinberg), path changes and whether the primary clock is still the right one. Ideally, operators would also monitor the accuracy of the time they provide from a consumer perspective, and would let an external auditor check the accuracy of their time signal against atomic clocks on a regular basis.
In terms of open source, our addition to the best practices currently consists of a secure open-source implementation of a PTP boundary clock in the memory-safe programming language Rust, which is being developed by our partner Tweede Golf, funded by SIDN fonds. Our goal is to address PTP’s usability and security shortcomings, such as PTP not being designed for use in untrusted environments like the network of an ISP.
OSubTime4NL: a first small-scale testbed
To enhance our initial set of operational best practices, we are currently setting up a small-scale OSub time testbed in the Netherlands (“OSubTime4NL”), in collaboration with the Dutch internet exchanges AMS-IX and NL-ix. Our common goal is to learn what it takes to combine the roles of an internet operator, a time network, and a reference clock provider. To accomplish this, AMS-IX and NL-ix will act as PoPs, NL-ix will take on the role of wide-area time network provider, while we at SIDN will act as a reference clock provider using TimeNL.
To carry out the pilot, we moved our TimeNL equipment to a data centre in Arnhem, the Netherlands, where we can connect with NL-ix’s layer 2 network. NL-ix configured part of their infrastructure so that it is PTP-aware, allowing them to deliver sub-millisecond time to their customers.
AMS-IX and NL-ix participate in the pilot because they’d like to better understand how they can serve sub-millisecond time to their customers (ISPs). Our interest at SIDN Labs is to help developing the concept of an OSub time infrastructure, thus enabling future internet applications and to raise awareness of the further increasing importance of time infrastructures.
Connecting the pieces through OSubTime-DB
To set up and grow an OSub time infrastructure like the one in Figure 3, reference clock providers and ISPs need to be able to decide at which PoPs they would like to connect to a time network, analogous to how autonomous systems decide where to connect to the internet using the well-known PeeringDB. For example, ISP1 (Figure 3) needs to select the time networks with which it wants to peer (TN1 and TN2) and at what PoPs (P1, P2 and P3), and what providers of reference clocks (e.g. RCP3) it wants to use across those networks and offer to its customers.
To accomplish this, we introduce the OSubTime-DB, a database that describes the properties of time networks, such as their PoPs and the time signals they provide (e.g. accuracy and reference clock type). As an example, let’s assume that RCP1 wants to make its reference clocks (RC1 and RC2) available in the time infrastructure of Figure 3 through TN1. To do so, RCP1 first queries the OSubTime-DB for available time networks and their PoPs, and selects a subset of them, say PoP P1 of time network TN1. Next, RCP1 hauls a cable to P1 and places equipment at P1 (e.g. a PTP switch if TN1 is PTP-based). The provider of TN1 (TNP1) then updates its entry in OSubTime-DB to show that RC1 is available through TN1 and updates the metadata on its other PoPs to indicate the availability of RCP1. The latter includes the reference clocks that RCP1 uses (RC1 and RC2), their accuracy and other data such as security certifications, RCP1’s jurisdiction and owner.
Similarly, an ISP uses the OSubTime-DB to discover what time networks are available at which PoPs and what types of reference clock providers they support. Based on this data, it decides which networks (e.g. TN1), PoPs (e.g. P2) and reference clock providers (e.g. RCP1) to use and connect to.
OSubTime-DB governance
OSubTime-DB comes with an oversight body, like the governance structure of PeeringDB. We envisage the governance body being lightweight and defining a limited number of policies, for instance focusing on the quality of the data in OSubTime-DB and on acceptable use of OSubTime-DB. To maximise the resilience of OSub time infrastructures, the oversight body should reinforce the autonomy of players and limit policies that reduce the diversity of software, hardware and operational procedures. This is in line with the internet’s key property of decentralised and distributed ownership and control and contributes to the resilience of the OSub time infrastructure.
A governance body may define additional rules, for instance for sector specific OSub time infrastructures (e.g. for health care or energy grids). For example, they may require reference clock providers and time network providers to regularly audit their performance and security, or that reference clock providers in the time infrastructure can only use terrestrial atomic clocks. They can for instance include these requirements in template contracts.
As for its organisation, we envisage the oversight body representing the main stakeholders of a OSub time infrastructure: reference clock providers, time network providers, ISPs, PoPs and user groups such as critical infrastructure operators and governments. This is similar to the steering committees of other joint initiatives such as MANRS (for routing security) and the Dutch National Anti-DDoS Coalition (for collaborative defence against DDoS attacks).
What are we up to next?
At the end of the pilot with AMS-IX and NL-ix, we’ll capture our lessons learnt in a first iteration of a best practices report for OSub time infrastructures. We’ll also be looking to expand our testbed OSubTime4NL with additional reference clock providers, time network providers, PoPs and ISPs. We envisage that it will be a first step towards a public European OSub time infrastructure (“OSubTime4EU”), for instance also involving Netnod’s time network for Sweden as well as others.
As for research, we’re interested in developing and evaluating the OSubTime-DB and perhaps integrating it into PeeringDB. We’re also interested in NTP research, such as to map the types of reference clock of stratum 1 servers in the NTP Pool and other time services and how they evolve over time, for instance based on their (non-standardised) RefID field.
Want to join our initiative or provide feedback? Drop us a line!
If you’re interested in joining OSubTime4NL or if you’d like to share your thoughts about our work, then drop a line to sidnlabs@sidn.nl. We’re also interested in students who are interested in time-related research and would like to do their MSc projects with us.
Acknowledgments
We thank Jan van Boesschoten (AMS-IX) and Jan Paul Dekker (NL-ix) for reviewing the draft of this blog.
