All good things take time

A core internet protocol

The Network Time Protocol (NTP) is the most widely used protocol for synchronising the clocks on computer systems. It's also used in lots of less obvious places, such as smart lighting systems. NTP is therefore one of the core protocols underpinning the internet, along with DNS (Domain Name System) and IP (Internet Protocol versions 4 and 6).

However, NTP differs from those protocols in one important respect. Whereas the use of (authoritative) DNS and (distribution of) IP are centrally managed through clear structures and organisations such as ICANN, RIPE and SIDN, there's no formal structure for NTP. The protocol is maintained within the IETF, but there are few central agreements covering the provision or use of NTP services. That's perhaps surprising, given that NTP is an indispensable part of the modern world and that someone has to provide the (public) NTP services that are so essential. There are indeed people willing to do so, but that is perhaps down more to good luck than good judgement. There are, of course, reasons for the situation described. DNS and IP are concerned with unique identifiers and therefore require careful coordination in order to work properly. That's not the case with NTP. Moreover, if NTP services are temporarily unavailable, the consequences are not normally disastrous in the short term.

It's nevertheless interesting that a service on which so much depends, and which is so popular and successful, is so reliant on volunteer input. So, how are things done? And what are the implications?

How NTP services are provided

As previously indicated, almost every computer system benefits from knowing the right time. And, for any system connected to the internet, NTP is the normal way to establish the right time. Where an extremely high degree of precision is required, there are alternative solutions. For the vast majority of systems, however, NTP – which can support synchronisation with a precision of milliseconds – is entirely adequate.

NTP is a robust protocol that has been around nearly as long as DNS. Before it was introduced, most systems relied on built-in clock chips, whose time was liable to drift considerably over an extended period. Gradually, however, the need for a better way of synchronising time grew, and NTP was the response. Soon after NTP's introduction, several universities and specialist companies, as well as the US National Institute of Standards and Technology (NIST, since 1993) and the United States Naval Observatory (USNO) began offering public internet-based NTP services. Their involvement served as a catalyst for the use of NTP. In the early days, most of the NTP service providers were US-based organisations with a strong operational interest in accurate time measurement. Later, organisations of other types and in other countries also got involved. In the Netherlands, for example, SURFnet and the metrological institute VSL have started offering public NTP services, albeit without much fanfare. Similar developments are underway elsewhere. The service providers mentioned above therefore represent anything but a comprehensive list. Since 2019, SIDN Labs has entered the picture as well. That was when, as big fans of the internet's foundations and building blocks, we launched our own public NTP service called TimeNL.

Around the turn of the century, there was explosive growth in the number of systems seeking to use public NTP servers. The growth was so rapid that, in about 2003, various organisations started to have problems maintaining the stability of their public NTP services and therefore withdrew them. That led to the proposal of a new approach, involving a global body of NTP servers, to which anyone that wished could voluntarily contribute a server. The resulting NTP pool has since proved very successful and become very well known. It now includes more than 4,300 servers, all operated on a voluntary basis without any commercial motive. Meanwhile, with falling hardware prices and the arrival of compact GPS receivers, a market developed for NTP servers that companies could instal in their own networks. The availability of the servers also enabled countless hobbyists to set up their own servers and, if they wished, to offer public NTP services (independently or as part of the NTP pool). Some ISPs now operate NTP servers as well.

Developments didn't stop there, however. Tech giants such as Apple, Google, Microsoft and, more recently, Cloudflare and Facebook all now offer public NTP services – generally for use by their own products. Given how many products the companies in question sell, it will come as no surprise that the tech giants' NTP services are based on very substantial infrastructures. There is nowadays also a sizeable market for devices of many different kinds, but typically cheap embedded devices such as IP cameras, which have no built-in, battery-powered clock, and are therefore entirely reliant on NTP. Many such devices are configured by their manufacturers to get their time from the NTP pool.

The consequences of those various developments are huge growth in the demand for NTP services and diffuse wild proliferation of public NTP services, of variable quality.

Don't take NTP for granted

The good news is that the potpourri of available NTP services has at least been able to keep most things working. For that, we mainly have the NTP pool operators to thank. The service they provide is perfectly adequate for the great majority of everyday applications, and it is not without reason that millions of devices use the pool. For more critical environments, however, a more considered approach is advisable.

Every professional environment should define an NTP policy based on careful planning. Because the choice of service providers is extensive. And anyone looking to optimise the security of their infrastructure does need to take their NTP set-up into account. The reason being that the manipulation of time (or its synchronisation) is a potential attack vector in certain scenarios, with potentially serious implications.

Against that background, it's inadvisable to rely on a single provider. The NTP pool is a good example. Although the pool works well and meets a need, its continuity is not assured, because it's over-reliant on a single individual with limited time (Ask Bjørn Hansen). Where critical environments are concerned, reliance on the NTP pool therefore represents a risk, even before one considers the inconsistent quality of the service. The precision of the NTP signal varies, depending on which of the 4,300-plus servers you are connected to: a lot of them are domestic set-ups reliant on ADSL or cable.

However, many other public NTP service providers are far from transparent about their service levels. Since few offer clarity, we are left to assume that they operate on a 'best effort' basis, without any guarantee. When we investigated the situation, for example, we came across numerous NTP servers, including some operated by well-respected parties, running outdated firmware versions. It is therefore certainly worthwhile looking into the reputation of a service provider you are considering using. Fortunately, there are a few reliable players, including Sweden's Netnod, our own TimeNL, respected organisations such as national metrological institutes, and professionals for whom time is vitally important. Even if one uses such a service, things can still go wrong, however.

The best approach is therefore to reduce reliance on any single service provider. However, there is more to spreading risk that one might at first imagine. For example, many public NTP services use the US GPS system as their reference clock. Consequently, a set-up may still be vulnerable to a single point of failure even when configured to use multiple NTP services. When selecting NTP servers, it is therefore advantageous to opt for a pallet of servers that includes servers whose reference clocks are atomic clocks or the European Galileo system. The NTP protocol is then smart enough to identify and exclude any errant time sources. Unfortunately, however, it often remains difficult to ascertain the information needed to make informed choices, due to lack of transparency on the part of service providers. Lack of transparency is also a problem in relation to the practice of 'leap smearing'. It's undesirable to a use mix of servers that includes some that do leap smearing and some that don't, but many providers simply don't say what their policy is.

NTP responses can be spoofed, or manipulated and falsified. With that in mind, it's worth considering authenticated NTP where available, although one should be careful to use the good variants, not 'autokey'. The latest development in this field is Network Time Security (NTS), which is certainly worth investigating. Moreover, if you've bought your own NTP server and it relies on GPS, it's important to be aware that GPS signals can also be jammed or spoofed. So don't rely exclusively on such a server; build in redundancy if time synchronisation is vital for your activities.

Anyone whose firewall allows all their devices to interact directly with NTP servers over the internet should also be aware that such a set-up is vulnerable to 'data exfiltration'. It may be better to configure your devices to communicate with internal servers, and to permit only the internal servers to communicate with specified NTP servers on the internet. NTP does support that kind of tiered configuration (see Figure 1). Another advantage of that approach is that it reduces the load on public NTP services, especially if your local network has a lot of users. All internal systems then synchronise with the internal servers, and only those servers communicate with the public servers on the internet:

You're likely to encounter further difficulties if you want the ability to synchronise time not only using IPv4, but also using IPv6: many NTP services, including Apple's, don't currently support IPv6.

Problems also lie in wait for anyone who doesn't want to send data (including NTP queries) to big tech companies (Google, Facebook, Apple, Microsoft, Cloudflare, etc). In that context, it's worth noting, for example, that Apple's NTP server is hard-coded on all iPhones. So your queries are going to Apple whether you like it or not. Other manufacturers no doubt have similar set-ups.

All the issues covered above, and more, are considered in RFC8633, NTP Best Current Practices.

Where TimeNL fits in

At SIDN Labs, we've been thinking about the importance of NTP for some time. That's why we set up TimeNL: a public NTP service built on the principle of transparency, so that users know exactly what they can and can't expect.

For a while now, we've also been running an NTS pilot. NTS is Network Time Security, a recent security extension to the NTP protocol, which resolves the shortcomings of previous efforts in this field.

We've made both of our NTP services available to the NTP pool, to which we've added a global BGP-anycast-based service. TimeNL has proved a big success: our twenty-eight globally distributed servers now process about 100,000 NTP queries a second. Another indicator of TimeNL's success is that its users include Cloudflare (for its own NTP service), the NLNOG-ring (to support its own NTP hardware) and the Amsterdam Public Broadcasting Corporation (AT5). And, sooner or later, systems on your own network may be asking one of our servers for the time, if they are configured to use the NTP pool.

Looking to the future

Encouraged by the success achieved so far, we're planning to further extend and upgrade our NTP services this year. For example, we want to increase our capacity to handle the anticipated demand growth and create more redundancy. We're intending to invest in additional security for our NTP service as well. Also, although the risk of the signal to our NTP server being manipulated is quite small, we're looking at mitigating the consequences by adding a reliable, external 'holdover clock'. That's an extremely accurate rubidium clock, which we would connect to our time server, enabling us to continue providing a very accurate service for a prolonged period, even if all external reference clocks are taken off line or no longer provide a reliable time signal.

In short, we intend to help increase the resilience of the public NTP infrastructure, whose importance cannot be stressed enough, for all the reasons explained above.