Thesis: A day in the life of NTP: analysis of NTP Pool traffic

SIDN Labs contributes multiple NTP servers to the NTP Pool project. One of those servers is deployed at 30 sites by means of anycast, and serves millions of clients. There has been little research into the characteristics of traffic that is received on a public NTP server. Analysing the traffic received on the anycast NTP server that SIDN contributes to NTP Pool can help by shedding light on the characteristics of the incoming traffic. Useful information that can be obtained includes the types of client that use the NTP service, the catchments of the anycast sites, the presence of anomalies in the NTP traffic, etc. Such information can provide valuable insight into the current state of the NTP ecosystem.

Introduction

The Network Time Protocol (NTP) is used to "synchronize timekeeping among a set of distributed time servers and clients" [RFC 5905] and is the default time synchronisation protocol used on the Internet. It works on top of the Internet Protocol (IP) and User Datagram Protocol (UDP), and is essential to the distributed working of machines in networks.

The NTP architecture is organised as a tree where each level is a stratum (numbered from 1) consisting of servers. The accuracy of a server is related to how high it is in the tree: a server at a higher level (thus having a lower stratum value) is more accurate than a server at a lower level, due to network paths and local clock instability. Servers in Stratum 1 (primary servers) synchronise their time directly from an authoritative source like an atomic clock or GPS; Stratum 2 servers (secondary servers) synchronise time with Stratum 1 servers, and so on. The other side of the NTP ecosystem consists of clients that make use of NTP servers to synchronise their clocks. Clients that use NTP are extremely varied and range from servers and computers to IoT devices and mobile phones.

NTP Pool is a time provider that provides time for an estimated 5 to 15 million clients. NTP Pool lists more than 4,500 servers around the world, and is the default time provider for many vendors including Android, Linux, Asus, etc. Unlike other public time providers, NTP Pool does not host its own servers, but rather lists servers run by volunteers and aims to simplify access to those servers through DNS.

SIDN Labs contributes to the NTP Pool project by providing servers that are extremely popular in the pool and serve millions of clients worldwide. One of these servers, any.time.nl, serves both IPv4 and IPv6 clients and is deployed at 30 sites by means of anycast.

Thus, the aim is to characterise NTP traffic received at one anycast server that is listed in NTP Pool. This is done by collecting data for 24 hours from the NTP server from all 30 locations. The collected data is then centralised, pre-processed to extract relevant fields, enriched with geolocation information, and analysed to obtain valuable insights into the NTP ecosystem.

Dataset

The dataset that was used for this study consists of packet captures from all the 30 anycast NTP servers that are listed in NTP Pool. The servers receive traffic from clients that use NTP Pool as their NTP service provider.

NTP traffic arriving at the servers was captured using TCPDump for a period of 24 hours between 22 June and 23 June, 2022. Relevant IP, UDP and NTP data was extracted from the captured PCAP files and then enriched with client geolocation data including city, country, autonomous system number (ASN) and physical coordinates of client IPs. That information was gathered using the latest available Maxmind geolocation databases at the time of data collection. Subsequently, the IP addresses in the collected files were anonymised using the crypto-Pan algorithm. Anonymisation was in accordance with SIDN's privacy policies and was approved by SIDN's Privacy Board.

In total, about 4.7 TB of data was collected, which contained 13.67 billion NTP queries and responses across the 30 servers. Of those, 7.28 billion were client queries from ~158.7 million unique clients. The majority was IPv4 traffic (97.3%), and the rest was IPv6.

Server utilisation

https://images.ctfassets.net/yj8364fopk6s/2aOqLq4BMc3NBobgWFMBn6/aad7ee9154ae045390460542c1a4029f/Anycast_Site_Utilisation.png

The utilisation of the NTP servers was analysed to gather insights into which server/anycast site was particularly popular in the NTP ecosystem. This analysis gives an idea about the underlying client distribution on the internet. To determine the usage of NTP servers, the aggregated data was partitioned into queries received at each individual server.

It is apparent that the anycast sites in Asia are particularly popular on NTP Pool. The servers {bom1-1, bom1-2} combine to serve more than 1.59 billion client queries from 28.4 million clients. This constitutes for about 22% of the total traffic received on any.time.nl. Other servers in Asia also receive considerably more than traffic than their counterparts in North America and Europe as evidenced by the number of queries received at nrt1-{1-8}, icn1-1, sin1-{1-2}.

The NTP server sea1-1 stands out as one of the most utilised servers in North America with similar traffic received as bom1-{1-2}. Out of all the 7 North American sites, sea1-1 constitutes 49.3% of the traffic and 10.5% of total traffic among all 30 sites. A deeper look at the geographic distribution of the clients that use this server reveals that most of the traffic that this server receives is from China. In the observed period, sea1-1 received more than 5 times the amount of traffic from China than from United States and Canada.

https://images.ctfassets.net/yj8364fopk6s/3LfPXJvEP8VifhCo7hbnXW/7fc059d4b87ef14e1bdd460952ae5048/Number_of_Queries_per_Hour.png

Server utilisation was also analysed by looking at the number of queries each anycast server receives per hour for the duration of the packet capture. The above picture shows the number of queries per hour at each NTP server. The Y-axis is on a logarithmic scale for better representation of the difference in query numbers between the various servers.

At the outset, there are some patterns visible especially with respect to the Asian NTP servers. A clear dip in the number of queries is observed during night-time hours. That implies that the number of clients sending NTP queries is lower at night than in the daytime. On the other hand, traffic at the European and American sites stays mostly constant throughout the day. That observation is in line with previous studies into internet day-night traffic patterns.

Anycast catchments

Another important aspect of anycast deployments is the catchments of anycast sites. The Operation of Anycast Services RFC 4786 defines an anycast catchment as "the topological region of a network within which packets directed at an Anycast Address are routed to one particular node". Therefore, studying the catchments of a deployment gives an idea of the underlying routing behaviour on the internet, and is helpful for fine tuning the anycast deployment to serve clients in an efficient manner.

The latest catchments of anycast sites can be found here: http://dnstest.nl/anycast2020/heatmaps/

Anycast catchments are not static, and the catchment of each site can be influenced in various ways. In fact, catchments of some sites have already been improved to receive traffic only from the country.

NTP client characteristics

Figure 3: Number of queries per country.

China has the greatest number of clients (28 million) with Brazil, India, the US and South Korea close behind. Clients in China generate a staggering 3.5 billion queries, which amounts to an average of 126 queries per client. While China has only 8 million clients more than Brazil, those clients generate 7.8 times more queries than clients in Brazil. On the other hand, Brazil has 2.2 million clients more than India but clients in India still generate 1.3 times more queries than clients in Brazil. In general, Brazil has a low average number of queries per client when compared with the other top countries. That implies that, while Brazil has the second largest number of clients, the clients do not generate as many queries as clients from other countries. Japan also has a high density of queries generated by a relatively small number of clients. Of the top 10 countries, Japan has the second highest average number of queries per client: close to 50, which is behind only China.

The table above shows the distribution of NTP versions across clients, including the percentage of IPv4 and IPv6 queries and the number of clients for each NTP version. As expected, NTP version 4 is the version most widely used by clients: 81% of total queries are version 4. The NTPv4 RFC was proposed in 2010 and was adopted in 2011. The newest version of NTP also addresses a lot of security flaws that were present in previous versions of the protocol. In a similar study of NTP traffic characterisation conducted a few years back, it was observed that clients in Asia predominantly used NTPv3 (68%). But that number has fallen: in our dataset, only 17% of clients still used NTPv3. 82% of the clients in Asia now use NTPv4, and the percentages for other continents are comparable. Therefore, the overall trend in NTPv4 adoption is upward, which is desirable, as it implies that clients are using the latest NTP version.

Further, more than 7000 clients sent more than 100k queries with invalid NTP version numbers. They included NTP queries with version numbers 0, 5, 6 and 7. Most clients using invalid NTP version numbers seem to be located in the US and Canada, with more than 48k queries from US and 16k queries from Canada. The use of such NTP version numbers contravenes the protocol specification. There were also close to 40k clients that sent NTP packets with no NTP version specified or a malformed packet from which the NTP version could not be extracted. Such queries are represented in the table above as NTP version 'NULL'. More than 1 million such queries were received across all NTP servers. The bulk of the traffic with NTP version NULL was from Brazil, India, US, Argentina and Spain. Those 5 countries combined to produce more than 900k of the 1 million queries. Thus, the NTP version data shows that, while most clients conform to protocol specifications, there is still widespread use of older NTP versions (especially NTPv3).

Passive fingerprinting using the IP time-to-live (TTL) values in client NTP packets was performed to gain a better understanding of the client OSs and NTP software stacks that clients use. While the information obtained might not be accurate in all cases, it gives a rudimentary understanding of client characteristics. The table below shows the distribution of observed client IPv4 TTL values. As expected, most of the clientele (89%) are Linux-based, where the OS has a default TTL value of 64. That is because devices like routers, firewalls, Android phones, IoT devices, etc. are all based on the Linux kernel and likely form the majority of NTP clients of NTP Pool. It is to be noted that a major operating system like Windows has its own NTP service that is the default provider for all windows-based devices, which accounts for the paucity of Windows devices in this dataset.

The Great NTP Flood

It is usually observed by NTP server operators that NTP clients send more queries than they should to servers. That suggests that clients do not respect the value in the poll interval or KoD packets. To dig deeper into that phenomenon and identify rogue clients, analysis was done into how many queries each client IP sends and the time duration of client IP visibility. The table above shows the top 10 IPs in terms of the number of queries sent and the duration (in seconds) of the IP's visibility. The IP addresses are all anonymised as described earlier. The top IP sent a total of 8.7 million queries throughout the entire day of the capture, with an average number of queries per second close to 100. That is a lot higher than the lowest NTP minpoll setting of 4, which translates to 1 query per 16 seconds. The actual number of queries received from the top IP translates to 1600x the number of queries that the client can send. That clearly constitutes anomalous client behaviour. One possible explanation for the excessive number of queries could be that the client IPs belong to ISPs that use carrier-grade NAT (CGNAT). Use of CGNAT is especially common on continents like Asia, where the high number of users force ISPs to use CGNAT to avoid running out of public IPv4 addresses. CGNAT is a technique where many users are fronted by a single public IP. In this case, there could potentially be thousands of clients behind a single IP, which are sending NTP requests. Looking at the poll interval that the servers set for responses to those IPs, 4 is the most commonly used value, with the NTP servers even setting a poll interval of 17 for some IPs. If an IP gets a response with a poll interval of 17, it must not send any further NTP queries for a minimum of 1.5 days. But it is clear that client IPs do not respect the poll interval values set in the responses, as evidenced by the number of queries sent to NTP servers. Where CGNAT or another NAT technique is in use, it might not be possible for the NAT device to honour the poll interval, as it merely forwards requests from clients located behind it. Nevertheless, it is evident that clients send excessive numbers of queries to NTP servers, even when a maximum poll interval is set in the response sent to the client. That implies that clients are not complying with RFC in terms of sending appropriate numbers of queries.

It is important to note that the IPs themselves are anonymised but geolocation and ASN information was added to the dataset before anonymisation was performed. The top IP belongs to the LG Dacom corporation in South Korea. The Whois info for this ASN shows that it belongs to one of the largest telecom network operators in Korea. Hence, it is reasonable to conclude that this IP is probably a public NAT IP that is used by a huge number of mobile clients to synchronise time. Similarly, the second IP belongs to the ASN of an ISP in the Philippines. Therefore, this IP also is the NAT IP of the ISP's users. Of the top 10 IPs that sent the most queries, 9 of them are in Asia, including 5 in South Korea. The outlier is an IP in South Africa. The ASN of that IP was identified and subsequently the company's website was identified using Whoisinformation. The South African IP belongs to a popular mobile network operator like the first IP. The third and tenth IPs, both of which are South Korean, belong to another network operator, Korean Telecommunications Authority (KT), while the fourth IP (from Japan) belongs to NTT. In fact, all the IPs in the top 10 most talkative list belong to ISPs and mobile network operators. Therefore, it is safe to assume that all those IPs are not actual clients, but NAT IPs that have many thousands of clients behind them.

Malicious NTP clients

NTP has been used in multiple amplification and reflection DDoS attacks. That is because NTP, like DNS, is mainly a UDP-based protocol, which allows attackers to spoof queries. Also, NTP has certain queries like monlist, peerlist, etc. that return responses whose packet sizes are much larger than the requests. That allows attackers to craft small spoofed requests with the IP address of a victim, and send the requests to public NTP servers. The public servers then send large responses to the victim, thus inundating it with large amounts of unsolicited traffic, which constitute a DDoS or DoS attack.

The monlist and peerlist queries are administrative queries built into the protocol for server administrators to troubleshoot or gather diagnostic information from their NTP servers. In contrast to the client-server model, where a client sends a packet with NTP mode 3, such queries use NTP mode 7, which the RFC describes as "reserved for private use". Hence, NTP mode 7 is not really intended for clients to use when interacting with a public server. However, when NTP-based DDoS attacks were at their peak, there were hundreds of thousands of public NTP servers that supported public mode 7 operation. Over time, server administrators began to disable mode 7 for the public clients, so that their servers were not vulnerable. Nevertheless, there are still vulnerable NTP servers and malicious actors that look to use such servers in various attacks.

Since the problematic queries use a distinct NTP operating mode (7), it is quite easy to identify clients that send such queries to any.time.nl. In the collected dataset, there were a total of 8136 queries sent by 1021 clients across the 30 servers. The table above shows the 10 IPs that sent the most mode-7 queries, along with the anycast site that the IPs reached, and the country that the IP is from. While the IP 88.162.105.2 (anonymised) sent almost 1000 queries to the mia1-1 site, further analysis of the queries shows that the IP also sent 3 to 5 queries to each of the other 29 sites. That is not normal behaviour: once a client has been routed to a particular site, further queries should be sent to that site. There are of course some exceptions, where clients can send queries to other sites in the same country, as with sin1-1 and sin1-2. At first sight, such behaviour looks suspicious and typical of a scanner. Analysis on the temporal distribution of mode-7 queries from the IPs in question shows that the IPs send mode-7 queries in distinct bursts. Some IPs, like 195.158.90.138, are active only once during the entire day, while others, like 195.158.75.168, send 2 bursts of queries in the 24-hour period. Further, in contrast to a normal client traffic pattern, there are periods when the suspicious IPs do not send any requests. That, combined with the fact that the reputation scores for the IPs' ASN show a high level of spam activity, suggests that the IPs in question belong to scanners that search the IPv4 space for vulnerable NTP servers.

Conclusion

Since NTP is a core protocol on the internet, like DNS, and is critical to the functioning of devices, the data collected in my study is valuable insofar as it enables in-depth analysis of the state of the current NTP ecosystem. Extensive analysis of the data reveals that NTP Pool provides a crucial service to users but is in dire need of more servers to handle NTP traffic outside North America and Europe, especially in Asia and Africa. The anycast server provided by SIDN represents a vital contribution to addressing the scarcity of servers. Other similar initiatives are urgently needed. The anycast deployment proves to be extremely capable of handling the demand from clients in the current NTP ecosystem, especially since it covers locations that do not have many other NTP server options. My results also show that, while adoption of the latest NTP version has increased compared to previous years, a significant portion of clients still use outdated NTP versions. There is a lot of RFC non-compliant behaviour among clients, involving the setting of reserved or incorrect values in NTP packets. Furthermore, the architectural designs used by many ISPs and network providers actively hamper NTP usage by flooding servers with excessive NTP queries. Another problem is that, with the current architecture, in-built server rate limiting is not effective and detracts from the client's experience of using the protocol. Lastly, traces of malicious traffic were found in the dataset and analysed, leading to the conclusion that NTP is still a viable attack vector, and server administrators must take steps to protect their deployments.

Further reading

My full thesis is available on the SIDN Labs website. It contains much more detailed analysis of the observations, and considers various other points. Interesting previous research in the field of NTP traffic characterisation includes Masters of Time: An Overview of the NTP Ecosystem by Rytilahti et.al.