DNS TTL violations in the wild with Ripe Atlas

Introduction

The time-to-live value of a DNS record "is primarily used by resolvers when they cache RRs. The TTL describes how long a RR can be cached before it should be discarded" [1]. In other words, it is the maximum length of time (in seconds) that a DNS resolver should keep a domain in its cache.

A TTL violation involves a recursive resolver overriding the time-to-live value of a DNS record, as provided by the authoritative server. For example, as documented in [2], Amazon EC2 local resolvers override the TTL of .nl, changing it from 172800 to 60. Other researchers have previously reported on use of the practice on wired and mobile networks [3 and 4, respectively]. 

When a resolver violates the TTL specified by a DNS zone, one of two things can happen: if the involves reducing the specified value (e.g. from 2 days to 60 seconds), that will make the resolver query the authoritative server more often. If the TTL value is increased, that will make the domain reachable at an address that could in principle be wrong or illegitimate (for example, if a phishing website is taken down, it may still be available in the caches of a resolver that has increased its TTL).

The violation of TTLs divides opinion: some people see it as a legitimate practice, while others are against it [2,5,6]. There is also an internet draft that presents a method (that's currently being used by many cloud providers) to slate DNS query data when authoritative servers are unreachable [7].

In this post, we do not debate whether resolvers should violate TTL values provided by authoritative servers (please refer to [2,3,4,5,6] for that). The question we consider is: are TTL violations happening in the wild? TTL violations have been reported in other studies (e.g. involving wireless providers [4]), but the number of providers involved in the reported cases has been small.

To answer the question set out above, we have analysed data from the Ripe Atlas probes.

Measurements

We measured TTL violations in the wild using the following procedure:

1. Register a non-used domain name (cachetest.nl).

2. Set up two authoritative name severs for cachetest.nl:

   1. cachetest.nl

   2. cachetest.nl

3. Set up the zone files for each NS, using Ripe Atlas probe IDs as subdomains (so we can use macros to send unique queries from each probe to avoid caching -- i.e., $p.cachetest.nl, where $p is probeid):

   23559        333    IN      TXT   "this is ns1 responding to probe 23559"

   23560        333    IN      TXT   "this is ns1 responding to probe 23560"

   23561        333    IN      TXT   "this is ns1 responding to probe 23561" 

   23562        333    IN      TXT   "this is ns1 responding to probe 23562"

4. Run Atlas measurements with 10,000 Atlas probes:

   1. https://atlas.ripe.net/measurements/10341665/

5. Parse and analyse the results.

As will be apparent from step 3, we made sure each probe query involved a unique domain name, so that a resolver cache-miss situation is guaranteed, even if the same resolver was queried. In other words, every query was designed to make the resolver query one of our authoritative servers.

Dataset

After running the measurement for 1 hour [8], querying every 600s (almost twice the value of TTL of the records in our zone), we generated the final dataset shown in Table 1. As can be seen, 9,119 probes were involved in this measurement, querying more than 6,687 resolvers.

Since each probe can contact multiple resolvers, we see that, in the end, there are 15,923 vantage points, i.e. unique probe-resolver combinations.

Our 54,115 queries led to more than 94,805 answers, which we used in our analysis described below.

Table 1: Complete dataset

Analysis

Given that we set the TTL for every record in our demo zone to 333, the question is: how many resolvers changed that TTL value? And what is the typical change, if any?The expected value for the TTL of queries is 333. However, since multiple probes can use the same resolver, we may expect some TTLs to be slightly less than 333. However, no queries should have a TTL of more than 333.

We divided the dataset from Table 1 into three parts:

• Normal TTL: answers based on 320<=TTL<=333

• Decreased TTL: answer based on TTL < 320

• Increased TTL: answer based on TTL >333

Table 2 shows the results. As can be seen, the great majority of probes/queries/resolvers fall into the normal category, meaning their TTL deviates from the original 333 by no more than 13 (since multiple probes can use a same resolver). In the following paragraphs, we consider first the decreased TTL answers and then the increased TTL answers, in an effort to understand how much the values are being changed by the resolvers in question, and why.

Table 2: Breakdown of results

Decreased TTL answers

As shown in Table 2, 0.77% of all the valid answers in this measurement were based on decreased TTLs. Figure 1 shows the results. As can be seen, two types of resolver dominate: those that cap the TTL at around 50s, and those that cap it at around 250-300s.

Out of 130 resolvers that reduced the TTLs, 71 reduced them to less than 50. Many of those, however, were local resolvers using private address space ranges. Out of the 71, 24 were not that kind of resolver, but belonged to networks run by mobile operators and other research institutes.

Out of the 130 resolvers, 16 (non-private addresses) reduced the TTL from 333 to between 250 and 320. No particular pattern was found here -- several operators from various countries were doing the same thing. We also found cases where Google quad8 resolvers reduced the value, but that is an outlier, given the large volume of instances of their infrastructure.

Figure 1: Histogram of TTL values for the decreased TTL group (Table2)

Increased TTL answers

As shown in Table 2, 4.17% of the resolvers actually increased the TTL values of our RRs.

Figure 2: ECDF of TTL values for answers with TTLs of more than 333 (the set value). The practice of increasing TTLs is particularly worrying, since it means that the resolvers in question will send their clients RRs that may have already been expired in the corresponding zones.

Figure 2: ECDF of TTL values that were increased

Looking at the IP addresses of the resolvers that increased our TTLs, we see they are associated with certain ISPs and other cloud providers. The discussion in [2] and the associated feedback revealed various reasons why a cloud provider would reduce the TTL of a RR, but to increase the TTL will only reduce traffic to the authoritatives, and may put users at risk of being served with expired answers.

Wrap-up

The issue of DNS TTL violations is controversial, generating passionate arguments on both sides [2,5,6]. It is publicly known that some cloud providers and CDNs violate TTLs within their networks, overriding the original values provided by authoritative servers.

In this article, we report on the use of Ripe Atlas to ascertain whether TTL violation is happening in the wild. Although a small number of resolvers are known to violate TTLs, it is unclear how many users are affected.

A case can be made for reducing TTL values in certain circumstances, but the practice will ultimately lead to the resolver querying an authoritative more often. Therefore users should always be provided with the correct RR whenever the TTL value is reduced.

Increasing TTLs, on the other hand, may be dangerous to users, since they may be served with records that have expired. Consider, for example, the case of a domain that has been removed from a zone due to a phishing or malware attack: by extending the domain's TTL, a resolver will keep the domain alive for any client that looks up the domain during the extension.

References

[1] https://www.ietf.org/rfc/rfc1034.txt [2]: https://lists.dns-oarc.net/pipermail/dns-operations/2017-November/017039.html [3]: http://www.cs.wm.edu/~haos/papers/sigcomm-ccr-dns.pdf [4]: https://doi.org/10.1145/3143361.3143375 [5]: https://mailarchive.ietf.org/arch/msg/dnsop/zRuuXkwmklMHFvl_Qqzn2N0SOGY/?qid=ff8e732c964b76fed3bbf333b89b111f[6]: https://queue.acm.org/detail.cfm?id=2578510 [7]: https://datatracker.ietf.org/doc/draft-ietf-dnsop-serve-stale/?include_text=1 [8]: https://atlas.ripe.net/measurements/10341665/

Appendix: probes with increased TTL values

Format: ProbeID-ResolverIP

22372-fd00:10:246:200::1,22480-fd69:b46:ca9b::1,22891-fd48:71b9:382f::1,31959-

fd00::201:2eff:fe6e:63b8,10181-2a01:e00::1,

10181-2a01:e00::2,10816-2a01:e00::2,11243-2a01:e00::2,11828-2a01:e00::1,

11828-2a01:e00::2,12360-2a01:e00::1,12360-2a01:e00::2,13209-2a01:e00::1,

13209-2a01:e00::2,13317-2a01:e00::2,13407-2a01:e00::1,13407-2a01:e00::2,

13439-2a01:e00::1,14311-2a01:e00::1,14311-2a01:e00::2,14342-2a01:e00::1,

15973-2a01:e00::1,15973-2a01:e00::2,16007-2a01:e00::2,16047-2a01:e00::2,

16052-2a01:e00::1,16052-2a01:e00::2,16146-2a01:e00::1,16146-2a01:e00::2,

16644-2a01:e00::1,16644-2a01:e00::2,16703-2a01:e00::1,16703-2a01:e00::2,16872-

2a01:e00::2,16973-2a01:e00::1,16973-2a01:e00::2,16978-2a01:e00::1,16978-2a01:e00

::2,16998-2a01:e00::1,16998-2a01:e00::2,17025-2a01:e00::1,17025-2a01:e00::2,

17035-2a01:e00::1,17035-2a01:e00::2,17064-2a01:e00::1,17064-2a01:e00::2,17127-

2a01:e00::2,17147-2a01:e00::1,17147-2a01:e00::2,17153-2a01:e00::1,17153-2a01:e00

::2,17214-2a01:e00::2,17228-2a01:e00::1,17228-2a01:e00::2,17263-2a01:e00::1,

17263-2a01:e00::2,17848-2a01:e00::1,17848-2a01:e00::2,18429-2a01:e00::1,18710-

2a01:e00::2,19032-2a01:e00::1,19671-2a01:e00::2,20226-2a01:e00::1,20226-2a01:e00

::2,20480-2a01:e00::2,21173-8.8.4.4,21173-8.8.8.8,2139-62.141.32.3,21507-2a01:

e00::1,21597-2a01:e00::1,21990-2a01:e00::1,21990-2a01:e00::2,21991-2a01:e00::2,

22171-2a01:e00::1,22171-2a01:e00::2,22210-2a01:e00::1,22210-2a01:e00::2,22892-

2a01:e00::1,22892-2a01:e00::2,22963-2a01:e00::1,22963-2a01:e00::2,23289-2a01:e00

::1,23289-2a01:e00::2,23294-2a01:e00::1,24336-2a01:e00::1,24336-2a01:e00::2,

24520-2a01:e00::1,2576-83.219.128.10,26095-2a01:e00::1,26095-2a01:e00::2,27263-

2a01:e00::2,27607-2a01:e00::1,27607-2a01:e00::2,27622-2a01:e00::2,2862-92.244.96

.117,28889-2a01:e00::1,29425-2a01:e00::1,29425-2a01:e00::2,29622-2a01:e00::1,

29622-2a01:e00::2,29772-2a01:e00::1,30807-2a01:e00::1,30807-2a01:e00::2,30920-

2a01:e00::1,30920-2a01:e00::2,30922-2a01:e00::1,30922-2a01:e00::2,31148-2a01:e00

::1,31517-2a01:e00::1,31517-2a01:e00::2,31598-2a01:e00::1,31598-2a01:e00::2,

31749-2a01:e00::1,31749-2a01:e00::2,31770-2a01:e00::2,31853-2a01:e00::1,31853-

2a01:e00::2,31879-2a01:e00::1,31879-2a01:e00::2,31967-2a01:e00::1,31967-2a01:e00

::2,32016-2a01:e00::1,32016-2a01:e00::2,32057-2a01:e00::2,32070-2a01:e00::1,

32140-2a01:e00::1,32140-2a01:e00::2,32320-2a01:e00::1,32419-2a01:e00::1,32606-

2a01:e00::2,32647-2a01:e00::1,32647-2a01:e00::2,33202-2a01:e00::1,33202-2a01:e00

::2,33448-2a01:e00::1,33520-2a01:e00::1,33520-2a01:e00::2,33521-2a01:e00::1,

33521-2a01:e00::2,33924-2a01:e00::1,33924-2a01:e00::2,11729-10.10.16.1,12099-10.

2.2.1,12576-10.1.0.1,13195-10.237.138.137,13972-81.3.3.81,14387-83.172.40.200,

14387-83.172.41.200,17032-10.0.0.3,17858-62.141.32.3,2108-192.168.0.2,2160-192.

168.100.1,2181-192.168.1.254,22017-10.0.0.99,22372-10.246.200.1,23342-10.0.8.240

,23410-10.0.61.1,23796-84.54.64.34,24509-87.72.130.2,24509-87.72.22.66,2486-192.

168.1.254,2511-192.168.72.1,2553-192.168.0.1,2571-212.27.40.240,2571-212.27.40.

241,25727-80.250.1.155,25727-80.250.1.161,2610-109.110.160.172,2623-192.168.1.

254,2656-192.168.1.1,2684-192.168.1.1,2755-212.27.40.240,2755-212.27.40.241,

28186-10.200.99.1,28741-10.80.32.13,31502-46.175.231.193,31847-10.0.0.254,31918-

10.0.0.1,31985-85.21.192.3,32186-10.0.0.1,32317-10.67.71.2,3282-192.168.0.250,

32866-89.47.39.13,33382-88.158.158.3,33950-10.0.0.254,3520-192.168.0.254,3528-

200.3.169.65,3528-200.3.169.66,3588-209.193.0.2,3588-216.67.0.2,3657-192.168.1.1

,3763-192.168.1.1,3782-192.168.77.1,3825-192.168.1.1,3867-192.168.1.1,3869-192.

168.1.1,3880-192.168.0.1,3887-192.168.70.1,4045-192.168.1.254,4076-212.27.40.240

,4076-212.27.40.241,4108-212.27.40.240,4108-212.27.40.241,4264-192.168.0.254,

4266-192.168.0.254,4308-192.168.11.253,4372-192.168.88.1,4383-192.168.0.1,4889-

212.1.224.6,4889-212.1.244.6,4896-192.168.1.1,10673-212.27.40.240,10673-212.27.

40.241,10717-192.168.1.1,10718-192.168.0.1,10816-212.27.40.240,10816-212.27.40.

241,11022-192.168.0.254,11243-212.27.40.240,11243-212.27.40.241,11301-213.170.64

.33,11437-192.168.88.1,11437-195.239.225.93,11437-195.239.225.94,11531-212.27.40

.240,11531-212.27.40.241,11716-192.168.68.1,11828-192.168.0.254,11992-192.168.

100.1,12066-192.168.1.1,12099-192.168.0.1,12357-172.20.1.1,12360-192.168.0.254,

12443-192.168.2.1,12563-192.168.0.2,12584-195.245.120.135,12800-212.27.40.240,

12800-212.27.40.241,12873-192.168.1.1,13209-192.168.0.254,13307-192.168.1.254,

13310-212.27.40.240,13310-212.27.40.241,13393-212.27.40.240,13393-212.27.40.241,

13407-192.168.1.254,13439-192.168.22.254,13605-192.168.1.254,13807-101.100.188.

23,13807-103.7.200.10,14311-192.168.0.254,14772-192.168.1.1,14776-192.168.21.254

,15535-194.225.152.10,15973-192.168.0.254,16007-212.27.40.240,16007-212.27.40.

241,16020-192.168.30.1,16047-212.27.40.240,16047-212.27.40.241,16052-192.168.2.

254,16146-192.168.208.254,16644-192.168.0.254,16645-192.168.0.254,16670-212.27.

40.240,16670-212.27.40.241,16703-192.168.0.254,16818-192.168.0.254,16869-212.27.

40.240,16869-212.27.40.241,16872-212.27.40.240,16872-212.27.40.241,16973-192.168

.0.254,16978-192.168.0.254,16981-192.168.0.254,16998-192.168.0.254,17025-192.168

.0.254,17030-192.168.1.254,17032-212.27.53.252,17032-212.27.54.252,17035-192.168

.0.254,17064-192.168.0.254,17072-212.27.40.240,17072-212.27.40.241,17078-212.27.

40.240,17078-212.27.40.241,17106-192.168.1.1,17127-212.27.40.240,17127-212.27.40

.241,17131-192.168.0.254,17153-192.168.1.1,17174-192.168.0.254,17209-212.27.40.

240,17209-212.27.40.241,17214-212.27.40.240,17214-212.27.40.241,17221-192.168.

254.254,17228-192.168.0.254,17240-212.27.40.240,17240-212.27.40.241,17253-192.

168.0.254,17263-192.168.0.254,17397-212.27.40.240,17397-212.27.40.241,17420-194.

225.152.10,17436-194.225.152.10,17437-194.225.152.10,17502-192.168.10.1,17502-

192.168.10.11,17569-212.27.53.252,17569-212.27.54.252,17619-192.168.88.1,17781-

192.168.254.1,17848-192.168.1.254,18024-195.22.112.69,18564-192.168.1.1,18710-

212.27.40.240,18710-212.27.40.241,18884-113.212.113.212,18925-212.27.40.240,

18925-212.27.40.241,19374-109.71.47.200,19374-130.185.80.80,19504-194.225.152.10

,19627-172.16.0.1,19671-212.27.40.240,19671-212.27.40.241,19984-196.49.13.38,

20096-202.52.255.3,20096-202.52.255.47,20210-192.168.10.1,20223-192.168.5.2,

20226-192.168.1.1,20295-192.168.0.254,20480-212.27.40.240,20480-212.27.40.241,

20515-192.168.2.2,20717-192.168.11.1,20777-192.168.0.1,20909-195.88.158.8,20955-

192.168.1.1,21101-202.159.32.2,21438-192.168.1.254,21596-192.168.1.254,21597-192

.168.1.1,21833-192.168.1.1,21990-192.168.1.254,21991-212.27.40.240,21991-212.27.

40.241,22095-109.71.47.200,22095-130.185.80.80,22171-192.168.0.254,22210-192.168

.99.254,22370-212.27.40.241,22372-212.27.40.240,22429-192.168.0.22,22480-192.168

.0.56,22711-192.168.1.1,22891-192.168.1.1,22892-192.168.69.254,22898-192.168.42.

254,22918-192.168.0.254,22963-192.168.0.254,23254-192.168.71.1,23289-192.168.1.

254,23290-192.168.3.1,23387-212.27.40.240,23387-212.27.40.241,24192-192.168.100.

254,24336-192.168.0.254,24520-192.168.0.254,24801-192.168.1.10,25003-192.168.88.

1,26095-192.168.0.254,26129-192.168.0.1,26671-192.168.1.1,26756-192.168.1.1,

26756-212.27.40.240,26756-212.27.40.241,26900-192.168.8.1,26966-192.168.88.1,

26966-212.1.224.6,26966-212.1.244.6,27263-212.27.40.240,27263-212.27.40.241,

27383-192.168.1.1,27538-192.168.200.242,27607-192.168.0.254,27622-192.168.0.254,

27624-192.168.1.254,28489-212.27.40.240,28489-212.27.40.241,28531-192.168.10.1,

28888-192.168.1.254,28889-192.168.0.1,28951-192.168.0.254,29425-192.168.4.254,

29622-192.168.0.254,29674-192.168.1.254,29770-192.168.1.254,29866-192.168.1.1,

30255-192.168.10.1,30266-192.168.10.1,30409-192.168.0.3,30676-189.50.140.50,

30771-192.168.1.254,30775-192.168.0.254,30807-192.168.1.254,30920-192.168.1.1,

30922-192.168.0.254,30951-192.168.0.254,30962-192.168.3.1,31085-192.168.1.104,

31517-192.168.1.254,31548-192.168.1.1,31556-192.168.1.1,31598-192.168.1.46,31749

-192.168.1.254,31764-192.168.14.1,31770-212.27.40.240,31770-212.27.40.241,31853-

192.168.1.254,31879-192.168.0.1,31932-212.27.40.240,31932-212.27.40.241,31946-

192.168.2.254,31959-192.168.178.4,31985-213.234.192.8,32016-192.168.0.254,32057-

212.27.40.240,32057-212.27.40.241,32140-192.168.5.254,32419-192.168.0.254,32529-

192.168.0.254,32537-192.168.1.6,32568-172.16.1.1,32606-212.27.40.240,32606-212.

27.40.241,32638-195.191.180.4,32638-195.191.180.9,32646-172.31.4.100,32646-172.

31.4.2,32646-172.31.4.205,32647-192.168.1.10,32730-192.168.0.1,32884-185.164.252

.10,32884-185.164.252.11,32930-192.168.28.254,32968-192.168.1.1,33024-103.7.200.

10,33069-185.40.96.96,33069-185.40.96.97,33110-199.19.224.159,33202-192.168.0.

254,33448-192.168.1.254,33520-192.168.0.254,33812-192.168.10.254,33924-192.168.1

.254,33967-212.27.40.240,33967-212.27.40.241,17858-2001:4ba0::53:1,17858-2001:

4ba0::53:2,17951-2001:41d0:fc8b:c900::314,27383-2001:470:28:a3c::1