BGP Tuner: intuitive management of DNS anycast infrastructures

Note: we’re using two types of reference in this blog series: hyperlinks refer to more high-level background information, while numbers between straight brackets ([]) link to in-depth technical material such as academic papers.

Anycast

Anycast is an internet routing technology that enables a client request to be routed to any one of multiple server sites, with the routing system dynamically selecting the server site and path that involve the smallest number of hops from the client. Anycast can reduce latency, improve network resilience and defend against DDoS attacks [Moura2015]. It is used extensively by root DNS operators, DNS service providers and content distribution networks (CDNs), including Google, Facebook, and Cloudflare.

Figure 1 illustrates the basic concept of anycast, with multiple servers using the same IP address. In the diagram, 3 servers are sharing the address 10.0.0.1, but are hosted at different locations. Each client (bottom of the diagram), can reach the servers via distinct paths that depend on AS interconnection relationships (represented by clouds). Under the hood, the BGP protocol is responsible for using a set of parameters and policies to determine which server traffic from each client should go to, and what path the traffic should take [Moura2015]. That means that the behaviour of an anycast service is very tightly coupled to BGP.

Figure 1: Anycast network.

Different clients therefore ‘see’ an anycast address in different ways, depending on which server BGP routes them to. The pool of clients that reach a given server site is referred to as that site's ‘catchment’. In order to build up a picture of – and in order to influence – which clients fall within which catchment, we have to work with the BGP protocol. The BGP protocol has various traffic engineering features [Quotin2003] that enable ASs to set routing preferences and forward the information to their neighbours. While we cannot change the basic behaviour of BGP, we can set preferences that may change our anycast visibility. Moreover, the internet is dynamic and routing paths may change over time – due to instability or new AS interconnections, for example. Consequently, a client that is routed via a particular path to one server today may be routed to another server tomorrow. That is not a bad thing: dynamicity brings robustness to anycast networks, but it does imply the need for continuous measurement.

Why monitor anycast networks?

Tools that help operators to manage their anycasted DNS services are vital for the secure and stable operation of those servers. In the SAND project, we have been developing and evaluating such tools, specifically to help operators answer the following questions:

1. How do clients on the internet reach my anycasted DNS service?

2. How can we manage the visibility of our DNS services for clients by manipulating BGP configurations?

Operations teams such as SIDN's need to be able to answer those questions in various real-world settings. For example:

• Traffic engineering Managing the way the internet routing system distributes clients across anycast nodes. As described, anycast networks use BGP as a platform to route the traffic. However, BGP does not have load balancing by default. As a consequence, we cannot manage the amount of traffic each site will handle. That is tied to the locations of our clients and their network interconnection relationships. For instance, let's return to Figure 1. If the majority of requests come from the first client (bottom left), the first site will be overloaded. The good news is that we can try to shift traffic to the other sites using BGP features [Hesselman 2017].

• DDoS mitigation Big distributed denial-of-service attacks can overload anycast sites. By understanding the characteristics of such attacks and by manipulating anycast catchments, the operator can reconfigure the anycast service and shift the load across the sites. Operators can use different mitigation strategies to isolate the traffic in certain regions or redirect it to a site with more capacity that is more able to handle the traffic.

• User experience Ideally, each client should reach the closest anycast site in terms of AS path hops, and that should typically result in the lowest possible RTT. However, the concept of ‘closest’ in anycast routing is not always related to geo-location or latency. The closest site depends on a combination of routing policies and interconnections between ASs. For instance, if clients from a particular AS in the USA are being routed to a site in Japan, that will impact the service latency for those clients and may require an operator intervention. An operator therefore needs to monitor the catchments of their anycast service and, where appropriate, adjust their BPG settings to influence the latency experienced by users.

Tangled anycast testbed

‘Tangled’ is the name of the anycast testbed that we have developed in the SAND project. It is a fully functional infrastructure that runs at 13 locations around the globe, providing more than 50 BGP sessions with several ISPs and Internet Exchange Points (IXPs). Sites include the University of Southern California, the WIDE Project in Japan and several commercial cloud providers. Tangled has been in operation since 2017, but in the last 2 years of the SAND project we have made several major improvements, such as:

• Increasing the number of sites from 8 to 13

• Enabling IPv6 BGP sessions

• Creating a new AS dedicated to our testbed

• Adding a new IPv4 prefix (/23)

• Adding a new IPv6 prefix (/40)

• Adding a CLI interface

The majority of Tangled sites are run by volunteers, but we have total control of the BGP sessions. We use 'exabgp' as an interface to manage the session and to deploy our policies. Exabgp supports different BGP policies using AS-Path Prepending, communities and different prefix length announcements. That flexibility is vital for measuring the traffic distribution over the sites in distinct anycast setups.

Figure 2 shows the Tangled prefix allocations that we use in our experiments. We have a relatively big prefix length for both IPv4 and IPv6. That allows us to use more than one routing prefix in Tangled. For instance, we can use a more specific prefix and map how that affects the traffic distribution over the nodes.

Figure 2: BGP routing prefix used on the Tangled testbed.

How can we monitor anycast service visibility?

Our first research question reflects the need to understand how clients are distributed over the sites of the anycast service. Building up a distribution picture is known as catchment mapping, where we discover which clients go to which sites [ISI2020].

Broadly speaking, there are 2 possible approaches: (1) using client-side vantage points such as RIPE Atlas to determine which server each client is reaching, or (2) using the open-source tool ‘Verfploeter' that runs on anycasted DNS servers. Verfploeter carries out ‘global probing’, which means that it regularly pings a large fraction of clients and then records which anycast sites the responses end up at in order to calculate anycast catchments.

Figure 3 shows the general idea of Verfploeter. The server ‘verfploeter’ sends ICMP requests (pings) to all networks on the internet. The requests give the anycast IP address as their source address (10.0.0.1 in our example). Clients use that address as the destination for their responses, which therefore end up distributed across the various anycast sites.

Figure 3: Verfploeter global anycast catchment probing tool.

Verfploeter takes around 15 minutes to test the whole internet. In the process, it receives more than 3.9M ping replies from clients. The replies basically describe the network and its respective catchments, i.e. which anycast site each network reaches. To better visualise this information, we have developed an interactive web interface (Figure 4) that sheds light on anomalies and allows an operator to examine how clients are distributed across the anycast sites. For example, it is possible to identify which server clients from each country are reaching. We can also see which ASs within a country exhibit a particular behaviour.

Figure 4: SAND GUI: Verfploeter data analysis graphical interface.

How can we modify catchments?

To modify the catchment of an anycasted service means changing the set of clients whose requests end up at the server. An operator may, for example, want to change the client distribution to improve service performance (latency and user experience), to balance traffic between sites by means of traffic engineering, or to defend against DDoS attacks (see above).

BGP provides a set of features that operations teams can use to change the visibility of their sites, such as AS-Path Prepending and communities. By setting BGP policies, they can prioritise certain routing paths and reduce the routing system’s preference for others. The settings will change the traffic distribution across the anycast servers. With catchment modification, the key thing is to map the BGP policy effects in terms of load distribution across the sites.

When operations teams change BGP policies, it is essential that they are able to monitor how BGP propagates their prefixes and the consistency of the prefix announcements. BGP routes take time to propagate over the internet and multiple consecutive changes may result in inconsistent prefix visibility. To monitor such situations, we use BGP RIPE RIS, which has several route collectors and is extensively used by the operator and research communities alike.

Figure 5: BGP visibility and convergency time.

Figure 5 illustrates the visibility of our Tangled testbed's prefix across the internet when we performed an AS-Path Prepending experiment. According to the statistics in RIPE RIS, we would expect our prefix to be seen by 267 peers under normal circumstances. To map the convergence time and visibility, we made 2 changes to our BGP policy: the first one (withdraw) involved removing the prefix announcement; the second (announcement) involved reinstalling the announcement.

The convergence time here, just after 19:00 in Figure 5, was quite low: less than 5 minutes (from the green line up to the red line). Since the internet's convergence time is usually around 15 minutes [Teixeira2017], our experiment provides important input for tailoring our experiments on the testbed. Another interesting behaviour is the number of peers that do not ‘forget’ our prefix even after a withdrawal message has been sent. As the graph shows, around 50 peers kept seeing our prefix after we withdrew it. We did not investigate that behaviour, but it would be an interesting topic for future research.

Anticipated effects on anycast services

The client distribution over anycast sites depends on BGP routing. As explained, the internet is very dynamic, and the routes selected may change over time. Therefore, it makes sense to monitor the anycast load distribution on a regular basis.

Moreover, it would be interesting to understand how BGP policies can affect the load distribution across anycast servers. For example, what are the load distribution implications of implementing an AS-Path Prepend? We therefore developed a systematic way of measuring and mapping the side effects of each BGP configuration performed. We applied the methodology in the Tangled testbed, but the methodology would also work for production systems, such the systems used for .nl. The following steps were defined:

1. Build baseline: measure and map the distribution of clients across our anycasted sites using the regular BGP configuration.

2. Apply changes: change BGP policy and redo the measurement.

3. Traffic shift: redo the measurement and carry out mapping to compute how clients are distributed across our sites after the change (i.e. the deviation from the baseline).

In one of our experiments, we configured 6 sites as an anycast service. To name the sites we used airport code as identifiers. Table 1 present the client distribution per site according to the applied BGP policy. The policy baseline represents the distribution of clients over the sites when using our default policy (regular BGP prefix announcement). To map the deviations from the baseline, we changed the BGP prefix announcement using AS-Path Prepends, implementing a total of 78 distinct policies. The policy named 1xCDG involves a one-time AS-Path Prepend in Paris (CDG=Charles de Gaulle Airport). The other policies follow the same convention by describing the number of prepends and the respective anycast site.

Table 1: Client distribution: each line presents a measurement showing the traffic distribution percentage per site in the respective BGP policy.

Each line in Table 1 represents a measurement in the Tangled testbed and the respective percentage of /24 networks handled by each site for each BGP policy. The first line shows our service baseline, involving use of our regular BGP prefix announcement. For this configuration, 3.994% of the clients went to the site located in Paris (CDG); 8.453% went to Washington D.C. (IAD); 12.936% went to London (LHR); 8.832% went to Miami (MIA); 11.608% went to Porto Alegre, Brazil, and 9.494% went to Sydney. The ‘unknown’ column represents the percentage of networks (/24) that we did not get replies from (via Verfploeter). That means that we were unable to map 44.683% of /24 networks. Again, the efficacy of the methodology has been discussed [Vries2017].

The second line shows the results when one prepend in Paris is used. In other words, when we added one prepend on the Tangled site in Paris, but did not change the prefix announcement on the other sites. As seen in the second line, the BGP policy in question (1xCDG) decreases the load on CDG when compared with the baseline.

Together, the measurements presented in Table 1 can be used to predict the number of clients associated with each BGP configuration. Operators can use the table to configure their anycast services to achieve their desired site loads. In anycast services with many sites, however, the process may be confusing and error-prone. To address that challenge, we have come up with a tool called the BGP Anycast Tuner, which uses the table above as input.

BGP Anycast Tuner

The BGP Tuner is a prototype graphical interface that presents the distribution of clients over the anycast sites using a pre-determined BGP configuration. We used Verfploeter to carry out the measurements on Tangled, pre-processed them, and then used them as the input for our interface.

Figure 6: The BGP Anycast Tuner interface.

This interface (Figure 6) assists the operator in an intuitive way, allowing them to easily ‘equalise’ the catchments, or traffic volume at each site. An interactive display shows the traffic distribution across the sites when the previously measured BGP policy is applied. The bar chart in the middle of the display shows the distribution of clients across the respective anycast sites. In the figure, 6 sliders are shown, one for each site. The operator can play around with the sliders to increase/decrease the number of clients per site. Note that the sliders do not make linear adjustments; instead, every mark on the slider scale corresponds to a previously measured BGP policy. For some anycast sites, the marks are spread along the full length of the slider scale, while for others the marks are more concentrated. The differences reflect the various sites' ability to shift clients between anycast nodes. Some sites can exert more influence because of their relationships with their neighbours and their traffic agreements.

Our tool enables an operator to identify the best policy for their DNS anycast service, i.e. the policy that will achieve the optimum distribution of clients across the anycast sites. Operators can also use more advanced policies. For example, on the left side (drop-down menu) there are options such as ‘Bring traffic to Europe’ and ‘Reduce traffic to USA’. Such options will help operators to identify which of the presented policies are most suitable.

The BGP Anycast Tuner is an open-source tool and available here.

Conclusions

Anycast networks are increasingly being adopted by operators to improve the robustness of their services and the quality of the user experience. DNS operators and CDN providers have been using anycast networks to deliver content to users spread across multiple countries. The management of those networks has brought new challenges to the DNS operator community. How many sites are required? [Schmidt2017] Where should those sites be placed to optimise the traffic? How can BGP be used to achieve load-balancing across sites? How to debug the performance of clients? How can overall services be optimised in terms of latency and throughput?

In this last phase of the SAND project, we measured and mapped anycast services using the Verfploeter tool developed in the previous phase of the project. We also built an interactive interface (the BGP Anycast Tuner) that visualises the catchments of a service's anycast sites. In addition, a structured methodology was developed for measuring client distribution across anycast sites and measuring the effects of applying various BGP policies. Furthermore, we made the BGP Anycast Tuner freely available, enabling operators to analyse and manage the effects of different BGP configurations on their sites.

We are following up the work by working with SIDN's operations team to implement our measurement methodology as part of the anycast monitoring for .nl. The findings presented here and the testbed infrastructure also continue to be used in other research, such as the PAADDoS project.

Acknowledgments

This work was funded by SIDN Labs and NLnet Labs through the project Self-managing Anycast Networks for the DNS (SAND). SAND is a collaboration involving the University of Twente, SIDN Labs and NLnet Labs.

This blog relates to the second phase of the SAND project (2018-2020). The first phase have yielded various products, including the Verfploeter tool.

We also wish to thank the University of Southern California for its cooperation, particularly John Heidemann for his valuable feedback throughout the project.

References

• [Moura2015] Moura, G. C. M., Schmidt, R. D. O., Heidemann, J., De Vries, W. B., Möller, M., Wei, L., & Hesselman, C. (2016). Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event. Paper presented at the Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC.

• [Quotin2003] Quoitin, B., Pelsser, C., Swinnen, L., Bonaventure, O., & Uhlig, S. (2003). Interdomain traffic engineering with BGP. IEEE Communications Magazine, 41(5), 122-128. doi:10.1109/mcom.2003.1200112

• Broad and load-aware anycast mapping with verfploeter. Paper presented at the Proceedings of the 2017 Internet Measurement Conference, London, United Kingdom. https://doi.org/10.1145/3131365.3131371

• [Schmidt2017] Schmidt, R. D. O., Heidemann, J., & Kuipers, J. H. (2017, March). Anycast Latency: How many sites are enough? In Proceedings of the Passive and Active Measurement Workshop. Springer, Sydney, Australia (pp. 188-200).

• [ISI2020] The ANT Lab: Analysis of Network Traffic. Online: https://ant.isi.edu/anycast/catchment/index.html

• [Teixeira2007] Renata Teixeira, Steve Uhlig and Christophe Diot. BGP route propagation between neighboring domains. In International Conference on Passive and Active Network Measurement, pages 11–21. Springer, 2007.

• [Hesselman 2017] C. Hesselman, G. C. M. Moura, R. d. O. Schmidt and C. Toet, Increasing DNS Security and Stability through a Control Plane for Top-Level Domain Operators, in IEEE Communications Magazine, vol. 55, no. 1, pp. 197-203, January 2017.