BGPsec - Could you run it if you wanted to?

BGP: the glue of the internet

The Border Gateway Protocol (BGP) interconnects the more than 75,000 different networks making up the internet to form a single global communications infrastructure. It uses the concept of a "BGP path" to determine the best route to a network and with that the prefixes that network announces. A router learns these paths through BGP update messages from other routers.

Plain BGP does not come with security mechanisms to ensure that the BGP path asserted by a router is valid. Therefore, at its core, BGP is based on trust between parties running BGP routers. In the ever-growing internet infrastructure, such trust is not always a given anymore. For example, a malicious Autonomous System (AS) can purposely announce a prefix that it is not authorised to announce (prefix hijack), or it can modify an AS path to create a fake link to its own AS (AS path forgery). Both of these examples redirect traffic to an unintended network, where it can be intercepted or black-holed.

What is BGPsec?

BGPsec is an extension to BGP that provides the basis for path verification and can protect against the 2 types of attack mentioned above. To construct a valid BGP update with the BGPsec attribute, routers need to add a signature. The signature must cover the prefix announced by the update, the AS number (ASN) of the AS the update is sent to and all prior signatures. The private key routers use for signing must match the BGPsec router certificate stored in the Resource Public Key Infrastructure (RPKI) associated with the router's ASN.

BGPsec is one way to secure BGP. Another is to use Route Origin Authorization (ROA) objects, an approach that has attracted considerable uptake. ROAs allow a router to verify that the AS announcing a given prefix is authorised to do so, thus protecting against prefix hijacks. Furthermore, the RPKI may soon also store Autonomous System Provider Authorization (ASPA) objects that will make it possible to evaluate the plausibility of a received path based on a list of pre-authorised provider ASes. The rest of this blog will focus on BGPsec. For more current information on ROAs and ASPA, please refer to the article Adoption of RPKI/ROV security protocol progressing very quickly at sidn.nl.

BGPsec's problem: no deployment since 2017

While BGPsec protects against an additional class of attacks, it has not seen the same kind of growth as ROAs, despite being standardised for 8 years now. This can be traced back to a multitude of concerns within the research and operational community. These include increased processing costs and memory requirements for the current generation of routers, an increase in update messages and the lack of incentives for early adopters because the benefits of BGPsec are limited during partial deployment.

At the same time, BGPsec has potential for the next generation of routers in combination with ROAs and ASPA, as widely acknowledged BGP expert Job Snijders recently pointed out. We therefore empirically validate the functional and interoperability capabilities of current BGPsec-enabled router implementations, which must be mature to scale up BGPsec to larger deployments. Our approach is to conduct hands-on experiments in a small-scale testbed that we set up.

What would you need to deploy a BGPsec router?

To enable routers to sign and validate BGPsec updates, they must first obtain router certificates from a Regional Internet Registry (RIR). However, at the time of writing, none of the RPKI repositories hosted by the 5 RIRs support the publication of BGPsec router certificates. RIPE NCC states that implementation of support is planned for Q2 of 2025. Fortunately, it's also possible to make use of delegated RPKI. Delegated RPKI allows operators and researchers to store RPKI objects in a separate repository and make it available to the public without having to wait for the RIRs. Krill is a certificate authority software that can be used for this purpose, which supports BGPsec router certificates.

The second component that a BGPsec router needs is a validating cache. This validates and stores BGPsec router certificates that it retrieves from the RPKI via HTTPS or rsync. Three actively maintained validating caches are Routinator, FORT validator and rpki-client. Routinator and rpki-client support validation of BGPsec router certificates.

Finally, the router needs to support the RPKI-to-Router (RTR) Protocol. Routinator comes with a built-in RTR-server that supports RTR version 1. As described by RFC8210, version 1 of RTR adds functionality for BGPsec. FORT validator used to support BGPsec routercertificates until version 1.5.2. However, support was removed because "more testing" is required. From the selection of available RTR protocol implementations, StayRTR and RTRlib support BGPsec router certificates.

Evaluated routers supporting BGPsec

We evaluated 5 software routers with support for BGPsec. We obtained the 5 implementations from bgpsec.net: QuaggaSRx, GoBGP-SRx, ExaBGP-SRx, FRR and BIRD. These implementations are all between 2- and 10-year-old forks of existing, maintained software routers. There is currently no hardware router known to us that supports BGPsec.

QuaggaSRx, GoBGP-SRx and ExaBGP-SRx were built by a team at NIST and are part of a BGPsec reference implementation: the BGP (S)ecure (R)outing E(x)tension Software Suite (NIST BGP-SRx). Running a software router that is part of the suite requires building the SRxCryptoAPI. The API provides cryptographic capabilities for the router, allowing it to sign and validate BGPsec paths.

As well as the API, NIST's BGP-SRx provides an SRx-Server that can be used to outsource validation from the router. It can be connected to a validating cache to retrieve validated data from an RPKI repository, including available BGPsec router certificates. Certificates can also be provided to the router directly through the SRxCryptoAPI's local key store.

The 2 remaining BGPsec implementations we evaluated are forks of BIRD and FRR. Neither of them is part of NIST's BGP-SRx and both are independent from it. The BGPsec fork of BIRD predates standardisation of BGPsec and is the oldest fork in this list. Through some minor patching, we were able to fix the parts of the implementation that were clearly outdated. An example being the use of code that doesn't conform to the standard.

Our experimental BGPsec network

We set up Docker containers to evaluate and verify all 5 routers and their functionality. We compiled and patched the implementations where necessary and configured BGPsec-enabled BGP sessions between them. That involves setting a Subject Key Identifier (SKI) that identifies a BGPsec router certificate and a corresponding private key for signing. Where necessary, we specify the IP address of the SRx-Server or Routinator. Before running the routers, we prepare private keys and certificates in the required formats and according to defined naming schemes. The arguments required to configure BGPsec vary between router implementations.

We evaluated the routers with topologies consisting of 2 BGP speakers and announced 1 IPv4 prefix from each BGP speaker. For the implementations that can fetch data from a validating cache via RTR (or indirectly through the SRx-Server), we used Routinator as a validating cache and Krill as an RPKI testbed containing our BGPsec router certificates.

QuaggaSRx is integrated with the SRx-Server. As shown in figure 1, in our setup Routinator retrieves the BGPsec router certificates from Krill and communicates with the SRx-Server through RTR. Communication between the SRx-Server and QuaggaSRx is governed by the not yet standardised SRx Proxy Protocol. If configured, validation of signatures is outsourced to the SRx-Server. Private keys are stored locally in the SRxCryptoAPI's key store and are used to create signatures.

Figure 1. QuaggaSRx setup in our testbed.

While the project website of NIST's BGP-SRx states that GoBGP-SRx and ExaBGP-SRx both support the SRx-Server, the documentation linked on the same page describes "three possible scenarios on how the use of the SRx-Server could be facilitated". Based on our analysis and the existing documentation, we determined that the current implementations do not offer out-of-the-box integration with the SRx-Server. However, they do have access to BGPsec router certificates when provided through the key store associated with the SRxCryptoAPI. As shown in figure 2, we tested both routers by storing private keys and certificates locally. When the SRxCryptoAPI is initialised, they are loaded and can be used for signing and validating.

Figure 2. Setup to run ExaBGP-SRx or GoBGP-SRx in our testbed.

FRR makes use of RTRlib and its existing support for BGPsec router certificates to communicate with the validating cache. Figure 3 shows that the communication between FRR and Routinator happens directly via RTR without an intermediate server, as with QuaggaSRx.

Figure 3. FRR setup in our testbed.

BIRD does not have an integrated RTR client and does not communicate with a validating cache. This setup therefore only relies on the BIRD router itself, as shown in Figure 4. In contrast to ExaBGP-SRx and GoBGP-SRx, BIRD does not use BGPsec router certificates to validate but only requires locally stored corresponding public keys.

Figure 4. BIRD setup in our testbed.

Methodology: on-the-wire analysis of BGPsec announcements

To evaluate the routers' ability to create valid signatures and construct BGPsec path attributes using the configured private key, we announced IPv4 prefixes from each of them and analysed the traffic between them using Wireshark.

To test the validation of signatures, we constructed BGPsec updates with a private key that did not match the one that was configured for the sending AS in Krill or the local key store. We then examined the logs and the RIB of each router to see how the router deals with valid and invalid signatures. Listing 1 shows an example of BIRD accurately marking a route as BGPsec valid in its RIB.

bird> show route
10.2.0.0/16        via 10.0.0.2 on eth0 [bgp1 14:58:53] * (100) [AS64602i] ASP:1 (BSEC VALID: 64602)
10.3.0.0/16        via 10.0.0.3 on eth0 [static1 14:58:48] * (200)

Listing 1. BIRD showing route as BGPsec valid in its RIB.

We evaluated interoperability between routers by setting up BGP sessions between all combinations of 2 router implementations (so, 15 in total). If BGP updates with BGPsec attributes were successfully exchanged and processed, we deem the 2 implementations interoperable. Listing 2 shows an example of 2 routers where this was unsuccessful: after FRR receives a BGPsec update from BIRD, FRR aborts and the session is terminated.

frr_two     | BGP: Received signal 6 at 1741005719 (si_addr 0x1, PC 0x7f52ec57ab2c); aborting...
frr_two     | malloc(): unaligned tcache chunk detected

Listing 2. FRR aborting after receiving a BGPsec update from BIRD.

Result #1: BGPsec router functionality

Table 1 summarises our evaluation of the implementations' functional capabilities. All implementations correctly announce the router's BGPsec capability in the BGP OPEN message and construct valid BGPsec attributes in updates. For BIRD, we had to develop and apply a patch to align the code for the capability announcement and the BGPsec path attribute with the current BGPsec standard.

¹ Signing with a wrong SKI does not show up in a log or in the adj-rib in / out. Route is added.

² Signature validation failure shows up in the log but not in the RIB. Route is added. RIB shows that route was learned through a BGPsec UPDATE, not whether it was successfully validated.

³ Signature is marked as valid if SKI is present in the data received from Routinator. If the SKI used for signing matches the ASN, is not validated.

Table 1. BGPsec functions for each evaluated implementation.

We found that it is unclear whether ExaBGP-SRx validates the BGPsec signatures it receives. It does not log validation results and does not specify the result in the RIB either. Received routes are always added. FRR does validate received signatures. However, it does not ensure that the SKI used to sign corresponds to the AS that created the update. Any signature created with an SKI present in the RPKI data is accepted.

The connection to a validating cache via RTR functions as expected with QuaggaSRx through the SRx-Server and with FRR through RTRlib, as described in their documentation. The documentation of NIST's BGP-SRx also describes several methods that could be used to integrate ExaBGP-SRx and GoBGP-SRx with the SRx-Server to access RPKI data and outsource validation like QuaggaSRx, but that would require additional implementation work.

Result #2: BGPsec router interoperability

We also evaluated the routers' interoperability, as shown in Table 2. We observed that BIRD is the least interoperable with the other implementations. ExaBGP-SRx and GoBGP-SRx both closed their session with BIRD after sending a BGP NOTIFICATION message. In the case of FRR, receiving a BGPsec message from BIRD even lead to FRR being terminated due to a SIGABRT.

¹ BIRD does not correctly state in the OPEN message that it supports IPv6 unicast but then sends a MP_UNREACH_NLRI for IPv6 unicast, which leads to ExaBGP-SRx sending a NOTIFICATION message and closing the connection.

² After receiving a BGPsec UPDATE from BIRD, GoBGP-SRx sends a NOTIFICATION with code 0 and the session is closed.

³ BIRD sends a BGPsec UPDATE to FRR, which leads to a SIGABRT caused by a malloc call.

Table 2. Interoperability of each evaluated BGPsec implementation.

Where do we go from here?

We carried out a hands-on evaluation of 5 existing router implementations that support BGPsec using a small-scale testbed. The routers are functional and interoperable enough to set up a testbed and conduct experiments. However, the implementations are far from being able to support BGPsec in production, let alone on the scale that would be necessary to secure BGP routing globally. We provide containers for all implementations in our GitHub repository so you can easily construct your own testbed.

One way forward that we see for BGPsec is conducting experiments by replaying real-world routing data in a larger BGPsec testbed to gain a deeper understanding of BGPsec's impact on router performance and scalability in the wild. Another possibility could be to enhance our compatibility evaluation with more tests, which could help to address ambiguities in the BGPsec standard.

We will take the next months to identify where we at SIDN Labs could help to address those challenges. In the meantime, we invite you to experiment with the implementations yourself, using our testbed.

Happy to hear your feedback

We are curious to hear your opinion on this topic!

Have we missed any implementations? And what are your opinions on BGPsec and its future in general? Feel free to contact us at lisa.bruder@sidn.nl.