Network service provider Cloudflare recently published a blog to share details about a recent Border Gateway Protocol (BGP) route leak that lasted for 25 minutes, affecting IPv6 traffic and causing significant congestion, packet loss, and approximately 12Gbps of traffic loss.
The BGP system helps autonomous systems (AS) route data between different networks, and these autonomous systems send data to their destinations through smaller networks on the Internet, and this routing breach was caused by an unexpected policy misconfiguration that affected external networks beyond Cloudflare customers.
In a blog post Cloudflare said:
In the January 22nd route breach, we caused a similar route breach where we took routes from some of our peers and rerouted them to other peers and providers located in Miami, USA.
According to the route leak definition in RFC7908, we have a mixture of Type 3 (cross-peer leakage) and Type 4 (provider route leakage to peers) route leaks on the Internet. The problem occurs because of a policy change that was originally used to prevent Miami from publishing Bogota's IPv6 prefix.
Since removing a specific prefix list makes the export policy too permissive, this allows route type internal matching to accept all IPv6 routers internally (iBGP) and export them externally.
So Cloudflare redistributes all IPv6 prefixes within the backbone that comply with this policy, and the automated system then advertises them to all BGP neighbors in Miami (that is, affecting the routing of those peers).
Possible impacts of route leakage:
BGP route leakage occurs when an autonomous system violates the Valley-free Routing principle and mistakenly advertises a route learned by one peer or provider to another peer or provider. That is, traffic is sent to a network that should not carry the traffic.
The routing strategy itself is the result of careful optimization by major providers. Sending traffic to networks that should not be carried may lead to network congestion, packet loss, or the use of suboptimal paths. When firewall filters only allow traffic from specific providers to pass, abnormal traffic will be completely discarded.
The valley-free principle describes how routes should be propagated based on business relationships between networks. When these rules are violated, traffic is drawn onto longer or unstable paths that cannot be carried.
In addition to causing network congestion, packet loss, and traffic being dropped, such events can also present security risks by causing traffic to pass through unauthorized providers where packets may be intercepted or analyzed.
Cloudflare’s self-reflection:
The issue was detected shortly after it occurred, and Cloudflare engineers manually revoked the configuration and paused the automated process to prevent the impact of the issue for 25 minutes, after which the code changes that triggered the issue were directly undone.
Cloudflare said that this incident is similar to another BGP breach that occurred in July 2020. Cloudflare will take measures to avoid similar problems from happening again. Proposed measures include adding strict community-based egress safeguards, CI/CD checks for policy errors, improving early detection, validating RFC9234, and promoting the adoption of RPKI ASPA.