On June 8, 2021, a configuration change at Fastly took down dozens of major websites for roughly an hour. On October 4, 2021, a faulty BGP route leak at Facebook made their entire autonomous system disappear from the internet. On December 7, 2021, an AWS us-east-1 outage cascaded into failures across services that thousands of businesses depended on. And on July 18, 2024, a CrowdStrike configuration update bricked millions of Windows machines worldwide.
If you read the post-incident reports, you will notice something unsettling: the root causes sound almost identical to incidents from 2019, 2017, 2015, and earlier. BGP leaks. DNS misconfigurations. Expired TLS certificates. Cascading failures from single-region dependencies. These are not novel failure modes. They are well-documented, well-understood, and eminently preventable. Yet they keep happening.
The Same Four Failures, Again and Again
Internet infrastructure breaks in predictable ways. If you read outage reports across the last decade, you can sort almost every major incident into a small handful of categories.
BGP Route Leaks and Hijacks
Border Gateway Protocol is the routing protocol that holds the internet together. It works on trust: if an autonomous system announces a route, other systems accept it. When an ISP accidentally announces prefixes it should not—whether through misconfiguration or software bugs—traffic gets routed into black holes. This happened in 2008 when Pakistan Telecom tried to block YouTube domestically and instead made YouTube unreachable globally. It happened again in 2019 when a small ISP in Pennsylvania accidentally announced large chunks of the internet’s address space, diverting traffic through networks that could not handle it. And it keeps happening. BGPStream detects route leaks and hijacks on a near-daily basis. Most are small enough to avoid headlines. Some are not.
DNS Misconfigurations and Outages
DNS is the phonebook of the internet, and it breaks in boring, repetitive ways. Someone deletes a record. Someone forgets to renew a domain. Someone deploys a configuration file with a syntax error that propagates to all resolvers. In 2023, a Microsoft DNS misconfiguration took down multiple Azure services for hours. The cause was, at its core, the same class of error that has broken DNS for two decades: a human made a change, there was no adequate validation, and the change was pushed to production without sufficient rollback capability.
Expired Certificates
TLS certificates expire. This is not a surprise. The expiration date is embedded in the certificate itself. Yet organizations with engineering teams numbering in the thousands still let certificates expire, taking down services. In 2020, an expired certificate at CenturyLink took down a large portion of internet traffic for hours. Verizon had the same problem in 2019. These are companies whose core business is network infrastructure. If they cannot manage certificate rotation, who can?
Single-Region and Single-Vendor Dependencies
Every significant cloud provider outage reveals how many services depend on a single region—usually us-east-1—and fall over when that region has problems. Multi-region architectures are not a secret. They are covered in documentation, conference talks, and blog posts from every major provider. But building multi-region failover is hard and expensive, so many organizations treat single-region deployment as acceptable risk right up until it breaks.
Why Nobody Learns
The failure modes are known. The mitigations are known. So why do we keep repeating the same mistakes?
Institutional Amnesia
Engineering teams have high turnover. When the people who lived through the last outage leave, they take the institutional memory with them. The post-incident document survives, but the visceral understanding of what went wrong and why it mattered does not. New engineers arrive, see a system that has been running fine for two years, and assume it is solid. They do not know that it is solid only because of a fragile manual process that someone set up after the last incident and that nobody has maintained since.
This is not a knowledge management problem. It is a cultural problem. Organizations that treat incident reports as checkbox exercises rather than genuine learning opportunities will keep having the same incidents. The report gets filed, the remediation items get assigned, and then everyone gets busy and half the action items get dropped. Six months later, the same class of failure recurs.
Misaligned Incentives
Reliability work is invisible when it succeeds. Nobody gets promoted because a BGP route leak did not happen. Nobody gets a bonus because their DNS configuration validated correctly before deployment. The work that prevents outages is slow, unglamorous, and difficult to justify in quarterly planning. Shipping features and hitting launch dates are visible. Hardening infrastructure against edge cases is not.
This creates a predictable cycle. An outage happens. Leadership demands to know why this was allowed to occur. The team produces a post-incident report with strong remediation items. Some of those items get implemented. Time passes. The urgency fades. The next quarter’s priorities take over. The remaining items get deprioritized. Then the next outage happens.
The Complexity Ceiling
Modern internet infrastructure is staggeringly complex. A single request to load a web page might traverse dozens of systems—DNS resolvers, CDN edge nodes, load balancers, application servers, caching layers, databases—each with its own configuration, its own failure modes, and its own operators. No single person understands the full stack. No single team owns the entire dependency chain. This distributed ownership means that even when one team learns from a failure, that learning may not propagate to the teams operating adjacent systems.
The complexity also means that failures interact in ways that are hard to predict. A BGP leak combined with a DNS timeout combined with a certificate expiration can create cascading failures that nobody designed for and nobody tested. Testing for these scenarios requires investment in chaos engineering and failure injection—practices that many organizations still treat as optional rather than fundamental.
What Actually Works
Some organizations do learn. The difference between teams that have recurring outages and teams that do not comes down to a few practices that are technically straightforward but culturally difficult.
Automated Validation, Not Manual Review
Every BGP leak should have been caught before propagation. Every DNS misconfiguration should have been blocked at the configuration layer. Every expired certificate should have been rotated automatically. The technology to prevent these failures exists. Automated BGP validation through RPKI exists. DNS configuration linters exist. Certificate automation through ACME exists. The problem is deployment and enforcement.
If your organization relies on humans remembering to do the right thing, you will eventually have a failure caused by a human forgetting. The remediation is not better humans. It is systems that make the failure impossible or at least difficult. Route announcements should be validated against an allowlist before propagation. DNS changes should go through a validation pipeline that checks syntax and references before they reach production resolvers. Certificate renewal should be automated end to end with monitoring that alerts when renewal is approaching and has not happened.
Blameless Post-Incident Reviews That Actually Change Things
A post-incident review is worthless if it does not result in changes to systems, processes, or both. The review should identify not just what happened but why the system allowed it to happen. If a configuration error took down production, the question is not why did the engineer make the error—it is why the system allowed an erroneous configuration to reach production. Blaming individuals produces fear. Fear produces hidden failures. Hidden failures produce catastrophes.
The best post-incident reviews also assign owners and deadlines to every remediation item, and they follow up. If an action item from a review has not been completed within the agreed timeframe, that should be treated with the same urgency as a production bug.
Investing in Boring Infrastructure
The organizations that avoid repeated outages are the ones that invest in boring, unsexy infrastructure work. Multi-region failover. Automated failback. Chaos engineering. Runbook automation. Observability that covers the full request path. These are not new ideas. They are the same ideas that have been discussed at every infrastructure conference for the past decade. The gap is in execution.
If your roadmap has no room for reliability work, your roadmap is wrong. Full stop. You are gambling that the next outage will not happen during a critical business window. That is a bet you will eventually lose.
The Human Stakes
It is easy to treat internet outages as technical abstractions. A BGP route leak is just some routing tables getting messed up. A DNS outage is just some name lookups failing. But these failures have real human consequences. When healthcare systems go down, patients cannot access records and appointments get missed. When payment processors fail, people cannot buy food or pay bills. When government services go offline, people cannot access benefits they depend on.
Infrastructure is not just infrastructure. It is the substrate that people’s lives run on. Treating reliability as optional is treating the people who depend on your systems as optional. Every recurring outage is a choice. The knowledge to prevent it exists. The tools to prevent it exist. What is often missing is the will to prioritize the work.
The internet will keep breaking in the same ways until the organizations that build and operate it decide that preventing known failure modes matters more than shipping the next feature. That is not a technical judgment. It is an organizational one. And right now, most organizations are making the wrong call.
FAQ
Why do BGP route leaks keep happening if the problem is well known?
BGP was designed for a small, cooperative network where trust was assumed. It has no built-in authentication or validation. Technologies like RPKI exist to add verification, but adoption remains incomplete. Many networks still accept and propagate unvalidated route announcements. Until a critical mass of networks validate routes before accepting them, leaks will continue to propagate. Deploying RPKI is not technically difficult, but it requires coordination across thousands of autonomous systems, and the incentives for any individual network to deploy it are misaligned—they bear the cost of deployment but the benefits are distributed.
Are certificate expiration outages really still a problem with ACME and Let’s Encrypt?
Yes, they are. Automated certificate management exists and is widely deployed, but it is not universal. Many organizations still manage certificates manually, particularly for internal systems and legacy infrastructure. Even organizations using ACME can have failures if the automation breaks—monitoring for failed renewals is just as important as automating the renewal process itself. The problem is not the absence of technology. It is the failure to apply that technology consistently and to monitor the automation that is supposed to prevent the failure.
What can individual engineers do if their organization will not prioritize reliability?
Document the risks. Write them down, share them with leadership, and keep a record. When an outage happens that was predicted and could have been prevented, that documentation matters. Push for incremental improvements where possible—automating a single validation step, adding a single monitoring check, running a single failure injection test. Build relationships with adjacent teams so that when something goes wrong, you have the social capital to push for systemic changes rather than just fixing the immediate symptom. And if your organization consistently refuses to invest in reliability despite clear evidence that the risk is real and growing, consider whether that is an organization you want to continue building infrastructure for.
