No. 001 · Hall of Lost Packets

The Day Facebook Left the Internet

· about 6 hours

  1. 1Routine maintenance command meant to assess backbone capacity
  2. 2Command takes down every backbone connection between data centers
  3. 3DNS servers see the backbone is gone and withdraw their BGP routes
  4. 4facebook.com, Instagram and WhatsApp stop resolving worldwide
  5. 5Engineers need physical access to reset routers; internal tools are down too ✕

If your recovery tools live behind the thing that broke, you don't have recovery tools.

Source: Facebook Engineering blog, October 2021

Facebook outage, October 2021. Duration: about 6 hours. 5 hops from the first change to the failure.