No. 005 · Wing of the Global Push

One Customer, One Setting

· under an hour

  1. 1A software change deployed weeks earlier carries a latent bug
  2. 2One customer pushes a valid configuration change
  3. 3The change triggers the bug across the edge network
  4. 4Most of the network starts returning errors; major news sites go blank
  5. 5Engineers find the config, disable it, and traffic recovers

A bug can wait patiently for the one input that wakes it.

Source: Fastly blog, June 2021

Fastly outage, June 2021. Duration: under an hour. 5 hops from the first change to the failure.