No. 001 · Hall of Lost Packets
· about 6 hours
- 1Routine maintenance command meant to assess backbone capacity
- 2Command takes down every backbone connection between data centers
- 3DNS servers see the backbone is gone and withdraw their BGP routes
- 4facebook.com, Instagram and WhatsApp stop resolving worldwide
- 5Engineers need physical access to reset routers; internal tools are down too ✕
If your recovery tools live behind the thing that broke, you don't have recovery tools.
Source: Facebook Engineering blog, October 2021
No. 002 · Hall of Lost Packets
· about 2 hours
- 1Pakistan Telecom is ordered to block YouTube domestically
- 2It announces a more specific route for YouTube's address block to do so
- 3Its upstream provider passes the announcement on to the rest of the internet
- 4Routers everywhere prefer the more specific route
- 5Most of the world's YouTube traffic heads into Pakistan and disappears ✕
BGP believes what it is told. Filter what your customers announce.
Source: RIPE NCC case study, 2008
No. 003 · Wing of the Global Push
· hours to fix, days to weeks to clean up
- 1Security vendor ships a routine content update to its Windows sensor
- 2Update defines more input fields than the sensor code expects
- 3Kernel driver reads out of bounds and crashes
- 4About 8.5 million Windows machines boot-loop on a blue screen
- 5Airlines, hospitals and banks fix machines one by one, by hand ✕
Content updates are code updates. Stage them like code.
Source: CrowdStrike root cause analysis, August 2024
No. 004 · Wing of the Global Push
· 27 minutes
- 1New managed WAF rule is deployed globally at once
- 2Its regular expression backtracks catastrophically on ordinary traffic
- 3CPU on every edge machine hits 100%
- 4Sites behind the network return 502s worldwide
- 5Global kill switch for the WAF rules restores traffic ✕
The fastest deploy pipeline is also the fastest outage pipeline.
Source: Cloudflare blog, July 2019
No. 005 · Wing of the Global Push
· under an hour
- 1A software change deployed weeks earlier carries a latent bug
- 2One customer pushes a valid configuration change
- 3The change triggers the bug across the edge network
- 4Most of the network starts returning errors; major news sites go blank
- 5Engineers find the config, disable it, and traffic recovers ✕
A bug can wait patiently for the one input that wakes it.
Source: Fastly blog, June 2021
No. 006 · The Enter Key Gallery
· about 4 hours
- 1Engineer debugging a slow billing system runs an approved playbook command
- 2One input is mistyped; far more servers are removed than intended
- 3Two S3 subsystems in us-east-1 need a full restart
- 4They haven't been fully restarted in years; it takes hours
- 5A large part of the web goes down, including AWS's own status dashboard ✕
Tools should refuse to remove too much capacity too fast, however it's typed.
Source: AWS service disruption summary, March 2017
No. 007 · The Enter Key Gallery
· about 18 hours; roughly 6 hours of data lost
- 1Late-night fight with database replication lag
- 2Engineer deletes a data directory to reseed a replica
- 3The shell was on the primary, not the replica
- 4The backup methods turn out to be broken or empty
- 5Restore from a staging snapshot, livestreamed to the public ✕
A backup you have never restored is a hope, not a backup.
Source: GitLab postmortem, February 2017
No. 008 · The Enter Key Gallery
· 45 minutes
- 1New trading code reuses a flag that once enabled retired logic
- 2Deployment misses one of eight servers
- 3Market opens; that server runs the dead code
- 4It fires millions of unintended orders
- 5The firm loses about $440 million and needs a rescue ✕
Delete dead code. Verify every deploy target. Never recycle a flag.
Source: SEC order, October 2013
No. 009 · Cabinet of Wrong Numbers
· 37 seconds
- 1Guidance software reused from Ariane 4
- 2Ariane 5 flies faster; a horizontal velocity value grows larger
- 3Converting a 64-bit float to a 16-bit integer overflows
- 4Both inertial reference units fail the same way
- 5The rocket veers off course and self-destructs ✕
Reused code carries the assumptions of the system it was written for.
Source: Ariane 501 Inquiry Board report, July 1996
No. 010 · Cabinet of Wrong Numbers
· a 10-month trip, lost on arrival
- 1Ground software reports thruster impulse in pound-force seconds
- 2Navigation software expects newton-seconds
- 3Every small trajectory correction is off by a factor of about 4.45
- 4Orbiter arrives far too low over Mars
- 5It is lost in the atmosphere ✕
Units belong in the interface contract, not in someone's head.
Source: NASA Mishap Investigation Board, November 1999
No. 011 · Clock Room
· hours, depending on who you were
- 1A leap second is inserted at 23:59:60 UTC
- 2A Linux kernel timer bug makes some processes spin
- 3Java and MySQL servers pin their CPUs
- 4Reddit, Mozilla and others slow down or fall over
- 5Fix: set the clock again, or reboot ✕
Time is an input. Test the weird ones.
Source: Public postmortems and kernel mailing list, July 2012
No. 012 · Clock Room
· a long tail on old devices
- 1An old root certificate reaches its long-announced expiry date
- 2Modern systems use the newer root and don't notice
- 3Old phones, old OpenSSL and embedded devices still trust only the old one
- 4TLS handshakes start failing on those clients
- 5Fixes range from config changes to buying new hardware ✕
An expiry date years away is still a date. Put it in a calendar.
Source: Let's Encrypt announcements, 2020 to 2021