No. 004 · Wing of the Global Push

One Regular Expression

· 27 minutes

  1. 1New managed WAF rule is deployed globally at once
  2. 2Its regular expression backtracks catastrophically on ordinary traffic
  3. 3CPU on every edge machine hits 100%
  4. 4Sites behind the network return 502s worldwide
  5. 5Global kill switch for the WAF rules restores traffic ✕

The fastest deploy pipeline is also the fastest outage pipeline.

Source: Cloudflare blog, July 2019

Cloudflare regex outage, July 2019. Duration: 27 minutes. 5 hops from the first change to the failure.