No. 007 · The Enter Key Gallery

rm -rf, Wrong Terminal

· about 18 hours; roughly 6 hours of data lost

  1. 1Late-night fight with database replication lag
  2. 2Engineer deletes a data directory to reseed a replica
  3. 3The shell was on the primary, not the replica
  4. 4The backup methods turn out to be broken or empty
  5. 5Restore from a staging snapshot, livestreamed to the public ✕

A backup you have never restored is a hope, not a backup.

Source: GitLab postmortem, February 2017

GitLab database incident, January 2017. Duration: about 18 hours; roughly 6 hours of data lost. 5 hops from the first change to the failure.