Engineering Atlas › Problem Solving & Incidents (AI era)

GitLab Database Deletion (2017)

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Timeline (UTC, from GitLab's postmortem)

Technical root cause

The trigger was a destructive command run on the wrong host during a stressful, late-night recovery. The reason it caused data loss rather than an annoyance was that the safety nets had holes. The pg_dump job ran on an application server using PostgreSQL 9.2 tools against a 9.6 database, so it failed (PostgreSQL docs: pg_dump refuses newer servers); the failure emails were rejected because DMARC was not set up for cron emails, so nobody knew. Disk snapshots covered NFS servers, not databases. LVM snapshots existed for staging, not for disaster recovery. Replication is not a backup: it faithfully copies deletions, and here it had been wiped as part of the repair (Google SRE book, Data Integrity).

Contributing factors

Detection and mitigation

The deletion was noticed immediately by the engineer. The real "detection" problem was earlier and silent: backups had been failing for some time with no working alert. Mitigation was the restore itself, chosen as the least-bad option among broken backups.

What changed afterwards

GitLab's action items included hourly LVM snapshots and Azure disk snapshots for database servers, automated testing of backup restores, a backup monitoring dashboard, point-in-time recovery with continuous WAL archiving, better replication runbooks, sensible max_connections, an owner for data durability, and a redesign of abuse reporting so account removals are not triggered so easily.

What I would have done as the tech lead

Before: name an owner for data durability and require a scheduled, automated restore test with alerting on the result, not on the backup job's exit status. Enable WAL archiving and point-in-time recovery. Make destructive commands on database hosts require typing the host name, and colour-code production prompts. During: stop and hand off when the person doing recovery has been fighting the incident for hours; destructive steps in a recovery should be read aloud and confirmed by a second person. After: GitLab's transparency is worth copying, with names redacted, as they later decided.

Down the stack

Layer 1The contract

GitLab.com's implied contract was that user data was durably stored and recoverable. Their backup setup looked redundant on paper, five mechanisms, but none of them had a verified contract like "we can restore the production database to within 1 hour of any point in the last 7 days, in under 2 hours". Without a tested restore, a backup is an untested hypothesis.

Layer 2The mechanism

PostgreSQL streaming replication sends write-ahead log records from the primary to replicas. The primary keeps only a limited amount of WAL (controlled at the time by wal_keep_segments); if a replica falls further behind than that and there is no archive to fetch older segments from, it cannot catch up and must be rebuilt from a fresh base backup with pg_basebackup, which copies the whole data directory. pg_dump produces a logical backup and must be run by a version of the tools at least as new as the server, which is exactly what failed here. Point-in-time recovery needs a base backup plus continuous WAL archiving, which GitLab did not yet have (PostgreSQL docs).

Layer 3The machine

Recursively removing a data directory unlinks the files; for a database whose files are open, the space may not be freed until the process closes them, but the directory entries are gone and recovery from the file system is not practical. The 18-hour restore was dominated by copying hundreds of gigabytes from slow network-attached staging disks in another region, a throughput problem: at around 5 to 10 MB/s, 300 GB takes 8 to 17 hours. Fast restores need fast storage and a nearby copy, which is why restore time must be measured, not assumed (Google SRE book, Data Integrity).

Why it's built this way

  1. Backups exist to be restored, so only restore tests prove they work.
  2. Silent failure is the default for unattended jobs, so success must be actively verified.
  3. Replication protects against machine loss, not against deletions, so it is not a backup.
  4. Stressful late-night recoveries make mistakes likely, so destructive steps need friction and a second person.
  5. Ownership drives testing; without an owner, nothing gets verified.

Diagram

flowchart TD
  L["Load spike: spam and account removals"] --> R["Replica falls behind, no WAL archive"]
  R --> RB["Rebuild replica, pg_basebackup hangs"]
  RB --> W["rm -rf run on primary by mistake"]
  W --> D["~300 GB deleted"]
  D --> B{"Backups?"}
  B --> B1["pg_dump: silently failing for days"]
  B --> B2["Azure snapshots: not on DB servers"]
  B --> B3["Replica: wiped"]
  B --> B4["LVM snapshot: 6 h old, manual"]
  B4 --> RS["18 h restore from slow staging disks"]

Worked example

Apply it to your own team. You have nightly pg_dump backups to object storage, a streaming replica, and daily volume snapshots. List each one as GitLab should have: what it protects against, last success, last restore test, alert on failure, and restore time.

The audit finds: pg_dump succeeds but has never been restored; the dump of the 600 GB database takes 5 hours to restore. The replica protects against host loss only. Volume snapshots succeed, but the last restore test was 14 months ago. Alerts for backup failure go to an email list nobody reads.

Changes: WAL archiving with point-in-time recovery (RPO about 1 minute); a weekly automated restore of the latest base backup plus WAL to a scratch instance, running row-count checks and alerting on failure through the paging system; a named owner on the database team; and a host prompt that shows PROD-PRIMARY in red. The first automated restore test fails, because the archive bucket's lifecycle policy deleted WAL older than 3 days, shorter than the 7-day base backup interval. Exactly the kind of gap that would only have shown up on the worst day.

In the wild

Trade-offs

MechanismProtects againstDoes not protect againstRestore speed
Streaming replicaHost failureDeletions, corruption (copied instantly)Seconds (failover)
Logical dump (pg_dump)Deletion, corruptionData since last dumpSlow for large databases
Disk or volume snapshotsDeletion, corruptionData since last snapshotMedium
Base backup plus WAL archive (PITR)Deletion, corruption, to any pointLoss of the archive itselfMedium, depends on replay
Restore testsUndetected broken backupsNothing by themselves; they prove the others workn/a

Pitfalls in production

Leadership lens

Check yourself

  1. Why did GitLab's replica not help recover the deleted data?

    Show answer

    Replication protects against losing a machine, not against data loss on the primary; in this case the replica had also been cleared as part of the rebuild.

  2. Why did the pg_dump backups fail silently?

    Show answer

    They ran with PostgreSQL 9.2 tools against a 9.6 server and failed, and the failure emails were rejected because DMARC was not configured, so nobody was alerted.

  3. What single organisational gap did GitLab identify as the root cause of the backup failures?

    Show answer

    Nobody owned backups, so nobody was responsible for testing that restores worked.

  4. Why did the restore take about 18 hours?

    Show answer

    The usable snapshot was on slow staging storage in another region, so copying hundreds of gigabytes took most of that time.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →