Tailscale Tracks Down a 16-Year-Old SQLite WAL-Reset Bug

Over six months, Tailscale's control plane suffered 19 separate incidents of database corruption, all traced to a single bug deep inside SQLite. In a detailed blog post, the company explains how it worked with the database's core developers to find and fix the “WAL-Reset bug”, a rare data race estimated to have been present in SQLite for at least 16 years.

Single-writer shards and aggressive checkpoints

Tailscale's control plane is split into coordination shards, each holding its own SQLite database accessed exclusively by a single Go process — the single-writer design SQLite is built for. The company has used SQLite as its primary database since 2022, taking a full snapshot every few minutes and uploading it to S3.

In August last year, a pipeline reading those backups reported an error, and PRAGMA integrity_check confirmed the file was corrupted. The incident repeated: 19 times in total over six months. Each one forced a shard's control plane offline — initially for more than an hour — so devices coming online could not connect, and the admin console and API were unavailable.

Chasing an unreproducible fault

No single shard, customer, feature, or load level correlated with the corruption, and the team could not reproduce it. Tailscale signed a professional support contract with the SQLite developers and deployed forensic telemetry in production. A transaction-logging pipeline, which replays every modifying SQL statement onto a known-good backup, provided the first real clue: in two cases, data committed by one transaction was invisible to later transactions.

The WAL-Reset bug

The SQLite team built tmstmpvfs, a shim around the virtual filesystem layer, to trace checkpoints. The logs revealed a rare data race between a checkpoint and a write transaction: the checkpoint believes pages have been copied from the write-ahead log into the main database file when they have not, losing that data permanently and leaving indexes pointing at missing pages. The fix, released as SQLite 3.51.3, adds a check that detects when the WAL has been reset by another thread.

A first attempt, 3.52.0, had to be withdrawn after it triggered false corruption alarms across 13 Tailscale databases through stale expression indexes. Tailscale reduced its timestamp precision to integer seconds; SQLite later added a self-healing index feature. A two-month wait for a diagnostic warning ended with an alert proving the race still fires in production — followed by four months without incidents.