Tailscale在过去六个月内遭遇多次数据库故障，经过数月调查，最终与SQLite开发者联手发现了问题所在——一个至少存在16年的罕见数据竞争缺陷，被称为WAL-Reset bug[1]。这一bug在特定时刻的写入操作与checkpoint过程冲突时被触发，最终导致数据丢失和数据库损坏[1]。
Tailscale从去年年底至新年期间共经历了19次独立的数据库损坏事件[1]。为了定位这一根深蒂固的问题，Tailscale部署了新的调试工具tmstmpvfs shim[1]。调查发现，Tailscale采用非标准方式手动控制checkpoint过程并执行激进的checkpoint策略，这使得缺陷更容易被触发[1]。
SQLite官方在3.51.3版本中发布了针对WAL-Reset bug的修复[1]。Tailscale随即部署该版本，经过四个月的无故障运行确认问题已得到解决[1]。两个月后收到的警报进一步确认WAL-Reset bug确实在生产环境中发生过[1]。
Tailscale has identified the root cause of repeated database failures that plagued its systems over the past six months: a race condition in SQLite that has existed for at least 16 years.[1] The bug, known as the WAL-Reset bug, occurs when write operations collide with checkpoint processes at specific moments, resulting in data loss and database corruption.[1]
The company experienced 19 independent database corruption events between late last year and the new year, prompting a months-long investigation conducted jointly with SQLite developers.[1] Tailscale's non-standard approach to manually controlling checkpoint operations and its aggressive checkpoint strategy made the system particularly susceptible to triggering this latent defect.[1] To locate the problem, Tailscale deployed a new debugging tool called tmstmpvfs shim, which helped pinpoint the elusive bug.[1]
SQLite addressed the issue in version 3.51.3, which contained the WAL-Reset fix.[1] An earlier version, 3.52.0, had attempted to resolve the bug but introduced a separate rounding problem related to expression indexes, leading developers to release version 3.51.3 with only the WAL-Reset correction.[1] After deploying the patched version, Tailscale has operated for four months without database incidents, and alerts received two months into this period confirmed that the WAL-Reset bug had indeed occurred in production.[1]