testing

How KvSQL Was Tested With Power Cuts, Killed Processes and 30000 Statements Checked Against SQLite

A database makes one promise above the rest. When it says a commit is done, the data is there, even if the power goes a moment later. And if the power goes during a commit, the file holds the whole transaction or none of it, never part. Most of KvSQL’s tests are aimed at that promise, from several sides.

All the figures below come from one recorded run of the project’s test script, scripts/record-results.sh, on a Linux container with two virtual CPUs, on October 22, 2026. The full output is in the repository under test/results/latest. CI runs the same script on Linux and macOS on every change.

How KvSQL was tested, counts from one recorded run on a log scale: 20,000 simulated power cuts with 10,514 inside a commit; 30,000 random SQL statements checked against SQLite with every answer matching; 6,666 SQL writes under power cuts with indexes exact every time; 200 real writer processes killed with SIGKILL, with 9,736 acknowledged commits and none lost; 100 kvsqld servers killed under load, with 13,390 confirmed writes and none lost.

The main tests and their counts in one recorded run. Every check passed.

Power Cuts on a Simulated Disk

A real disk doesn’t fail neatly. Writes that weren’t synced yet may land, or not, or land in a different order, or land half written. The test suite has a simulated disk that does all of that. It cuts the power after a random number of writes and syncs, then builds the file the disk could hold afterwards. It might keep every unsynced write, keep none, keep any subset, or keep subsets with some 512-byte sectors torn.

The recorded run did this 20,000 times, and 10,514 of the cuts landed inside a commit. After each one the file was reopened and checked page by page. It had to hold exactly the last finished commit, or the one in flight if it made it whole. Half the rounds also held a reader open across the commits, and its view had to stay the same; that was checked 16,673 times. A second test ran 6,666 SQL writes on a table with two indexes under the same kind of cuts. After each, every index had to match its table exactly.

Tests that pass are only worth something if they can fail. During development, bugs were planted on purpose: the sync before the meta page removed, pages reused while the last commit still reached them, freed pages reused while a reader still needed them. The power-cut tests caught each one at once.

Killed Processes

Simulations can share a blind spot with the code they test, so real processes get killed too. One test starts a real writer process, lets it commit as fast as it can, and kills it with SIGKILL at a random moment. It did that 200 times in the recorded run. Every one of the 9,736 commits the writer had reported as done was in the file afterwards.

The server gets the same treatment under load. Four clients send SQL transfers, row inserts and raw key writes to a real kvsqld process, and the process is killed mid-stream. Then the file is opened directly. Across 100 kills, all 13,390 writes that had received an HTTP 200 were there, the file verified, and the money total moved by the transfers never changed.

Every SQL Answer Checked Against SQLite

The SQL side is tested against SQLite itself. A generator writes random statements: inserts, updates, deletes, filters, joins, GROUP BY, ORDER BY and comparisons between text and numbers. It runs each one on KvSQL and on SQLite with STRICT tables, then compares the rows, the change counts and whether both succeeded or both failed. The recorded run checked 30,000 statements, and every one matched. Some WHERE clauses are steered toward key and index lookups so the planner’s shortcuts get checked too. An off-by-one in a key range, planted on purpose, was caught.

An Outside Pair of Eyes

Before 0.1.0, a separate reviewer that hadn’t seen the code went through it looking for bugs. It confirmed eight. Deeply nested SQL could crash the server. A failed key-value write could leave an index out of step. Some indexed lookups disagreed with a full scan. Opening a small file that wasn’t a database overwrote it. Each was fixed with a test that fails without the fix, and the reviewer re-ran its checks to confirm.

Some things aren’t proven yet. The power-cut tests run on a simulated disk, and the killed-process tests prove what survives a crash of the program, which is not the same as pulling the plug on a real machine. Real disks that ignore sync requests can break any database’s promise, KvSQL’s included.