Invariant is a small payment processor. Its whole point is a list of rules that must hold however the process dies: money is never created or destroyed, nobody is charged twice, every status change produces a webhook. Unit tests can check those rules after a request finishes. They can't check them after a request is cut off halfway, which is the case that matters.
So the last phase of the project was a test suite that kills the app. Not a mocked crash or a thrown exception. A real SIGKILL, so no finally block runs, no shutdown hook runs, and Postgres is left holding an open transaction from a client that no longer exists.
The app runs as a child process
The tests build the app and start dist/src/main.js as a child process on a free port, against the test database and Redis. Killing it is process.kill(pid, 'SIGKILL'). Restarting it is starting it again.
One snag: the expiry and reconciliation workers run every ten minutes, hard-coded, and a test can't wait ten minutes. The child is started with a preload, -r fast-timers.cjs, that swaps any setInterval of 600000 ms for 300 ms. The app code doesn't know.
Stopping at an exact statement
Killing a process at a random moment mostly tests nothing, because most moments are between requests. The interesting kills are inside a transaction, after some writes and before the commit. I didn't want to add if (process.env.CRASH_HERE) lines to money code to get there.
Postgres already has a way to make a transaction wait at a known point: a row lock. Approving a payment updates the store's account row last, right before commit. So the test opens its own connection and locks that row first.
await pg.query('BEGIN');
await pg.query(
'SELECT 1 FROM "Account" WHERE id = $1 FOR UPDATE', [storeAccountId]);
// fire the approve request, then wait until the app is stuck on our lock
await waitFor(() => pg.query(
"SELECT 1 FROM pg_stat_activity WHERE wait_event_type = 'Lock'"));
app.kill(); // the claim, debit, ledger row and outbox row are written
await pg.query('ROLLBACK');When pg_stat_activity shows the app waiting on a lock, it has already claimed the payment, debited the user and written the ledger and outbox rows, all uncommitted. Then it dies. The same trick works for top-ups, by locking the funding account, and for the expiry worker, by locking one of the payments it's about to expire.
There's one more wait after the kill. Postgres only notices a client is gone when it next reads from its socket, so for a moment the dead app's transaction still looks alive. The harness waits until no backend is idle in transaction or waiting on a lock before it checks anything.
The webhook case
The hardest crash for an outbox is after the HTTP request reaches the store and before the worker records that it did. The test webhook receiver handles that by killing the app the moment an event arrives. After restart, the event is still PENDING with a lease that hasn't expired. When the 60 second lease runs out, a worker picks it up again and sends it with the same event id, which is how the store is meant to recognise the duplicate.
The chaos loop
The last test kills the app eight times at random while four users and two stores create, approve and top up, payments expire after two seconds, and the receiver answers 10% of webhooks with a 500. After the last restart everything is allowed to drain, then the test checks:
- Every create, approve and top-up the app acknowledged before dying is in the database.
- Every top-up that got no answer, retried with the same idempotency key, applied exactly once.
- Every event id reached the receiver at least once.
- The reconciliation job reports zero drift, and the ledger and payment invariants hold row by row.
What it found
Nothing in src had to change, which was the result I was hoping for and not the one I expected. It found two smaller things. The production start script pointed at the wrong file, because the build output nests under dist/src. And a crash between the PIN attempt counter's INCR and its DEL burns one of the user's five attempts until the window expires. I left that one in and wrote it down. It costs a user one guess.
The suite takes about two and a half minutes. It's the slowest thing in the repo and the test I trust most.