Invariant Ledger
A small payment processor: stores charge users through a hosted checkout and hear back by webhook. The crash tests kill the process mid-payment, then check that every balance still adds up.
- Role
- Solo, backend
- Timeline
- Sep – Oct 2026
- Stack
- NestJS · PostgreSQL · Redis · Vitest
- Status
- Phases 1–3 done
Eight invariants
Users top up a fake balance. Stores create a payment with an API key, the user approves it on a hosted checkout page, money moves, and the store is told by webhook. The balances are fake. The system still has to behave as if they were real.
Every test and every review was checked against a fixed list of rules. Four of them:
- Conservation: money is only moved. The sum of every balance, the funding account included, is zero.
- At most once: a payment is charged once, however many approvals arrive at the same moment.
- No overdraft: a user balance never goes below zero.
- Every status change owes a webhook, written in the same transaction as the change.
- 333
- End-to-end
tests - 8
- Random kills
per chaos run - 0
- Reconciliation
drift
One transaction
Approve is a sequence of conditional updates
Claim the payment only if it is still pending and not expired. Deduct the user only if the balance covers it. Write one ledger row and one outbox row. Credit the store last, because every payer touches that row. Zero rows matched at any step means stop and roll back.
Webhooks go through an outbox
The event row is committed with the payment, so it exists if and only if the payment does. A polling worker claims rows with a lease, sends them signed with HMAC, and fences every write-back on the lease token so a slow worker can't overwrite the one that replaced it.
Retries, then a dead letter
Backoff from 5 seconds doubling to an hour, with jitter. After 34 attempts, about a day, the event is dead-lettered, the store gets one alert, and the store can replay it with the same event id.
Reconciliation reads, it doesn't fix
A job runs four checks inside one repeatable-read snapshot: conservation, cached balance against the ledger, payment against transfer against event, and the direction of every transfer. Drift is reported with ids.
Killed on purpose
The crash suite runs the built app as a child process and sends it a real SIGKILL at known points: mid-approval, mid-top-up, mid-expiry-batch, and after a webhook is sent but before it is marked delivered. To stop the app at an exact statement without adding fault-injection code, a separate Postgres connection holds a row lock the app's transaction needs, and the harness waits until the app is blocked on it before killing it.
A chaos loop then kills the app eight times at random under mixed load with a receiver that fails 10% of deliveries. Afterwards every acknowledged write is there, every retried top-up applied exactly once, every event reached the receiver, and reconciliation reports zero drift. Nothing in src had to change.
Where it landed
Phases one to three are done: money moves correctly, stores find out reliably, and both survive the process dying. 333 end-to-end tests run against a real Postgres and Redis, because a mocked database can't show a race. The image builds in CI and is being deployed to AWS on EC2, RDS and ElastiCache.
Phase four is the store's balance row, which every payer locks. The plan is to credit stores off the request path, since a store balance only ever goes up, and to measure it before and after.