Two workers, one order

FluxBuy is a flash-sale backend. A seller lists a fixed quantity that goes on sale at a set time, and everyone tries to buy at once. Getting it to hold 900 concurrent buyers against 900 units took a few rewrites, and the last real bug wasn't in the hot path at all. It was between two workers that run after the buyer already has their answer.

How an order moves

Stock lives in Redis. When an order comes in, a Lua script checks the stock key and decrements it in one step, so two buyers can't both see the last unit:

local stock = tonumber(redis.call("get", KEYS[1]) or 0)
local want = tonumber(ARGV[1])
if stock >= want then
  return redis.call("decrby", KEYS[1], want)
else
  return -1
end

If that succeeds, the order is written to Postgres as PENDING, a job goes onto a BullMQ payment queue, and the buyer gets a response in milliseconds. The payment worker picks the job up and calls the mock provider, which takes one to four seconds and fails 20% of the time on purpose. A failed charge puts the stock back.

Jobs can get lost. The process can die mid-charge, or a job can sit in the queue too long. So a cleanup worker runs every ten seconds, finds orders stuck for more than a minute, settles them, and returns their stock.

The bug

Both workers finished an order the same way: write the final status by order id. Nothing checked what the status was before.

Picture an order whose payment job is still waiting in a slow queue when it passes the one minute mark. Cleanup marks it FAILED and puts the stock back. Then the payment job finally runs, charges the card, and writes PAID. The buyer has paid for a unit that has also gone back on sale. Run it the other way round and stock gets restored twice. Either way the count is wrong.

A status as the lock

I didn't want a second system to coordinate the two, like a Redis lock per order. That's another thing to keep in sync, and another thing to lose. The order row already has a status. I added one value to it: PROCESSING, meaning a worker has claimed this order.

Every worker now claims before it acts, with a conditional update, and gives up if the claim didn't match:

const { count } = await prisma.order.updateMany({
  where: { id: order.id, status: 'PENDING' },
  data: { status: 'PROCESSING' },
});
if (count === 0) return; // cleanup already took it

const result = await payments.chargeCard(/* ... */);

await prisma.order.updateMany({
  where: { id: order.id, status: 'PROCESSING' },
  data: { status: result },
});

Cleanup does the same in reverse. It only fails orders that are still PENDING, or PROCESSING and older than the cutoff, and it restores stock only for the rows its update actually returned. One line in the cleanup worker still carries the comment // <- guard was missing next to the status: 'PROCESSING' condition on the PAID branch, because that's where it went wrong.

Making the race happen

A race you only reason about isn't fixed. scripts/order-race-test.js starves the payment queue on purpose so jobs sit past cleanup's one minute window while still queued, which forces both workers onto the same orders.

In the run I kept, 400 buyers went after 400 units. Cleanup swept 127 orders while their jobs were still queued. You can spot those afterwards because they're FAILED with no payment row ever created. The payment worker reached all 127 later, lost the claim on every one, and did nothing. Final count: 192 units left and 208 paid, which is 400.

The same idea came back in my next project, a payments ledger, where every money movement is a conditional update with a row count checked after it. If the update matched zero rows, someone else got there first, and the only correct move is to stop.