Payment Webhooks in Node.js: How to Never Apply a Payment Twice
Payment providers resend webhooks, and a handler that runs twice charges, ships or emails twice. Signature checks, Redis claims, locks and a reconciliation job that close the gap.
Abhijeet Maharana · 8 min read
- Node.js
- Payments
- Redis
- E-commerce
Short version: payment providers send the same webhook more than once. Verify the signature on the raw body, claim the event ID in Redis with
SET NXbefore doing any work, put a unique index under it, lock refunds and cancellations, and run a reconciliation job for the webhooks that never arrive. Most of this comes from an e-commerce platform I built end to end. This post walks through each piece.
Why does the same webhook arrive twice?
Every payment provider retries. If your server does not answer with a 2xx quickly enough, the provider assumes the delivery failed and sends it again. Stripe, for example, keeps retrying a failed event for up to three days.
That sounds harmless until you look at what can make a delivery "fail":
- Your handler did the work but took too long, so the provider timed out and resent it.
- A deploy restarted the server after the work was done but before the response went out.
- A load balancer or proxy dropped the response.
In all three cases the payment was already applied. The retry applies it again. Depending on what your handler does, that means a second order, a second shipment, a second confirmation email, or a customer credited twice.
The handler has to be safe to run many times with the same event. The usual name for that is idempotent.
The naive handler
This is what most first versions look like:
app.post("/webhooks/payments", express.json(), async (req, res) => {
const event = req.body;
if (event.type === "payment.captured") {
await markOrderPaid(event.data.orderId);
await createShipment(event.data.orderId);
await sendConfirmationEmail(event.data.orderId);
}
res.sendStatus(200);
});
It has two problems. It trusts whoever calls the URL, and it does all the work again on every retry.
Step 1: verify the signature on the raw body
Providers sign each webhook with a shared secret, usually an HMAC-SHA256 of the request body. Two details catch people out.
First, the signature is computed over the exact bytes the provider sent. If express.json() parses the body first and you call JSON.stringify on it again, the bytes can change and the check fails. Read the raw body on this route.
Second, compare with a constant-time function. A normal string compare returns early at the first different character, which leaks timing information.
import { createHmac, timingSafeEqual } from "node:crypto";
app.post("/webhooks/payments", express.raw({ type: "application/json" }), async (req, res) => {
const expected = createHmac("sha256", process.env.WEBHOOK_SECRET!).update(req.body).digest();
const received = Buffer.from(req.header("x-signature") ?? "", "hex");
// timingSafeEqual throws on buffers of different length, so check that first.
if (received.length !== expected.length || !timingSafeEqual(received, expected)) {
return res.sendStatus(401);
}
const event = JSON.parse(req.body.toString("utf8"));
// ...
});
The header name and the exact signed string differ by provider. Stripe signs a timestamp plus the body and its SDK checks it for you with stripe.webhooks.constructEvent. If your provider has an SDK helper, use it.
Step 2: claim the event before doing any work
Every webhook carries an event ID that stays the same across retries. Use it as the key of a claim in Redis. SET with NX only writes the key if it does not exist yet, and it does that in one atomic step, so two copies of the same event arriving at the same moment cannot both win.
const claimed = await redis.set(`webhook:${provider}:${event.id}`, "1", "EX", 86_400, "NX");
if (!claimed) return res.sendStatus(200); // already handled: acknowledge and stop
await applyPaymentEvent(event);
res.sendStatus(200);
Return 200 for a duplicate. If you return an error, the provider keeps retrying something you already handled.
On the platform I built, every payment and shipping webhook goes through this: signature check, then the Redis claim, then the work. A retried webhook finds the key taken and stops.
What if the work fails after the claim?
This is the part most examples skip. If you claim the event and then applyPaymentEvent throws, the key is still there. The provider retries, finds the claim, and your code acknowledges an event it never finished. The payment is lost silently.
Two ways to handle it:
- Release the claim on failure. Wrap the work in
try/catch, delete the key in thecatch, and return a 500 so the provider retries. - Claim in two states. Write
processingwith a short expiry (say five minutes), then overwrite it withdoneand the long expiry when the work succeeds. A crash mid-way lets the key expire, and the next retry runs the work again.
const key = `webhook:${provider}:${event.id}`;
if (!(await redis.set(key, "processing", "EX", 300, "NX"))) return res.sendStatus(200);
try {
await applyPaymentEvent(event);
await redis.set(key, "done", "EX", 86_400);
res.sendStatus(200);
} catch (err) {
await redis.del(key);
res.sendStatus(500);
}
Step 3: put a unique index under it
Redis is the fast guard, not the last one. A key can expire, Redis can be flushed, and a retry can come after your TTL. So make the database refuse the duplicate too. A unique index on the provider's payment ID means a second insert fails, however it got there:
CREATE UNIQUE INDEX payments_provider_payment_id ON payments (provider, provider_payment_id);
Then write the payment row and the order status change in one transaction. If the insert hits the unique index, the whole thing rolls back and nothing is applied twice.
Step 4: lock the actions people can click twice
Webhooks are not the only source of duplicates. An admin double-clicks "Refund". A customer taps "Cancel order" twice on a slow connection. A cron job overlaps with a manual action.
On the platform, cancellations, refunds, referral payouts and the daily cron each take a short Redis lock. The lock value is a random token, and releasing it is a compare-and-delete in a Lua script. Without the token, a request whose lock already expired could delete a lock that another request now holds.
import { randomUUID } from "node:crypto";
const release = `
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
end
return 0`;
async function withLock<T>(name: string, ms: number, work: () => Promise<T>) {
const token = randomUUID();
if (!(await redis.set(`lock:${name}`, token, "PX", ms, "NX"))) {
throw new Error("Already in progress");
}
try {
return await work();
} finally {
await redis.eval(release, 1, `lock:${name}`, token);
}
}
await withLock(`refund:${orderId}`, 10_000, () => issueRefund(orderId));
Keep the lock time longer than the slowest normal run of the work. The lock only makes a double run unlikely, so keep the work itself idempotent too.
Step 5: reconcile the webhooks that never arrive
The opposite failure is just as expensive. The customer paid, but the webhook never came: the provider had an incident, your server was down for longer than the retry window, or a firewall rule changed. The order sits as unpaid while the customer waits.
The fix is not to trust webhooks alone. On the platform, a delayed job and a daily cron check every order whose payment confirmation never arrived. Both ask the payment gateway for the real status and apply it through the same idempotent code path as the webhook. A dropped webhook turns into a short delay, not a support ticket.
Two related rules:
- The server decides the price. Checkout recomputes every total on the server. Never trust the amount the browser sends.
- Scheduled work runs from a scheduler, not a
setIntervalinside your API process. A timer in the web server dies on every deploy. I used EventBridge Scheduler calling a small Lambda; a cron on a worker machine works too.
A checklist for your payment webhooks
- Signature verified on the raw body, with a constant-time compare
- Event ID claimed atomically before any work (
SET NXwith an expiry) - Claim released, or left to expire, when the work fails
- Unique index on the provider's payment ID as the last guard
- Payment row and order status written in one transaction
- Refunds, cancellations and payouts behind a lock with a token
- A reconciliation job for orders whose confirmation never arrived
- Prices computed on the server
- 200 for duplicates, 5xx only for real failures you want retried
None of this is much code. Most of it is deciding, up front, that every callback from the outside world is untrusted and may arrive twice, late, or never. The case study shows how these pieces fit into the rest of the platform: queues, rate limits, shipping carriers and the ops dashboard.