Webhooks are the connective tissue of modern SaaS integrations. Stripe uses them to announce subscription renewals, GitHub uses them to trigger CI builds, and Shopify uses them to sync orders.
Despite their ubiquity, webhooks are notoriously difficult to implement securely and reliably.
On the ingestion side, developers frequently verify HMAC signatures using basic string comparisons like sig === expectedSig, exposing their servers to timing attacks. On the delivery side, external customer endpoints fail constantly: servers crash, certificates expire, and networks timeout. If you do not handle retries and idempotency properly, you end up double-charging customers or flooding partners with duplicate events.
A few weeks ago, Amir and I built a mission-critical billing and fulfillment webhook system. Here is the exact production architecture we used to verify signatures in constant time, rotate signing secrets with zero downtime, and guarantee idempotent processing.
Constant-Time HMAC Signature Verification
When a webhook arrives, an attacker can exploit the time it takes for a standard string comparison (===) to evaluate characters sequentially, gradually guessing valid signature bytes.
To eliminate timing attacks, compute an HMAC-SHA256 digest and compare the raw buffers using crypto.timingSafeEqual:
// lib/webhook-verifier.ts
import crypto from 'crypto';
interface VerifyOptions {
rawBody: Buffer;
signatureHeader: string; // Format: "t=1726135200,v1=9b1deb4d..."
secret: string;
toleranceSeconds?: number;
}
export function verifyWebhookSignature({
rawBody,
signatureHeader,
secret,
toleranceSeconds = 300,
}: VerifyOptions): boolean {
// Parse header parts
const parts = signatureHeader.split(',').reduce<Record<string, string>>((acc, item) => {
const [k, v] = item.split('=');
if (k && v) acc[k] = v;
return acc;
}, {});
const timestamp = parseInt(parts['t'], 10);
const receivedSignature = parts['v1'];
if (!timestamp || !receivedSignature) {
return false;
}
// Reject replay attacks outside tolerance window
const now = Math.floor(Date.now() / 1000);
if (Math.abs(now - timestamp) > toleranceSeconds) {
return false;
}
// Compute expected HMAC over "timestamp.rawBody"
const payloadToSign = `${timestamp}.${rawBody.toString('utf-8')}`;
const hmac = crypto.createHmac('sha256', secret);
const expectedSignature = hmac.update(payloadToSign).digest('hex');
const expectedBuffer = Buffer.from(expectedSignature, 'utf-8');
const receivedBuffer = Buffer.from(receivedSignature, 'utf-8');
if (expectedBuffer.length !== receivedBuffer.length) {
return false;
}
// Constant-time buffer comparison
return crypto.timingSafeEqual(expectedBuffer, receivedBuffer);
}Notice that we sign ${timestamp}.${rawBody}. This prevents replay attacks: even if an eavesdropper captures a valid payload and signature, the timestamp becomes invalid after the tolerance window expires.
Zero-Downtime Secret Rotation
When security policies mandate rotating webhook secrets, you cannot simply change the key in environment variables: older events in flight or webhooks sent before clients update their secrets will fail.
Support dual-secret verification during the rotation window:
export function verifyWithRotation(rawBody: Buffer, header: string, activeSecret: string, oldSecret?: string): boolean {
// Try active secret first
const isValid = verifyWebhookSignature({ rawBody, signatureHeader: header, secret: activeSecret });
if (isValid) return true;
// Fallback to retiring secret during transition window
if (oldSecret) {
return verifyWebhookSignature({ rawBody, signatureHeader: header, secret: oldSecret });
}
return false;
}Transactional Idempotency Constraints
Because network timeouts happen between sending a webhook and receiving a 200 OK response, the sender will inevitably retry the delivery.
Never execute business actions without recording the unique webhook delivery ID inside your database transaction:
-- PostgreSQL Ingestion Ledger
CREATE TABLE received_webhooks (
event_id TEXT PRIMARY KEY,
sender TEXT NOT NULL,
event_type TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('PROCESSING', 'COMPLETED', 'FAILED')),
received_at TIMESTAMPTZ DEFAULT now() NOT NULL,
completed_at TIMESTAMPTZ
);In your handler:
// api/webhooks/incoming.ts
import { db } from '../../lib/db';
export async function handleIncomingWebhook(eventId: string, eventType: string, payload: any) {
// Step 1: Atomically claim event or detect duplicate
const claim = await db.query(
`INSERT INTO received_webhooks (event_id, sender, event_type, status)
VALUES ($1, 'stripe', $2, 'PROCESSING')
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;`,
[eventId, eventType]
);
if (claim.rowCount === 0) {
// Already ingested or currently processing. Return 200 OK to stop retries.
return { status: 'duplicate_acknowledged' };
}
try {
// Step 2: Execute actual business logic (e.g. fulfill order)
await processOrderFulfillment(payload);
// Step 3: Mark completed
await db.query(
`UPDATE received_webhooks SET status = 'COMPLETED', completed_at = NOW() WHERE event_id = $1`,
[eventId]
);
return { status: 'processed' };
} catch (error) {
await db.query(
`UPDATE received_webhooks SET status = 'FAILED' WHERE event_id = $1`,
[eventId]
);
throw error;
}
}Returning 200 OK on duplicates stops the sender from hammering your server, while the unique database constraint guarantees you never process a transaction twice.
Outgoing Delivery: Exponential Backoff and Dead Letters
When delivering webhooks to customers:
- Use Exponential Backoff: Retry intervals like 15s, 1m, 5m, 30m, 2h, 8h, 24h.
- Set Hard Timeouts: Abort HTTP connections after 5 seconds to prevent hanging sockets.
- Route to Dead-Letter Queues (DLQ): If an endpoint fails repeatedly for 48 hours, mark the subscription disabled and notify the customer via email.
Comments
Comments are reviewed before appearing publicly.
No comments yet — be the first.