When web traffic spikes unexpectedly, an unprotected database will crash under the load.
Adding Redis as an in-memory cache is the standard solution. But basic caching implementations often introduce subtle failure modes that can be just as destructive as the traffic spikes themselves.
The most notorious of these is the Cache Stampede (or Dog-Piling Effect). When a heavily requested cache key expires under high concurrency, dozens of incoming requests experience a cache miss simultaneously. Every single worker process immediately hammers the primary database with identical heavy queries, knocking the database offline.
At the same time, naive rate limiters that execute multiple distinct Redis calls (GET, increment, and EXPIRE) suffer from race conditions and fail in clustered Redis environments with CROSSSLOT errors.
A few weeks ago, Amir and I redesigned the caching and rate-limiting infrastructure for an API serving ten thousand requests per second. Here is the exact architecture we deployed using probabilistic early expiration (the XFetch algorithm) and atomic Lua-scripted sliding window limiters.
Preventing Cache Stampedes with the XFetch Algorithm
Standard Cache-Aside waits until a key is 100% expired before querying the database.
The XFetch algorithm (a proven probabilistic optimal early expiration technique) computes whether a background worker should refresh the cache before it actually expires. The closer the key is to expiration, and the heavier the compute delta, the higher the probability that a random concurrent reader will recompute and refresh the value in the background:
Probability Test: -beta * delta * ln(random()) > (expiry - now)Here is the TypeScript implementation:
// lib/xfetch-cache.ts
import { Redis } from 'ioredis';
interface CacheEnvelope<T> {
value: T;
deltaMs: number; // Duration it took to compute the value
expiresAt: number; // Unix timestamp in milliseconds
}
export async function getOrComputeWithXFetch<T>(
redis: Redis,
key: string,
ttlSeconds: number,
computeFn: () => Promise<T>,
beta: number = 1.0
): Promise<T> {
const raw = await redis.get(key);
const now = Date.now();
if (raw) {
const envelope: CacheEnvelope<T> = JSON.parse(raw);
const timeRemaining = envelope.expiresAt - now;
// Probabilistic early expiration check
const shouldEarlyRefresh =
-beta * envelope.deltaMs * Math.log(Math.random()) > timeRemaining;
if (!shouldEarlyRefresh) {
return envelope.value;
}
}
// Measure computation time and refresh value
const start = Date.now();
const freshValue = await computeFn();
const deltaMs = Date.now() - start;
const newEnvelope: CacheEnvelope<T> = {
value: freshValue,
deltaMs,
expiresAt: now + ttlSeconds * 1000,
};
await redis.setex(key, ttlSeconds, JSON.stringify(newEnvelope));
return freshValue;
}Because only one background process triggers the refresh as expiration approaches, your database never experiences a stampede when the key reaches its nominal TTL.
Atomic Sliding Window Rate Limiting via Lua
To prevent API abuse, you need a sliding window rate limiter. However, writing separate ZREMRANGEBYSCORE and ZCARD calls from your application server introduces race conditions under high concurrency.
Executing a Lua script inside Redis guarantees atomicity. In clustered Redis environments, any multi-key operation must also force keys to the same cluster shard using Redis Cluster Hash Tags: {...}.
Here is the production Lua script:
-- scripts/sliding_window.lua
-- KEYS[1]: Rate limit key containing cluster hash tag, e.g. ratelimit:{user_123}:api
-- ARGV[1]: Current timestamp (milliseconds)
-- ARGV[2]: Window size in milliseconds
-- ARGV[3]: Max requests allowed in window
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local clearBefore = now - window
-- Remove old request records outside the sliding window
redis.call('ZREMRANGEBYSCORE', key, '-inf', clearBefore)
-- Count remaining requests in the active window
local currentRequests = redis.call('ZCARD', key)
if currentRequests < limit then
-- Add current timestamp as both score and member
redis.call('ZADD', key, now, now .. '-' .. math.random(1000, 9999))
redis.call('PEXPIRE', key, window)
return 1 -- Allowed
else
return 0 -- Throttled
endAnd here is the Node.js caller ensuring hash tag formatting:
// lib/rate-limiter.ts
import { Redis } from 'ioredis';
import fs from 'fs';
const luaScript = fs.readFileSync('./scripts/sliding_window.lua', 'utf-8');
export async function checkRateLimit(
redis: Redis,
identifier: string, // e.g. client IP or user ID
endpoint: string,
limit: number,
windowMs: number
): Promise<boolean> {
// Use curly braces {identifier} so Redis Cluster hashes to the same shard
const key = `ratelimit:{${identifier}}:${endpoint}`;
const now = Date.now();
const result = await redis.eval(luaScript, 1, key, now, windowMs, limit);
return result === 1;
}Key Namespace and Memory Eviction Hygiene
- Standardize Key Namespaces: Follow
cache:<domain>:<entity_id>for entities andratelimit:{<id>}:<endpoint>for throttles. - Explicit Eviction Policies: For caching clusters, configure
maxmemory-policy volatile-lruorallkeys-lruso Redis gracefully drops older keys if RAM fills up instead of throwing out-of-memory errors.
Comments
Comments are reviewed before appearing publicly.
No comments yet — be the first.