Redis caching and invalidation that actually works

Most caching bugs we get called in to fix are not performance problems. They are correctness problems that show up only under load, for one customer, and nobody can reproduce them. The cache was added in an afternoon, it worked, and six months later the support queue is full of "my cart shows the old price". Here is how we build Redis caching that stays correct.

Pick a write strategy on purpose

Cache-aside reads Redis, and on a miss reads Postgres and populates. Writes go to the database and invalidate the entry. It is the default because it fails safe: if Redis dies, every read is a miss and you degrade to database load instead of serving garbage. The cost is a cold-start penalty per key and the fact that cache and database can disagree.

Write-through puts the cache in the write path. It earns its place for small, hot, high-fanout data: feature flags, pricing tiers, a tenant config blob that 400 requests per second read and one admin edits weekly. It adds latency to every write and needs a story for partial failure, because a successful Redis write with a failed Postgres commit is a lie you serve until the TTL runs out.

Write-behind buffers writes and flushes asynchronously. Fast, and it loses data when a node dies. We use it for view counters and last-seen timestamps, never for anything a customer would notice missing.

The race that makes naive cache-aside wrong

Request A misses, reads price = 100, then stalls 30ms on GC. Request B writes price = 120 and sets the cache to 120. Request A wakes up and writes its stale 100. That value is wrong until the TTL expires.

Two rules fix it. Delete on write, do not update on write: a delete is idempotent and cannot resurrect a stale read. And if you read from Postgres replicas, add a delayed double-delete: delete the key, commit, then schedule a second delete 500ms to 1s later to catch anything a reader pulled from a lagging replica in between. We run that second delete through the same worker queue as our outbox events so it survives a restart.

Key naming is a schema

app:v3:user:{user_id}:cart
app:v3:product:{sku}:detail:{locale}
app:v3:category:{slug}:tree

• Embed a schema version (v3). When the payload shape changes, bump it in config and the whole old generation goes cold in one deploy. No migration, no scan, no stale JSON deserializing into a null field.
• Never put unbounded user input in a key. A raw search query is a memory exhaustion vector. We cap free text at a 16-char sha1 prefix.
• Keep keys short. Redis costs the key string plus roughly 50 to 90 bytes of overhead per entry. A 40k-key namespace at 80 bytes is about 6MB before a single value is stored. On a 512MB managed instance that is real money.

TTL is the safety net, not the mechanism

"Cache forever and invalidate perfectly" is a trap. You will miss a path: a migration script, an admin panel, a Stripe webhook that predates the cache layer. The TTL limits the blast radius of the bug you have not found yet.

Tier by volatility: session and cart 30 minutes, product detail 10 minutes, category tree and nav 24 hours. Then add jitter. Populate 8,000 product keys during a deploy with an identical 600s TTL and they all expire in the same second. We use plus or minus 10 percent, enough to smear expiry across a minute.

Stampede protection

When a hot key expires and 900 concurrent requests miss it, all 900 hit Postgres with the same query. We have watched a healthy database go from 12 percent CPU to pool exhaustion in four seconds because one category key rolled over during a spike. Three defenses:

1. Single-flight lock. SET lock:key 1 NX PX 5000. One caller rebuilds; the rest wait briefly or return stale.
2. Stale-while-revalidate. Store a logical expiry inside the payload and a longer physical TTL. Past logical expiry, serve stale and rebuild in the background.
3. Probabilistic early expiration (XFetch). Each reader rolls a dice weighted by rebuild cost and proximity to expiry, so one unlucky request refreshes early while the rest stay warm.

import asyncio, json, random, time
from redis.asyncio import Redis

async def cache_aside(r: Redis, key: str, ttl: int, loader, jitter=0.1, beta=1.0):
    raw = await r.get(key)
    if raw is not None:
        env = json.loads(raw)
        gap = env["expires_at"] - time.time()
        if gap > beta * (env["build_ms"] / 1000) * random.expovariate(1.0):
            return env["value"]

    lock = f"lock:{key}"
    if not await r.set(lock, "1", nx=True, px=5_000):
        if raw is not None:
            return json.loads(raw)["value"]   # serve stale; someone is rebuilding
        await asyncio.sleep(0.05)
        again = await r.get(key)
        if again is not None:
            return json.loads(again)["value"]

    try:
        started = time.perf_counter()
        value = await loader()
        physical = int(ttl * (1 + random.uniform(-jitter, jitter)))
        envelope = {
            "value": value,
            "expires_at": time.time() + physical,
            "build_ms": (time.perf_counter() - started) * 1000,
        }
        # Physical TTL outlives logical expiry so stale-while-revalidate works.
        await r.set(key, json.dumps(envelope), ex=physical * 2)
        return value
    finally:
        await r.delete(lock)

Tag-based invalidation

Sooner or later one write must invalidate 40 keys across three templates. Maintain a tag index: one Redis set per tag holding dependent keys. On populate, SADD tag:product:SKU-1183 app:v3:product:SKU-1183:detail:en. On invalidate, SMEMBERS the tag, UNLINK members in batches of 500, then UNLINK the set. UNLINK frees memory on a background thread; DEL on a 10k-member set blocks the event loop long enough to show up in your latency graph.

Never run KEYS * in production. It is O(N) and it blocks; use SCAN with COUNT 500. Budget for the index too: tag sets run 5 to 8 percent of cache memory in our deployments, and they leak unless each gets a TTL slightly longer than its longest member.

Knowing your cache is correct

Hit rate tells you the cache is being used. A 99 percent hit rate on stale data is worse than no cache.

• Shadow reads. On 1 percent of traffic, fetch from cache and source, compare, log mismatches with both payloads. We caught a broken invalidation on discounted variants this way, three days before a sale.
• A ?nocache=1 debug path behind staff auth. Support answers "stale or genuinely wrong" in one request.
• Metrics: hit rate per namespace, miss latency p95 (your worst-case user), evicted_keys per minute, used_memory against maxmemory. Rising evictions while hit rate looks fine is the warning before the cliff.
• Eviction policy is a decision. allkeys-lru when Redis is pure disposable cache. volatile-ttl when the same instance holds rate limiters or job locks that must not vanish. The default noeviction turns a full instance into write errors.

# Hit ratio, evictions, memory pressure
redis-cli INFO stats  | grep -E 'keyspace_(hits|misses)|evicted_keys'
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|fragmentation'

# Walk a namespace without blocking the server
redis-cli --scan --pattern 'app:v3:product:*' --count 500 | head -50

# What one entry costs, and how hot it is (OBJECT FREQ needs an LFU policy)
redis-cli MEMORY USAGE app:v3:product:SKU-1183:detail:en
redis-cli OBJECT FREQ  app:v3:product:SKU-1183:detail:en
redis-cli TTL          app:v3:product:SKU-1183:detail:en

redis-cli --bigkeys -i 0.01   # sampled scan for fat keys

What never goes in a cache

Authorization decisions. Cache a role if you must, but re-evaluate permissions per request; a cached "yes" outliving a revoked role is a security incident, not a stale page. Payment and subscription state lives in Postgres and Stripe, read at the decision point.

Decide where the cache sits, too. Redis in front of the database helps every consumer including background jobs. CDN caching is far cheaper per request but only helps anonymous responses. Most storefronts need both.

How we work

On NorthStackHub projects, caching is designed alongside the data model, not bolted on at the end. We define the key schema and every invalidation path up front, wire metrics before the first key is written, and keep TTLs short enough that a missed invalidation heals itself in minutes. On a recent catalog API, moving to a versioned namespace with tag invalidation and single-flight locks took p95 from 820ms to 190ms and halved the Postgres instance at renewal. The number we watch afterward is not hit rate. It is stale-data tickets in the client's inbox.