Fast but wrong, forever: the NO_EXPIRY cache and the bug that never expires

1. What a TTL was quietly protecting you from

Consider a cohort statistic — a player's rank against their peer group, an account's standing against a benchmark — computed by an expensive query and cached. With a TTL, if a deploy introduces a sorting bug that ranks everyone off by one, the wrong values live in the cache only until they expire; the next rebuild, assuming you fix the bug, is correct. The blast radius in time is the TTL. That is the property you were paying for and probably didn't notice.

Switch that cache to NO_EXPIRY and the same bug behaves completely differently. The wrong ranks are written once and never re-evaluated. Every user sees them. Fixing the code does nothing, because the code no longer runs against those keys — they're already populated. The only way to correct the data is to manually invalidate every affected key, which requires first knowing that it's wrong, which is exactly the thing nothing is telling you.

# NO_EXPIRY: fast, and permanently wrong if the writer was wrong.
cache.set(key, compute_cohort_ranks(tournament_id))  # no ttl

# The insurance a TTL buys you back — bounded blast radius:
cache.set(key, compute_cohort_ranks(tournament_id), ttl=3600)
# A broken deploy poisons at most one hour of data, then self-heals.

2. Invalidate-then-prefill, and the partial-failure window

NO_EXPIRY caches are kept fresh by an explicit two-step dance on every underlying change: invalidate the stale keys, then prefill the new values. The order is load-bearing — prefill before invalidate and you build tables just to delete them. But the deeper hazard is that these are two separate operations that can fail independently.

If invalidate succeeds and prefill fails, the cache is empty and every request pays the full uncached query cost — a latency cliff, but at least correct. If prefill runs but only partially completes, you get the truly insidious state: some keys hold fresh data, adjacent keys hold stale data, and a single request assembles a view from both. The user sees a report where half the numbers moved and half didn't, internally inconsistent, with no error anywhere.

3. Choose the failure you can detect

The engineering decision is not "TTL or NO_EXPIRY" — it is "which failure mode can I detect and afford." A TTL fails slow-but-correct and self-heals; its cost is tail latency and query load. NO_EXPIRY fails fast-but-wrong and persists; its cost is correctness you cannot see erode. For anything a customer will make a decision on, slow-but-correct is almost always the right trade.

If you genuinely need NO_EXPIRY for performance, you must manufacture the signal the TTL used to give you for free. Version every cache entry with the schema and deploy SHA that produced it, so a contract change misses cleanly instead of serving a stale shape. Emit the freshness of what you serve — the age of the underlying computation — as a metric, so "data hasn't refreshed since the bad deploy" becomes a graph line, not a customer email. And treat a failed or partial prefill as a paging event, because a permanent cache with a broken refresh is not a degraded system; it is a confidently wrong one.

# Version the key so a changed contract misses instead of serving a stale shape.
key = f"cohort:v{SCHEMA_VERSION}:{DEPLOY_SHA}:{tournament_id}"

# Emit the freshness of what you serve, so silent staleness becomes visible.
entry = cache.get(key)
metrics.gauge("cache.data_age_seconds", now_monotonic() - entry.computed_at)

# A partial prefill is not a warning to log — it is a page to answer.
if prefill.failed_tables:
    alert.page("cohort prefill partial", tables=prefill.failed_tables)