The shape
The instinct when sizing a cache is to compare it against the corpus, meaning everything you have stored. "We have 12 PB of images and 200 GB of cache, so we can hold 0.002 percent of it, therefore caching is hopeless." That reasoning is wrong, and it is wrong in a way that changes designs.
A cache is sized against the working set: the data actually being requested in a given window. In almost every real system the working set is a tiny fraction of the corpus, because access is heavily skewed toward recent and popular items. A cache that holds 0.002 percent of the bytes can easily serve 95 percent of the requests, and those two percentages have almost nothing to do with each other.
Stated generally: hit ratio is a function of what you are caching against, not how much you are caching. Two systems with identical cache sizes can have wildly different hit ratios purely because one has a concentrated working set and the other does not.
The same idea appears in several places that look unrelated:
- A CDN holding a small hot set and serving most requests from it while the origin servers hold everything.
- A database buffer pool that is a fraction of the table size but holds nearly every page anyone touches.
- An LRU in front of a key-value store, where memory is set by active users rather than registered users.
The thing to internalise: when someone asks whether a cache will work, the answer comes from the shape of the access distribution, not from the size of the data.
The mechanism
Working set size is measurable rather than a matter of intuition. It is the number of distinct items requested within a window, and the window is however long you can afford to keep something resident.
Two properties of that curve drive every decision here.
It is steep before the knee and flat after it. Going from a cache that holds half the working set to one that holds all of it is transformative. Doubling again past that point does close to nothing, because what remains is the long tail: items requested once and then not again, which no cache size defeats.
The knee moves with the window. A larger cache is equivalent to a longer retention window, and a longer window admits more distinct items. So sizing is really about choosing the window in which you expect a re-read, and that is a property of the product: a photo shared this morning gets re-read all day, an invoice from March does not.
A photo service. Three inputs:
everything stored, the originals plus
every resized copy made from them 12 PB
new objects uploaded per day 100M
size of the copy you actually serve 150 KB
Requests pile onto recent and popular objects. That is
the assumption an interviewer will push on, so take it
from access logs rather than guessing.
uploads in the last 48 hours
100M x 2 days 200M
what those uploads occupy
200M x 150 KB 30 TB
and 90% of all requests land inside that 30 TB
a further 8% land on an older popular set 10 TB
───────────────────────────────────────────────────
working set, serving 98% of requests 40 TB
total size of everything stored, in TB 12,000 TB
40 / 12,000 0.3%
or a ratio of 300 : 1
The other 2% is a long tail: objects read once and then
never again, which no amount of extra cache touches.
0.3% of the bytes answers 98% of the requests. The
12 PB never entered that calculation.The practical consequence: measure the working set before choosing a cache size, and be explicit about the assumption. The 90 percent figure above is the load-bearing claim in that whole calculation, and it is measurable from access logs rather than something to guess at.
How to tell you have this problem
Your hit ratio is low and adding memory is not helping. That means you are already past the knee, so the miss traffic is genuine long tail. More memory is the wrong lever; better admission or a longer TTL might be, or the misses might simply be irreducible.
Your hit ratio is low and nobody has measured the working set. The common case. Before buying more, count distinct keys requested per hour from a sample of access logs, then multiply by the average item size. If that total is bigger than your cache, or anywhere close to it, the cache is too small to hold the working set. Memory is exactly the right lever here.
Your hit ratio changed without your cache changing. Then the access distribution changed, which usually means a product change: a new surface that links to old items, a crawler, or a feature that shows people content from further back. The working set grew and the cache did not.
The measurement that answers all three is the same: distinct keys per window, and their total size. A sketch does this cheaply on a stream of keys without storing them, which is the natural pairing between these two patterns.
What it costs
The measurement is only valid for the window you measured. Working sets grow on a schedule: weekday against weekend, a campaign, a holiday. Sizing against a quiet Tuesday means being under-provisioned exactly when it matters.
A working set can be enlarged by your own product decisions. Adding a surface that links to archived content, or a feature that shows people their history, can multiply the working set overnight without any traffic increase. This is a design review question, not a sizing one.
Small hit ratio changes are large changes at your origin servers. Those are the machines behind the cache that hold everything and do the real work, and they never see the hit ratio. They see the misses. At 95 percent, 5 percent of requests reach them. At 90 percent, 10 percent do. So five points off the hit ratio means twice the traffic at the origin servers, and twice as many of them to serve it. The closer your hit ratio sits to 100, the more each single point is worth.
Eviction policy matters most exactly at the knee. When the cache roughly equals the working set, the policy decides whether you hold the right items. LRU is a reasonable default and it is defeated by a scan: one crawler walking your whole corpus evicts the entire working set. Scan-resistant policies exist for this and the cost of not having one is a sudden, unexplained hit ratio collapse.
Where it shows up in the course
| article | the form it takes there |
|---|---|
| Bloom Filters and Probabilistic Structures | admission control, where a filter stops one-hit wonders from evicting the working set |
| Caching Strategies and Invalidation | how much cache to buy, and the curve where more capacity buys latency and fragility together |
| CDNs and Edge Compute | cache key dimensions multiplying object count until the working set stops fitting in a PoP |
| the edge holds a small hot set against a 12 PB corpus, and the hit ratio it achieves is what sizes the fleet of origin servers behind it | |
| URL Shortener | 150 TB at rest against a working set of tens of gigabytes, which is the reason the whole design is cheap to run |
| YouTube | the extreme case: an exabyte library where a small recent and popular slice takes nearly all the requests |
Getting it wrong
Sizing against the corpus. The comparison that makes caching look hopeless and is almost never the relevant one.
Guessing the working set instead of counting it. It is a distinct-key count over a window, and it is cheap to measure continuously.
Treating hit ratio as an outcome rather than a design variable. It is set by size, admission, TTL and key design, all of which are choices. Origin cost then follows from it multiplicatively.
Ignoring scans. A crawler or a backfill job that reads every object you have will push the whole working set out of the cache on its way through. If your access pattern includes a scan, plan for it in the policy rather than discovering it on a dashboard.
Assuming the working set is stable. It moves with the product. A feature that surfaces old content is a sizing change even though nothing about traffic volume changed.