Graceful Degradation

The shape

Something you depend on is unavailable, late, or stale. You have two options: fail the whole request, or return an answer that is missing a part. This pattern is about the second option, and about the fact that it only works if you decided which part beforehand.

The decision is cheap before the incident and impossible during it. That asymmetry is the entire reason this is a pattern rather than an instinct. At 3am, with a dashboard full of red, nobody is going to reason their way to "the recommendation carousel is optional but the price is not." That conclusion has to be sitting in the code already, written by somebody who was calm.

Notice how many superficially different decisions are the same move underneath. Choosing which reads may be stale and by how much. Choosing which parts of a page are allowed to be missing. Choosing to fall back to a local rate limit when the shared counter store is unreachable. Choosing which operations may proceed during a network partition. Choosing to serve an expired cache entry rather than nothing. All of these are one question asked in five places: which part of this answer is optional?

The ladder

Degradation is not binary. There is a sequence of things to give up, and the useful discipline is to write it in order, because that ordering is the design.

The observation worth sitting with: most systems have rung one and rung eight, and nothing else. So a single late dependency takes the whole request from perfect to nothing, and the six intermediate answers that users would have happily accepted were never available, because nobody built them.

Where the parts come from

Four sources of degradation, roughly in order of how often they are the right answer.

Optional sections. A page has a required core and a set of enhancements. Recommendations, related items, activity counts, personalisation. Each enhancement gets a deadline, and a contributor that misses its deadline is dropped rather than waited for. This is the highest-value version because it makes the request's latency the core's latency rather than the slowest contributor's.

Stale instead of absent. An expired cache entry is almost always better than an error. Serve it, mark it, and refresh in the background. The setting people forget is the second timeout: how stale is too stale to serve at all, which is a different number from the TTL and should be much larger.

Approximate instead of exact. Counts, rankings, recommendations, and quota checks can usually tolerate approximation. A rate limiter that has lost its shared data store can fall back to a per-instance limit. That is wrong in aggregate and much better than either failing open or failing closed.

Local instead of shared. When a shared dependency is unreachable, a local approximation of it is often available. Each instance's own view, a bundled snapshot, a last-known-good config. The fallback is worse and it is present.

The rule that prevents cascades

There is one form of degradation that is not about answers, and it belongs here because getting it wrong turns a small failure into an outage.

A component at its capacity limit should shed or spill, not die. This matters most in any tier where load is routed by key: a cache ring, a sharded store, a consistent-hash-routed service. In those tiers a node's death does not remove its load, it relocates it onto a neighbour, which makes the neighbour more likely to die, which relocates more load. That is a cascade. It is why the correct behaviour at the ceiling is to reject cheaply, spill to the next node, or serve something degraded, rather than to fall over.

The same logic applies to a replica that has fallen behind. A lagging replica whose health check passes will happily serve data from 40 minutes ago. The health check asks whether the process is alive and not whether the answer is current. Liveness and correctness are different questions, and a read pool should eject a replica on a staleness threshold, automatically, without a human deciding during an incident.

A page assembled from 1 core call and 4 optional contributors,
each independently slow (above its p99) 1% of the time.

Wait for everyone
  P(at least one is slow)   1 − 0.99^5        = 4.9%
  → ~1 request in 20 is as slow as the worst
    of 5 services

Drop any contributor past a 50 ms deadline
  P(the page is slow)       = P(the CORE is slow)
                            = 1.0%
  → the page's latency is now the core's latency

  ─────────────────────────────────────────────────────
  4.9% → 1.0%, and the cost is that about 4 percent of
  page views are missing one optional section.

The question is not technical. It is whether a missing
recommendation strip is worse than a page that takes
2 seconds, and product owns that answer.

How to tell you need this

Here's the symptom. If a page calls 6 services and every single one has to succeed, your page's availability is the product of all six, not close to any one of them. Say each of those 6 services is up 99.9 percent of the time on its own. The chance all 6 are up at the same time is 0.999^6, about 99.4 percent, so your page ends up less reliable than any single service it depends on.

The design-review tell is simpler: ask what happens if a named dependency returns nothing, and see whether the answer is specific. "It would fail" is honest. "It would fail, and that is correct because the data is required" is a decision. "I am not sure" means the ladder above has one rung.

What it costs

Three costs, and the pattern is usually described as though it had none. In order: a second code path, answers you cannot see are degraded, and the operations that must never degrade at all.

Every degraded mode is a second code path, and therefore a second set of behaviours to test, monitor and reason about. Two paths that both work are more than twice the work of one path that works, because you also own the transition between them.

Degraded answers need to be visible, or you will serve them silently for months. A stale answer with no timestamp, or a partial page with no indication a section is missing, produces support tickets nobody can reproduce. Emit a metric per degraded mode and alert on the rate, not just on the errors it replaced.

And the honest one: some things must not degrade. A balance check before a withdrawal, an authorisation decision, a payment. Naming those explicitly is as much a part of this pattern as naming the optional parts, and a design that degrades everything is as badly considered as one that degrades nothing.

Where it shows up in the course

Getting it wrong

Having only two rungs. Full answer or error. Every intermediate response has to be built deliberately, and if none exist then every dependency is effectively required.

A fallback nobody has run. Untested degradation paths fail when invoked, which is precisely the worst moment. Exercise them on a schedule.

Degrading silently. If a degraded answer looks identical to a good one, you will ship it permanently by accident. Mark it, measure it, alert on the rate.

Failing open on something that protects you. A rate limiter that lets everything through when its store is down has degraded into an absence. Fall back to a local approximation instead, so the protection is worse rather than gone.

Treating a health check as a correctness check. A process that is alive is not a process whose data is current. These need separate signals.

Degrading the thing that mattered. Some answers have no acceptable approximation. The list of what must never degrade is part of this pattern, not an exception to it.