500 requests a second are hitting a dependency that is completely dead. Every single one gets a clean timeout after 2 seconds, exactly as configured. And you are still down.

That sentence is the whole problem with treating resilience as a single setting. A timeout did its job, nothing hung forever, and the aggregate is still an outage, because you're holding 1,000 concurrent doomed requests at once, and every one of them is load on a dependency that's trying to restart.

There isn't one pattern that fixes this. There are three, stacked, each one picking up exactly where the last one's guarantee runs out. Get the order wrong, or skip one, and you don't get partial protection, you get a different failure mode wearing the same symptoms.

This is that stack: timeouts and retries, circuit breakers and bulkheads, backpressure and load shedding. What each one actually bounds, why the one before it isn't enough, and the specific way each gets misconfigured in a way that looks fine until the day it doesn't.

Layer One: A Timeout Doesn't Wait Patiently, It Fails Together

Most client libraries default to no overall timeout. fetch in Node has no total deadline unless you add one. Plenty of database drivers wait indefinitely. That default isn't neutral, it's a decision to couple your availability to your slowest dependency, and Little's Law explains exactly how much:

A dependency's p99 goes from 50ms to 30 seconds. Your arrival rate hasn't changed. Your concurrency just rose by a factor of 600, and every one of those in-flight requests is holding a socket, a pool connection, a request slot, memory. At 200 requests/second and a 30-second service time, you need 6,000 concurrent slots. You don't have 6,000.

Nothing failed. A dependency got slow, and the absence of a bound propagated it outward. A timeout is how you convert someone else's latency problem into your error rate, which sounds like a downgrade and isn't, because an error is bounded and recoverable, while unbounded latency spreads.

The queue at a single service desk, where one customer's transaction is taking forty minutes. With no policy, the twenty people behind them wait forty minutes, the next twenty leave, and the shop's throughput collapses over one difficult case. With a policy, "five minutes per customer, then take a ticket and come back", one customer is inconvenienced and the queue keeps moving.

Only one of your four timeouts actually bounds anything

A single number labelled "timeout" is usually not the number you think it is:

• Connect bounds the TCP handshake. Catches a host that's down or unroutable.
• Time to first byte bounds server thinking time. Catches a slow query or a saturated server.
• Idle / socket bounds the gap between bytes. Catches a stream that stalls mid-response.
• Total / overall bounds the whole operation, including retries. This is the only one that actually protects your resource usage.

A 2-second connect timeout and a 5-second read timeout don't add up to a guarantee, a response trickling one byte every 4 seconds never trips either one. Set the overall deadline, always, and treat the other three as diagnostics that let you fail earlier with a clearer reason why.

Choose the number from the p99.9 of the successful response distribution, not the average. Too short and you abandon requests that would have succeeded, converting them into retries, a load amplifier disguised as a safety measure. Too long and you hold resources through a failure you could have detected sooner.

And here's the arithmetic that catches nearly everyone:

A retry policy has to fit inside the overall budget: per-attempt timeout is remaining_budget / max_attempts, not a number picked independently.

Not every failure is safe to retry, and a status code isn't proof

Connection refused, DNS failure, 503, 502, safe to retry, nothing happened yet. A 429, safe, and honour Retry-After. A 500 or a timeout, safe only if the operation is idempotent, because both are the canonical ambiguous failure: the request may have already succeeded server-side and you just never heard back. A 400, 422, 404, never retry; identical bytes fail identically.

HTTP method semantics are a hint, not a guarantee. GET and PUT are specified idempotent, and plenty of real handlers aren't, a GET that increments a view counter, a PUT that appends. Decide from what the operation does and whether it carries an idempotency key, not from the verb.

Backoff needs jitter, and jitter isn't a refinement

Fixed-interval retries fail for a specific reason: a thousand clients that failed at the same moment retry at the same moment.

Exponential backoff fixes the frequency of retries. It does nothing for synchronisation. Without jitter, a recovering dependency gets knocked over by the first aligned burst, which restarts the whole cycle.

Cap the exponential, bound the attempt count, and make the first retry near-immediate for a plain connection refusal, that failure costs the dependency nothing to answer again.

Retry amplification: the incident that outlives its own cause

Here's the part that turns a thirty-second blip into a two-hour outage.

Retries multiply through layers. A request path where every layer retries three times:

Each layer is individually reasonable. Together they're an 81× amplifier, and it engages exactly when the deepest component is failing, because that's what triggered the retries in the first place. A database at a 50% error rate, hit with three times its normal load, doesn't recover. It produces more errors. Which produce more retries. Which produce more errors.

This is a metastable failure state: one that sustains itself after its trigger is gone. A 30-second network blip causes a wave of retries. The retries push load above capacity. Being above capacity produces timeouts, which produce more retries. The network has been fine for an hour, and the system does not recover, because the load keeping it down is the load its own failure generated. Removing the original cause changes nothing. The only way out is reducing load: shedding traffic, opening circuit breakers, or manually pulling the service out of rotation until queues drain.

Two rules follow from this. Retry at one layer, not at every layer, pick the one closest to the business logic, the one that holds the idempotency key, and make every other layer pass failures straight through. If a service mesh retries for you, the application must not also retry, or you've silently rebuilt the multiplier.

And bound it structurally with a retry budget, this is the single most valuable thing in this entire layer, and the mechanism most systems lack:

With a per-request retry count, a 100% error rate produces 3× or 81× load. With a 10% retry budget, that same 100% error rate produces 1.1× load, the system fails fast and cheap, and leaves the dependency enough headroom to actually recover.

Why this matters in production: the trigger for these incidents is almost always mundane, a failover, a deploy, a brief partition. The duration is set entirely by whether the system can shed the load its own retries created. A retry budget and a circuit breaker cost about an afternoon each to build, and they're the difference between a five-minute blip and a two-hour incident.

Cancellation has to actually cancel

A client-side timeout does not stop the server. This is the detail that surprises people the most, and it matters most at the database:

A Node query timeout abandons the response. The Postgres backend keeps executing the query, holding the connection, the snapshot, and its locks.

So a client-side timeout on a slow query gives you the worst of both worlds, the client has moved on and may retry, doubling the work, while the server runs the original query to completion anyway. Enforcement has to happen server-side:

Layer Two: A Breaker Stops Making the Calls At All

Layer one bounds each call. It does not stop you from making a thousand doomed calls a second.

Go back to that dependency that's genuinely down, 2-second timeout, 500 requests/second arriving:

The timeout did its job. Each request failed in bounded time. The aggregate is still an outage, you're spending your entire concurrency budget discovering, five hundred times a second, something you already knew.

A circuit breaker is the observation that after enough failures, you can just stop asking. It converts a 2-second failing call into a sub-millisecond local rejection, and that does three things at once: it frees your resources instantly, it removes load from the dependency, giving it room to actually recover, which is the exit from the metastable state above, and it fails fast enough that a fallback becomes viable. A 2-second wait before serving a cached value is a bad experience. A 1-millisecond rejection followed by a cache read is a fine one.

That second point is easy to undervalue and is often the decisive one. A dependency at 100% error rate does not recover while it's still receiving full traffic plus retries. A breaker is how the traffic actually stops, without a human doing it by hand.

The electrical breaker the pattern is named after doesn't protect the appliance that shorted, that appliance is already broken. It protects the rest of the house, by isolating the one circuit so the wiring doesn't overheat and every other room keeps its lights on. And it needs a delay before reclosing, because closing it immediately onto an unfixed short achieves nothing but another trip.

The four ways a breaker gets misconfigured

Trigger on a failure rate over a rolling window, not consecutive failures. A consecutive-failure counter fails in both directions, on a low-traffic endpoint, five consecutive failures might span twenty minutes and open a breaker for a problem long since resolved, and on a dependency where half of calls succeed, a consecutive counter never reaches five at all. Use a rate with a minimum-volume guard, so a single failure after a quiet period doesn't read as a "100% failure rate over a sample of one":

A 4xx is not a failure. This is the misconfiguration that causes the most damage in practice. A 400, 404, or 422 means the dependency is healthy and rejected your request, bad input, a missing resource, failed validation. Count those as breaker failures and a single client bug, a URL scanner, or one malformed integration takes down a perfectly healthy dependency for every other caller.

Threshold on slow calls too, not just errors. A dependency at 100% success and 8 seconds per call is doing just as much damage as one that's down, arguably more, because nothing about it looks like an error. A slow-call-rate threshold ("if 60% of calls exceed 1 second, open") is the check most implementations skip entirely.

Half-open must admit a trickle, not a flood. A cooldown that expires and lets the full 500 requests/second back through at once re-hammers a recovering dependency and reopens the breaker instantly. Admit one to three concurrent probes, require a few successes before closing, and ideally ramp: half-open at 5% of traffic, then 25%, then closed.

Bulkheads: stop one dependency from starving everything else

A ship's hull is divided into watertight compartments so one breach floods a single compartment, not the vessel. The system equivalent: partition the resource requests compete for, so one dependency's slowness can't consume all of it.

In Node there's no thread pool to partition, which leads some people to conclude bulkheads don't apply. They do, the shared resource is event-loop time, the socket pool, and memory held by pending promises. The mechanism is a semaphore that rejects when full, not one that queues without bound:

A semaphore that queues indefinitely isn't a bulkhead, it's a delay, and the memory held by that queue is exactly the resource you were trying to protect. Size it from Little's Law: a dependency with a 50ms p99 sustaining 200 requests/second needs 10 concurrent slots, not 200. And the single highest-value bulkhead most teams skip is separate connection pools per workload, web traffic and a reporting job should never draw from the same Postgres pool.

Degradation is design work, and it hides the failure that follows

Breakers and bulkheads decide when to stop calling something. Degradation decides what the user gets instead, and it's the part that can't be configured away, it needs a decision per dependency, often a product decision rather than an engineering one.

The rule that catches the most bugs: the fallback must not depend on what failed.

Redis going down doesn't just remove the cache, it redirects the entire cache hit rate onto the database. A cache at a 95% hit ratio failing means the database sees 20× its normal read load, and the fallback has just converted a cache outage into a database outage. The correct version serves the degraded path behind a bulkhead and a request-coalescing lock, and sheds the excess rather than manufacturing a second failure.

And here's the trap specific to this layer, worth sitting with: degradation removes failure from your error rate. Once it's working, a failing dependency produces no errors. Requests succeed. Latency might even improve, because a fast fallback replaced a slow call. Your dashboard is flat. Your alerts are quiet.

So a system can serve the generic homepage instead of the personalised one for three weeks, because a breaker opened after a deploy changed a hostname and never closed, and nobody notices, because nothing is red. The first report comes from a product manager asking why engagement dropped. The fix is instrumenting the thing degradation hides: breaker state as a metric with an alert on "open for longer than N minutes," and a degraded-serve rate tracked as its own SLI, right next to your error rate, because it is the error rate you decided not to show users.

Layer Three: When Nothing Is Broken and You're Still Down

Layers one and two both answer the same question: a dependency I call is broken, what do I do? This layer answers a different one entirely.

Everything downstream is healthy. Every query is fast. And 9,000 requests a second are arriving at a service that can serve 6,000.

No breaker opens, because nothing is failing. No bulkhead helps, because no single dependency is being monopolized, the resource is being consumed by legitimate, healthy work. Queues just grow, latency climbs uniformly across everything, and eventually it all times out at once.

The counterintuitive fact underneath this: capacity isn't a ceiling you bump into. It's a knee, after which things get worse, not just full. Throughput is requests completed per second. Goodput is requests completed per second that anybody still wanted, inside the caller's deadline. Past the knee, throughput can look almost flat while goodput falls off a cliff, because the work still being completed is work whose caller gave up on seconds ago. A service can sit at 100% CPU, complete 6,000 requests a second, and deliver a genuinely useful 400, and its throughput graph will look perfectly fine the entire time.

A kitchen that takes every order the floor brings. At 30 covers it's fine. At 90, tickets pile up, and by the time each dish is plated the table has left. The kitchen is working flat out, food is going out at the maximum rate the stoves allow, and nobody is eating it. The fix isn't a bigger ticket rail, it's the maître d' at the door saying "we're full, forty-five minute wait." That refusal is the only intervention that gets the guests who are seated actually fed.

An unbounded queue is not a buffer

"Add a queue so we can absorb the spike" is correct advice for a transient burst and catastrophic advice for sustained overload. The arithmetic is Little's Law again:

Every one of those 50,000 items gets processed eventually. Every single one gets processed after its caller has already timed out. The system does the full amount of work and delivers none of the value, and the memory holding that queue is itself now a failure mode.

Worse, the queue changes the shape of the failure into a less useful one. Without it, request 6,001 gets an immediate 503, the client knows right away, can retry with backoff, can fall back, can tell the user something. With an unbounded queue, request 6,001 gets accepted and times out 30 seconds later, after the client has waited and held its own resources for nothing, learned nothing sooner, and now retries, adding a second item to the queue for work you'd already started.

So: every queue is bounded, and the bound comes from the latency target you actually care about.

Serving at 500/s with a 2-second latency target means a queue of 1,000 items, not one more. Beyond that, reject. This applies to your HTTP accept queue, your connection-pool wait queue, your job queue, every in-process channel. An unbounded queue anywhere in the request path is exactly where the latency will accumulate.

Backpressure where you can, shedding where you can't

Backpressure means the consumer tells the producer to slow down, and the producer can. It's strictly better wherever it's available, because no work gets discarded, the rate just matches. TCP flow control is the original version of this idea; HTTP/2 flow-control windows, reactive streams, bounded channels where the producer blocks, and consumer pause/resume on a Kafka worker are all re-implementations of it. In Node, writable.write() returning false is backpressure, ignoring that return value and writing anyway is the standard way to leak memory in a pipeline.

Backpressure works inside a closed system, your own pipeline, code you control. It fails completely at an open boundary: you can't tell a million browsers, a partner's integration, or a mobile app fleet to send fewer requests. There's no window to shrink. At that boundary, the only option left is load shedding, refuse work immediately and cheaply, so the work you do accept actually completes. Shedding isn't a failure of design. It's the design. The choice was never "shed or serve everyone", it's "shed deliberately, or let the overload choose for you by timing everything out instead."

Shed on the signal that tells the truth earliest

Not every signal is equally honest, and the ranking matters:

Queue wait time is the best signal, it is the latency you're about to violate, and it leads everything else. Concurrency in flight versus your limit is next, direct and cheap. Queue depth is good but needs a service rate to interpret. Event-loop delay in Node measures the actual contended resource. CPU utilisation is mediocre, saturation isn't linear in CPU, and I/O-bound work can show low CPU while queueing badly. Latency p99 lags, by the time it moves, the queue is already deep. And error rate is the worst signal of all: it's the outcome you were trying to prevent in the first place, arriving last.

For Node specifically, event-loop delay is an excellent, cheap, local signal:

And a shed response is only cheap if it happens early, reject before authentication, before deserialising a large body, before touching the database. A 503 that costs as much to produce as a 200 protects nothing at all.

The Order Matters, Not Just the Presence

Put together, the three layers cover three genuinely different failures, and none of them substitutes for another:

A timeout bounds one call to a dependency that's slow. It does nothing about the volume of calls you keep making to one that's dead, that's what a circuit breaker stops. A breaker does nothing about legitimate, healthy traffic simply exceeding your own capacity, that's backpressure and shedding. And underneath all three sits the retry logic that, misconfigured, turns any one of these into a self-sustaining outage regardless of how well the other two are built.

This is also, not coincidentally, the fuller version of a claim made in passing in an earlier piece on connection pooling in serverless environments: that platform retry policies, re-invoking every failed serverless request during a connection exhaustion event, are "one of the few failure modes where client retries are strictly harmful." That's retry amplification, in exactly the shape described above, a thousand concurrent invocations failing at once, each one retried by the platform, adding load to a connection table whose entire problem was already too many clients. A retry budget at the right layer, or a breaker that stops the calls outright, is the actual fix; raising max_connections again is not.

None of these three layers is optional once you're running anything with a real dependency graph. But they answer different questions, they fail in different ways when misconfigured, and the order in which you reach for them, bound the call, stop the calls, then shed what you can't backpressure, is the order that actually holds under load.

Sourced from the System Design In-Depth course, Timeouts, Retries, and Backoff, Circuit Breakers, Bulkheads, and Graceful Degradation, and Backpressure and Load Shedding.