Resilience Pattern
Resilience is a system's ability to continue providing an acceptable level of service when components fail, become slow, overloaded, or temporarily unavailable—and to recover gracefully when the underlying problem is resolved.
A resilient system does not assume that failures can be eliminated. Instead, it assumes:
Failures will happen; the architecture must contain, absorb, and recover from them.
This is especially important in distributed systems because a request may cross many independently failing components.
1. Why resilience is necessary
Consider a simple request path:
┌─────────────┐
│ Client │
└──────┬──────┘
│
▼
┌─────────────┐
│ API Gateway │
└──────┬──────┘
│
▼
┌─────────────┐
│ Order │
│ Service │
└──────┬──────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
┌──────────┐ ┌─────────┐ ┌─────────┐
│ Payment │ │Inventory│ │Shipping │
│ Service │ │ Service │ │ Service │
└──────────┘ └─────────┘ └─────────┘Suppose the Payment Service becomes slow.
Without resilience:
Payment slows
│
▼
Order requests wait
│
▼
Threads/connections consumed
│
▼
Order Service becomes slow
│
▼
API Gateway requests queue
│
▼
Clients retry
│
▼
Even MORE load
│
▼
Cascading failureThis is one of the most important concepts in resilience engineering:
Cascading failure
A failure in one component propagates into otherwise healthy components.
Resilience patterns attempt to break this propagation path.
2. The major resilience patterns
A useful way to categorize them is:
| Pattern | Primary purpose |
|---|---|
| Timeout | Prevent waiting indefinitely |
| Retry | Recover from transient failures |
| Circuit Breaker | Stop calling an unhealthy dependency |
| Bulkhead | Isolate failures/resources |
| Rate Limiter | Control incoming/request rate |
| Load Shedding | Reject work when overloaded |
| Fallback | Provide degraded functionality |
| Backpressure | Prevent producers overwhelming consumers |
| Health Checks | Detect unhealthy instances |
| Load Balancing | Distribute work across instances |
| Queueing | Absorb temporary traffic spikes |
| Idempotency | Make retries safe |
| Caching | Reduce dependency load |
| Graceful Degradation | Continue with reduced functionality |
These patterns are generally combined, rather than used individually.
3. Timeout
A timeout limits how long a service waits for another operation.
Without a timeout:
Service A
│
│ request
▼
Service B
│
│ ............ never responds
│
└───────────────────────────────►
waiting foreverWith a timeout:
Service A
│
│ request
▼
Service B
│
│
│ 2 seconds
│◄────────────────── timeout
│
▼
Return failure / fallbackFor example:
PaymentService.call()
│
├── timeout = 2 seconds
│
▼
Payment ProviderIf the provider doesn't respond within two seconds, the caller stops waiting.
Why it matters
Without timeouts, slow dependencies can consume:
- threads
- connection pools
- CPU
- memory
- request slots
Eventually the caller can fail too.
Important principle
Every remote call should have an explicit timeout.
That includes:
- HTTP
- gRPC
- database calls
- message brokers
- external APIs
4. Retry
Retries are useful when failures are transient.
Examples:
- temporary network failure
- connection reset
- HTTP 503
- temporary database unavailability
Basic retry:
Request
│
▼
┌─────────┐
│ Service │
└────┬────┘
│
failure
│
▼
Retry
│
▼
┌─────────┐
│ Service │
└────┬────┘
│
success
▼
DoneBut blindly retrying is dangerous.
Suppose 1,000 clients are already generating heavy traffic:
1,000 requests
│
▼
Service
│
X failure
│
▼
1,000 retries
│
▼
Service
│
X
│
▼
1,000 more retriesThis can produce a retry storm.
Exponential backoff
Instead of retrying immediately:
Attempt 1 ──X
│
└── wait 100 ms
Attempt 2 ──X
│
└── wait 200 ms
Attempt 3 ──X
│
└── wait 400 ms
Attempt 4 ──X
│
└── wait 800 msTypically:
delay = base × 2^attemptwith a maximum delay.
Add jitter
If thousands of clients fail simultaneously, deterministic backoff can cause synchronized retries.
Instead:
retry delay =
exponential backoff
+ random jitterThis spreads requests over time.
When NOT to retry
Do not automatically retry:
- validation errors
- authentication failures
- authorization failures
- most 4xx responses
- operations that aren't safely repeatable
Retries also need idempotency for operations such as payments or order creation.
5. Circuit Breaker
The Circuit Breaker is one of the most important resilience patterns.
It prevents repeatedly calling a dependency that is known to be unhealthy.
Think of it like an electrical circuit breaker.
Normal operation
Client
│
▼
┌──────────────┐
│ Circuit │
│ CLOSED │
└──────┬───────┘
│
▼
Payment ServiceRequests flow normally.
Failure threshold
Suppose Payment Service starts failing:
Request ──X
Request ──X
Request ──X
Request ──X
Request ──X
│
▼
failure threshold
│
▼
Circuit OPENNow:
Client
│
▼
┌──────────────┐
│ Circuit OPEN │
└──────┬───────┘
│
├──────► Fail fast
│
└──────► FallbackThe application doesn't even call the unhealthy service.
This protects both sides:
Without circuit breaker:
Application ──► failing dependency
Application ──► failing dependency
Application ──► failing dependency
Application ──► failing dependency
↓
wasted resources
With circuit breaker:
Application
│
▼
Circuit Breaker
│
X
│
Fail fastCircuit breaker states
A conventional circuit breaker has three states:
failures exceed threshold
┌───────────────────────────┐
│ ▼
┌──────────┐ ┌────────┐
│ CLOSED │ ──────────────► │ OPEN │
└────┬─────┘ └────┬───┘
▲ │
│ │ timeout
│ ▼
│ ┌────────────┐
└──────────────────── │ HALF-OPEN │
success └─────┬──────┘
│
test request
│
┌──────────┴──────────┐
│ │
success failure
│ │
▼ ▼
CLOSED OPENCLOSED
Normal traffic flows.
OPEN
Requests are rejected immediately or routed to a fallback.
HALF-OPEN
After a recovery period, a small number of requests are allowed through to test the dependency.
6. Bulkhead
The Bulkhead pattern isolates resources so that failure in one area doesn't consume everything.
The name comes from compartments in a ship.
If one compartment floods, the entire ship doesn't necessarily sink.
Without bulkhead
Imagine one shared thread pool:
Application
│
┌───────┴───────┐
│ Shared Pool │
│ 100 threads │
└───────┬───────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
Payment Search Shipping
│
becomes slow
│
▼
consumes 100 threads
│
▼
EVERYTHING
becomes slowWith bulkheads
Application
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Payment │ │ Search │ │ Shipping │
│ 30 slots │ │ 50 slots │ │ 20 slots │
└──────────┘ └──────────┘ └──────────┘If Payment consumes all 30 slots:
Payment ██████████████████████████████ 30/30
Search █████████████████████████ 35/50
Shipping ███████████ 11/20
Payment failure
↓
contained hereSearch and Shipping can continue operating.
Bulkheads can isolate
- thread pools
- connection pools
- CPU
- memory
- queues
- tenants
- workloads
- database connections
7. Rate Limiter
A rate limiter controls how many requests are accepted within a period.
For example:
Maximum:
100 requests / secondArchitecture:
Clients
│
│ 1,000 req/s
▼
┌─────────────────┐
│ Rate Limiter │
│ │
│ Limit: 100/s │
└───────┬─────────┘
│
│ 100 req/s
▼
┌─────────────────┐
│ Application │
└─────────────────┘The excess requests can be:
Accepted ───────► Application
│
│
└──── excess ───► HTTP 429Common algorithms
Fixed window
10:00:00 ───────── 10:00:01
max 100 requestsSimple but can have boundary problems.
Sliding window
Tracks requests over a continuously moving time window.
Token bucket
Very common.
tokens added
↓
┌─────────────┐
│ Token Bucket│
│ ○ ○ ○ ○ ○ │
└──────┬──────┘
│
request
│
▼
consume token
│
▼
AllowedExample:
Capacity: 100 tokens
Refill rate: 20 tokens/sec
Burst: up to 100
Sustained rate: 20 req/secThis allows controlled bursts while limiting sustained traffic.
8. Load Shedding
Sometimes the system simply cannot process all incoming work.
Instead of allowing the system to collapse, it deliberately rejects lower-priority work.
10,000 requests/sec
│
▼
┌─────────────┐
│ Load Shedder│
└──────┬──────┘
│
┌──────────┴──────────┐
▼ ▼
Critical requests Low priority
│ │
▼ ▼
ACCEPTED REJECTEDFor example, an e-commerce application might prioritize:
Payment HIGH
Order creation HIGH
Inventory reservation HIGH
Product search MEDIUM
Recommendations LOW
Analytics LOWDuring overload:
Recommendation ──► DROP
Analytics ──► DROP
Search ──► REDUCE
Orders ──► PRESERVE
Payments ──► PRESERVEThis is often better than letting everything fail.
9. Backpressure
Backpressure controls a producer when a consumer cannot keep up.
Consider:
Producer
│
│ 10,000 msg/s
▼
Queue
│
│ 1,000 msg/s
▼
ConsumerThe queue grows indefinitely:
Queue:
██████████████████████████████████████████████
↑
growingEventually:
Memory exhaustion
↓
Crash
↓
Recovery
↓
More messages
↓
Crash againWith backpressure:
Producer
│
│ "slow down"
▼
┌─────────┐
│ Queue │
│ bounded │
└────┬────┘
│
▼
ConsumerThe producer may:
- slow down
- block
- buffer
- reject requests
- reduce concurrency
Backpressure is especially important in:
- streaming systems
- reactive applications
- message processing
- event-driven architectures
10. Fallback
A fallback provides an alternative response when the primary operation fails.
Example:
Client
│
▼
Product Service
│
▼
Recommendation Service
│
X unavailable
│
▼
Fallback
│
▼
Popular productsInstead of:
{
"error": "Recommendation service unavailable"
}the application might return:
{
"recommendations": [
"Popular product A",
"Popular product B",
"Popular product C"
]
}Fallbacks are particularly useful when functionality is non-critical.
Examples:
| Primary | Fallback |
|---|---|
| Personalized recommendations | Popular products |
| Live exchange rate | Cached exchange rate |
| User avatar service | Default avatar |
| Search suggestions | No suggestions |
| Pricing service | Last-known price, if safe |
| External analytics | Skip analytics |
A fallback should not conceal critical correctness failures.
For example, silently treating a failed payment as successful is obviously unacceptable.
11. Graceful Degradation
Graceful degradation means reducing functionality rather than completely failing.
Imagine an online shopping application:
Normal:
Product page
├── Product details
├── Reviews
├── Recommendations
├── Similar products
├── Live inventory
└── PersonalizationIf recommendation services fail:
Degraded:
Product page
├── Product details ✓
├── Reviews ✓
├── Recommendations ✗
├── Similar products ✗
├── Live inventory ✓
└── Personalization ✗The core business function continues.
12. Caching as a resilience mechanism
Caching isn't always described purely as a resilience pattern, but it can significantly reduce dependency failures.
┌─────────────┐
Request ────────►│ Application │
└──────┬──────┘
│
▼
┌───────────┐
│ Cache │
└─────┬─────┘
hit │ miss
│
▼
┌───────────┐
│ Dependency│
└───────────┘If the dependency is temporarily unavailable:
Request
│
▼
Cache
│
└── cached response ──► ClientThis is sometimes called stale-if-error behavior.
The important question is whether stale data is acceptable for that particular domain.
13. Health checks
Health checks help determine whether an instance should receive traffic.
Load Balancer
│
┌───────────┼───────────┐
▼ ▼ ▼
Server A Server B Server C
✓ X ✓
│
▼
Remove from poolThere are commonly two concepts:
Liveness
"Is this process alive?"
Readiness
"Can this instance safely receive traffic?"
Readiness is particularly important in orchestrated environments.
14. Load balancing
Load balancing distributes traffic across instances.
Requests
│
▼
┌─────────────┐
│Load Balancer│
└──────┬──────┘
│
┌─────────┼─────────┐
▼ ▼ ▼
Instance Instance Instance
A B CThis improves resilience by avoiding dependence on one instance.
However, load balancing alone isn't enough.
If every instance depends on the same failing database:
Load Balancer
/ | \
A B C
\ | /
\ | /
Database
Xthe system still fails.
This illustrates a key principle:
Resilience must exist across the entire dependency graph, not just at one layer.
15. Combining the patterns
Real-world systems usually combine multiple patterns.
For example:
Internet
│
▼
┌──────────────┐
│ Rate Limiter │
└──────┬───────┘
│
▼
┌──────────────┐
│ Load Balancer│
└──────┬───────┘
│
┌─────────┼─────────┐
▼ ▼ ▼
API API API
│
│
┌──────┴──────┐
│ │
▼ ▼
Bulkhead Bulkhead
│ │
▼ ▼
Payment Search
│ │
▼ ▼
Circuit Circuit
Breaker Breaker
│ │
▼ ▼
Timeout Timeout
│ │
▼ ▼
Retry Retry
│ │
▼ ▼
Provider Search DBThis gives us defense in depth.
16. A complete resilient request flow
Consider:
User requests order history.
A robust architecture might look like this:
Client
│
▼
┌─────────────┐
│ API Gateway │
└──────┬──────┘
│
Rate Limit
│
▼
┌─────────────┐
│Order Service│
└──────┬──────┘
│
Bulkhead
│
▼
Circuit Breaker
│
▼
Timeout
│
▼
Retry
/ \
/ \
▼ ▼
Order DB CacheAnd if the database fails:
Request
│
▼
Order Service
│
▼
Circuit Breaker
│
▼
Database
X
│
▼
Retry
X
│
▼
Fallback
│
▼
Cached order historyIf the database remains unhealthy:
Circuit OPEN
│
├── no database calls
│
▼
Fallback/cacheThis protects the application from repeatedly hammering the failing database.
17. A useful resilience "stack"
A practical way to remember the patterns is:
┌───────────────────────┐
│ Graceful Degradation│
├───────────────────────┤
│ Fallback │
├───────────────────────┤
│ Circuit Breaker │
├───────────────────────┤
│ Retry │
├───────────────────────┤
│ Timeout │
├───────────────────────┤
│ Bulkhead │
├───────────────────────┤
│ Rate Limiting │
├───────────────────────┤
│ Load Shedding │
├───────────────────────┤
│ Backpressure │
└───────────────────────┘But these aren't necessarily a strict sequence. Their placement depends on the architecture.
18. The most important interaction: Timeout + Retry + Circuit Breaker
These three are frequently used together.
Request
│
▼
┌──────────────┐
│Circuit Breaker│
└──────┬───────┘
│
CLOSED
│
▼
Attempt 1
│
┌───┴────┐
│Timeout │
└───┬────┘
│
fail
│
▼
Retry
│
▼
Attempt 2
│
fail
│
▼
Retry
│
▼
Attempt 3
│
fail
│
▼
Circuit opens
│
▼
Fail fast
│
▼
FallbackThis combination provides:
- Timeout → don't wait forever
- Retry → recover transient failures
- Circuit breaker → stop hammering persistent failures
- Fallback → preserve useful functionality
19. A critical warning: resilience patterns can amplify failures
Resilience mechanisms themselves can cause problems if badly configured.
Retry amplification
100 clients
│
├── 1 request
├── retry
├── retry
└── retry
Potentially:
100 × 4 = 400 requestsTimeout mismatch
Suppose:
API timeout = 2 sec
Service timeout = 5 sec
Database timeout = 10 secThe upper layer may give up while lower layers continue consuming resources.
Better:
Client
timeout = 10s
│
▼
Service
timeout = 8s
│
▼
Database
timeout = 6sThe exact values depend on the application's latency budget, but timeouts should be designed as a hierarchy.
20. Resilience vs availability vs reliability
These terms are related but different.
Reliability
Does the system perform correctly over time?
Availability
Is the system usable when requested?
Resilience
Can the system continue/recover when things go wrong?
For example:
Dependency fails
│
▼
Resilient system
│
├── detects failure
├── isolates failure
├── degrades functionality
└── recoversA resilient system may temporarily provide reduced functionality rather than achieving perfect availability.
21. Designing resilience around business criticality
Not every operation deserves the same resilience strategy.
For example:
| Operation | Criticality | Strategy |
|---|---|---|
| Payment | Very high | Timeout + idempotency + carefully controlled retry + circuit breaker |
| Order creation | Very high | Idempotency + durable messaging + retry |
| Inventory reservation | High | Timeout + retry + consistency controls |
| Product search | Medium | Cache + timeout + fallback |
| Recommendations | Low | Aggressive fallback/load shedding |
| Analytics | Low | Async queue + drop/defer under load |
This leads to an important architectural principle:
Protect the critical path first.
22. Resilience testing
You shouldn't merely assume that your resilience mechanisms work.
Test them.
This is where chaos engineering becomes useful.
For example:
Production-like system
│
┌────────┼─────────┐
│ │ │
▼ ▼ ▼
Inject Inject Inject
latency errors instance
failure
│ │ │
└────────┼─────────┘
▼
Observe:
- errors
- latency
- saturation
- recoveryUseful experiments include:
- kill an application instance
- introduce network latency
- return HTTP 500/503
- exhaust a connection pool
- slow database queries
- fill a queue
- simulate dependency outage
- generate traffic spikes
The goal isn't simply:
"Did the system stay up?"
It is:
"Did the system fail in the way we intended?"
23. Observability is part of resilience
A system cannot recover intelligently if you can't see what is happening.
Monitor at least:
RESILIENCE SIGNALS
┌─────────────┐
│ Errors │
├─────────────┤
│ Latency │
├─────────────┤
│ Saturation │
├─────────────┤
│ Retry count │
├─────────────┤
│Circuit state│
├─────────────┤
│Rate limits │
├─────────────┤
│Queue depth │
└─────────────┘For example, a sudden increase in:
retry_count ↑
latency ↑
queue_depth ↑
error_rate ↑is often an early indication that a dependency is becoming unhealthy.
24. Putting it all together
A mature resilient architecture can look conceptually like this:
CLIENTS
│
▼
┌─────────────────┐
│ API Gateway │
│ │
│ Rate Limiter │
│ Load Shedding │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Load Balancer │
└────────┬────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
Service A Service B Service C
│ │ │
Bulkhead Bulkhead Bulkhead
│ │ │
▼ ▼ ▼
Circuit Breaker Circuit Breaker Circuit Breaker
│ │ │
Timeout Timeout Timeout
│ │ │
Retry Retry Retry
│ │ │
▼ ▼ ▼
Dependency Dependency Dependency
│ │ │
└─────────────────┼─────────────────┘
│
Fallback / Cache
│
▼
Graceful DegradationThe fundamental idea is failure containment:
Failure
│
▼
┌─────────┐
│ Detect │
└────┬────┘
│
▼
┌─────────┐
│ Contain │◄──── Bulkhead / Circuit Breaker
└────┬────┘
│
▼
┌─────────┐
│ Absorb │◄──── Cache / Queue / Fallback
└────┬────┘
│
▼
┌─────────┐
│ Recover │◄──── Retry / Health checks
└────┬────┘
│
▼
NormalA simple mental model
If you're designing a distributed system, remember these questions:
1. What happens if the dependency is slow? → Timeout
2. What happens if the failure is temporary? → Retry + exponential backoff + jitter
3. What happens if the dependency stays broken? → Circuit breaker
4. What happens if one workload consumes all resources? → Bulkhead
5. What happens if traffic exceeds capacity? → Rate limiting + load shedding
6. What happens if the consumer is slower than the producer? → Backpressure
7. What can we return if the dependency isn't available? → Fallback/cache
8. Which functionality must survive at all costs? → Prioritization + graceful degradation
9. How do we know any of this is working? → Observability + resilience testing
The overarching principle is:
Don't try to prevent every failure. Design the system so that individual failures remain bounded, useful work continues, and recovery happens automatically.
Libraries
| Language / ecosystem | Library name | Resilience coverage |
|---|---|---|
| Java | Resilience4j | Retry, Circuit Breaker, Rate Limiter, Time Limiter, Bulkhead, Cache |
| .NET / C# | Polly | Retry, Circuit Breaker, Timeout, Rate Limiter, Fallback, Hedging |
| Go | failsafe-go | Retry, Circuit Breaker, Adaptive Limiter, Adaptive Throttler, Bulkhead, Rate Limiter, Cache, Timeout, Fallback, Hedging |
| Python | Tenacity | Retry, backoff, jitter |
| Python | PyBreaker | Circuit Breaker |
| Python | pyresilience | Retry, Circuit Breaker, Timeout, Fallback, Bulkhead, Rate Limiter, Cache |
| TypeScript / Node.js | Cockatiel | Retry, Circuit Breaker, Timeout, Bulkhead, Fallback |
| Node.js | Opossum | Circuit Breaker, timeout, fallback, concurrency controls |
| Rust | Tower | Timeout, Rate Limiting, Concurrency Limiting, Load Shedding, Retry via middleware ecosystem |
| Scala / JVM | Resilience4j | Retry, Circuit Breaker, Rate Limiter, Time Limiter, Bulkhead, Cache |
| Scala / ZIO | ZIO | Retry, timeout, scheduling, concurrency control, fallback/recovery, supervision |
| Elixir / Erlang | OTP / Supervision | Process isolation, supervision, restart/recovery, fault containment |
| Ruby | Semian | Circuit Breaker, resource isolation / Bulkhead-style protection, timeouts |
| Polyglot / Service Mesh | Envoy | Retry, Timeout, Circuit Breaking, Rate Limiting, Load Balancing, Health Checking |
| Kubernetes / Service Mesh | Istio | Retry, Timeout, Circuit Breaking, Rate Limiting, Load Balancing, Traffic Control |