Every network call is a small bet that the other side is up, fast, and willing. In production that bet loses constantly - a dropped packet, a slow dependency, a service mid-deploy. Resilience policies are how you turn those losses from outages into shrugs. Polly gives you the building blocks; the skill is composing them in the right order, because a wrong order quietly makes things worse.
Three policies, three jobs
The three policies you reach for most do genuinely different things:
- Retry handles the truly transient - a blip that succeeds on a second attempt. With exponential backoff and jitter, it spreads retries out instead of stampeding a recovering service.
- Circuit breaker handles the not-transient - when a dependency is clearly down, it stops hammering it and fails fast, giving the dependency room to recover.
- Timeout handles the never-answers case - it caps how long any one attempt may hang so a slow call cannot exhaust your threads.
Order is everything
The composition order determines behavior, and the correct outer-to-inner order is retry, then circuit breaker, then timeout. Read it from the outside in: a retry wraps the breaker, which wraps a per-attempt timeout. That way each individual attempt is time-bounded, the breaker counts real failures, and the retry decides whether to try again.
builder.Services.AddHttpClient<BillingClient>()
.AddResilienceHandler("billing", pipeline =>
{
pipeline.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
});
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
BreakDuration = TimeSpan.FromSeconds(15),
});
pipeline.AddTimeout(TimeSpan.FromSeconds(10));
});
Get the order wrong - a timeout outside the retry, say - and a single outer timeout can cut off all your retries at once, defeating the purpose.
Warning: Retrying non-idempotent operations can double-charge a customer or duplicate an order. Only retry calls that are safe to repeat, or make them idempotent with a request key first.
Drive it from configuration
Hard-coding thresholds means a tuning change is a redeploy. We bind the numbers from appsettings so operators can adjust backoff, failure ratios, and timeouts per environment without touching code. Sane defaults in code, overrides in config.
Resilience beyond HTTP
HTTP is the obvious case, but the same thinking applies across the system. Message consumers use the broker's retry and dead-letter mechanics rather than blocking retries, and background jobs lean on the scheduler's automatic retry. The principle is constant: distinguish transient from persistent failure, and respond to each appropriately instead of treating every error the same way.
Make failures observable
A resilience policy that fails silently is a debugging nightmare. Tag retries, breaker state changes, and timeouts onto your traces and metrics so you can see when a dependency is degrading. Often the breaker tripping is the earliest signal of an incident - if you are watching for it.
Wrapping up
Resilience is not about never failing; it is about failing in controlled, recoverable ways. Compose retry, circuit breaker, and timeout in that outer-to-inner order, only retry what is safe to repeat, drive the thresholds from configuration, and make every policy decision visible in telemetry. Do that and the inevitable network losses become noise your system absorbs instead of incidents your team chases.
Want your integrations to shrug off the bad days? Let's talk resilience.
