Circuit Breakers Need a Recovery Probe
Limit calls to a failing dependency while making recovery bounded, observable, and resistant to late results from earlier attempts.
A dependency starts timing out. Every request keeps calling it, and retries add more attempts. A circuit breaker can stop some of that work by temporarily refusing calls after enough eligible failures. The difficult moment comes later: when the waiting period ends, how much traffic should the application release to discover whether the dependency has recovered?
If every waiting caller is admitted at once, the recovery check becomes another burst. If no caller is ever admitted, the breaker cannot observe recovery. A useful design treats the transition back to normal operation as a bounded experiment, with explicit ownership of the recovery probe.
Define failures before counting them
Choose the operation and dependency boundary protected by the breaker. A shared breaker for unrelated endpoints can turn one failing operation into a wider outage. A separate breaker per arbitrary request identifier can create unbounded state. The scope should reflect a failure domain, with bounded cardinality and clear ownership.
Count outcomes that indicate dependency trouble, such as selected connection errors or service-unavailable responses. A user validation error usually says little about dependency health. Authentication failures can indicate bad local configuration rather than a recoverable remote outage. Decide how caller cancellations and exhausted request deadlines affect the measurement.
The reference model below uses consecutive failures for clarity. Production policies may use a failure ratio over a sampling window with a minimum throughput requirement. These are policy choices with different behavior under low traffic; the example is not an implementation of a particular resilience library.
Admit one probe and reject stale completions
All calls to this Python model are serialized. A ticket identifies the state generation in which an attempt was admitted. Each admitted attempt must report completion once. The model demonstrates state transitions, not thread safety, network calls, or distributed coordination.
class Breaker:
"""Serialized, single-process reference model; times are monotonic."""
def __init__(self, threshold=2, cooldown=5):
self.threshold, self.cooldown = threshold, cooldown
self.state, self.failures, self.generation = "closed", 0, 0
self.retry_at = 0
def acquire(self, now):
if self.state == "open" and now >= self.retry_at:
self.state = "half-open"
self.generation += 1
return (self.generation, True)
if self.state != "closed":
raise RuntimeError("circuit refuses this attempt")
return (self.generation, False)
def complete(self, ticket, success, now):
generation, probe = ticket
if generation != self.generation:
return # old attempts cannot settle a newer state
if probe:
if success:
self.state, self.failures = "closed", 0
self.generation += 1
else:
self.trip(now)
elif self.state == "closed":
self.failures = 0 if success else self.failures + 1
if self.failures >= self.threshold:
self.trip(now)
def trip(self, now):
self.state, self.retry_at = "open", now + self.cooldown
self.generation += 1
b = Breaker()
late = b.acquire(0)
b.complete(b.acquire(0), False, 0)
b.complete(b.acquire(0), False, 0)
assert b.state == "open"
b.complete(late, True, 1)
assert b.state == "open"
for now in (1, 4):
try:
b.acquire(now)
except RuntimeError:
pass
else:
raise AssertionError("cooldown bypassed")
probe = b.acquire(5)
try:
b.acquire(5)
except RuntimeError:
pass
else:
raise AssertionError("second recovery probe admitted")
b.complete(probe, False, 5)
assert b.state == "open"
probe = b.acquire(10)
b.complete(probe, True, 10)
assert b.state == "closed"
The first two eligible failures open the circuit. An earlier request then reports success, but its generation is obsolete and it cannot close the newer open state. At the recovery time, one caller acquires the probe ticket. Another caller is refused until that probe settles. A failed probe restarts the waiting period; a later successful probe closes the circuit.
The generation check addresses delayed results rather than elapsed time. Without it, a response from work admitted before the outage can arrive during recovery and accidentally decide the new state. Real implementations need to make admission and transition atomic under their concurrency model, and they must prevent duplicated completion callbacks from being counted twice.
Give the probe a deadline
A probe that never finishes leaves the breaker half-open indefinitely. Bound the probe with the request's remaining deadline and a dependency timeout. Its completion path must report a failure or another explicitly defined outcome when it times out. The controller should not depend on a successful HTTP response to release the probe slot.
A single successful request is limited evidence. Some systems close immediately; others increase admitted traffic gradually or require several successes. Choose the policy from the dependency's recovery characteristics. A cold cache, a reconnecting pool, or an autoscaling service may need a slower ramp than a process that has simply restarted.
Multiple application instances usually have independent breaker state. One probe per instance can still be a large burst across a fleet. Jittering recovery times or limiting probes at a shared boundary may help, but centralized coordination adds its own failure modes. Describe the actual scope instead of calling a local breaker a global protection mechanism.
Compose with other limits deliberately
A breaker does not cap the number of calls already in flight. Use a concurrency limit or bulkhead to bound that resource. It also does not define retry safety. Retrying a timed-out write still requires an idempotency or reconciliation contract, even when the circuit permits the attempt.
Ordering a retry policy and breaker policy changes what gets counted. If the breaker sees every underlying attempt, a single logical request can contribute several failures. If it sees only the final retry outcome, it may react more slowly while the dependency receives extra traffic. Pick the behavior deliberately and test it under the expected retry configuration.
Refusal should be an explicit outcome. A cached fallback may be useful if the product accepts its age and scope. Returning a fabricated success can conceal lost work or stale decisions. For required writes, failing quickly with a retryable or reconciliation-aware response is often more honest than pretending the dependency was available.
Observe recovery as well as failure
Measure rejected attempts, state transitions, probe latency and outcomes, and the time spent open. Distinguish a low request rate from an open breaker that has seen no opportunity to probe. In an admission-driven design, the waiting period makes a probe eligible; it does not itself create network traffic.
Test late completions, simultaneous recovery callers, a hung probe, low traffic, and repeated failed recovery. The valuable property is not simply that a breaker opens. It is that the application can explain when work is refused, who may test recovery, and which evidence permits normal traffic to resume.