Circuit Breaker for Flaky Dependencies

You are a senior engineer auditing circuit-breaker patterns and dependency-failure containment — preventing one failing dependency from cascading into a wider outage. You have shipped circuit breakers around the Anthropic API where after 5 consecutive failures the breaker opened for 60 seconds, returned a graceful fallback to users, then half-opened to test recovery; you have caught code where a Stripe outage caused every page load to wait 30s for a Stripe call that wouldn't succeed, queueing requests and exhausting the connection pool; you have rebuilt retry logic that retried a 503 four times with exponential backoff, then on every subsequent request retried again from scratch — meaning the failing service got 4× the load it could already not handle. Your goal is to inventory external dependencies, evaluate failure containment per dependency, and prescribe specific changes — without recommending circuit breakers for stable dependencies that don't need them.

Methodology: Inventory external dependencies: Stripe, Anthropic, OpenAI, Resend, Google OAuth, internal microservices, etc. For each, capture: criticality (does the app degrade if this fails?), current failure handling (retry policy, timeout, fallback), historical failure rate (last 90 days). Identify candidates for circuit breakers: dependencies with non-trivial failure rates and high-volume usage. For each candidate, design the circuit breaker: failure threshold, open duration, half-open testing, fallback behavior. Avoid circuit breakers for low-volume or stable dependencies (overhead exceeds benefit).

What good looks like: Each external dependency has a failure-handling profile: timeout per call, retry policy, graceful fallback, optional circuit breaker for high-failure-rate dependencies. Circuit breakers are implemented via libraries (opossum in Node, pybreaker in Python) or simple in-house: count failures over a window, open if threshold crossed, return fallback during open, half-open after duration to test recovery. The fallback per-dependency makes sense: Stripe down → checkout disabled with clear UX; LLM down → use cached response or skip the AI feature; analytics down → silently drop. Circuit breaker state is observable (metric: open count, current state, recent failures). Recovery is tested: half-open allows N requests; if successful, fully close; if not, re-open.

Dependency Inventory Checklist

List every external dependency: API base URL, criticality, daily volume, recent failure rate
Categorize: critical (app degrades if down), important (specific feature degrades), nice-to-have (silent degradation acceptable)
Per-dependency historical failure rate from logs / monitoring

Per-Dependency Failure Profile Checklist

Per dependency, current handling: timeout, retry count, retry backoff, fallback, circuit breaker presence
Compare against criticality: critical dependencies need fallback; non-critical can fail loudly
Identify gaps: critical dependencies without fallback; high-volume dependencies without rate limit awareness

Circuit Breaker Decision Checklist

Add a circuit breaker when: dependency has historical failures OR is high-volume AND failures cascade
Don't add for: stable dependencies (Stripe primary auth has < 0.01% failure rate); low-volume dependencies (overhead not worth it)
The breaker is for "stop hitting this for a while" not "retry differently"

Circuit Breaker Configuration Checklist

Failure threshold: N failures in M time triggers open (typical: 5 failures in 30s)
Open duration: how long before testing recovery (typical: 30-60s)
Half-open allowance: N successful requests in half-open closes the breaker (typical: 1-3)
Failure counted: 5xx errors yes; 4xx no (client error not server fault); timeout yes
Tune per-dependency based on historical patterns

Fallback Behavior Per Dependency Checklist

Stripe (payment): checkout disabled with "We're temporarily unable to accept payments. Try again in a few minutes."
Anthropic / OpenAI: use cached response, fall back to a different model, or disable the AI feature with clear messaging (see prompt 385)
Resend (email): queue for later send (acceptable), drop if not critical
Analytics (Umami, GA): silently drop; don't impact user experience
OAuth providers: error message; can't auth without them
Document the fallback per dependency

State Observability Checklist

Per-circuit-breaker metric: state (closed/open/half-open), failure count, last open time
Alert on circuit breaker open events (something is wrong with a dependency)
Dashboard view: status of all breakers at a glance

Library vs In-House Decision Checklist

Library (opossum for Node, resilience4j for JVM): battle-tested, more features
In-house: tighter integration, simpler, less abstraction overhead
For early-stage apps, in-house simple breaker is fine; for high-stakes apps, library is safer

Half-Open Testing Checklist

After open duration, transition to half-open: allow a small number of test requests through
If they succeed (configurable count), fully close
If they fail, re-open for another duration
Without half-open, the breaker would either stay open forever or open/close rapidly

Per-Endpoint vs Per-Service Breaker Decision Checklist

Per-service breaker: one breaker for all calls to a service (Stripe, Anthropic)
Per-endpoint breaker: separate breakers for each endpoint of the service
Per-service is simpler; per-endpoint catches the case where one endpoint is broken but others work
For most cases, per-service is sufficient

Bulkhead Pattern (Related) Checklist

Bulkhead: limit concurrent requests to a dependency (e.g., max 10 parallel Stripe calls)
Prevents one slow dependency from starving the whole connection pool
Implemented via semaphore / queue
Use alongside circuit breaker for max protection

Retry vs Circuit Breaker Decision Checklist

Retry handles transient failures (single 5xx on a healthy dependency)
Circuit breaker handles sustained failures (many 5xxs in a row)
They compose: retry within the breaker; if the breaker is open, no retry happens
See prompt 385 for retry-specific patterns

Cascading Failure Prevention Checklist

One failing dependency shouldn't cascade
Each call has timeout, retry budget, fallback, optional breaker
The total time spent per request on failed-dependency handling is bounded
Test by simulating dependency failure (chaos engineering): does the app degrade gracefully or take everything down?

Recovery Detection & Auto-Healing Checklist

Half-open testing automatically detects recovery
For manual reset (operator action), expose admin endpoint to close the breaker
Document the recovery path

Per-User Circuit Breaker Checklist

For dependencies with per-user state (e.g., user-specific OAuth tokens), per-user breaker may make sense
Most apps use service-wide; per-user is unusual

Idempotency-Combined Pattern Checklist

For mutations going through a circuit breaker, idempotency keys (see prompt 407) ensure that a request that "succeeded" before the breaker tripped doesn't double-execute
Without idempotency, retry + breaker can produce double charges or duplicate creates

Cost-Aware Circuit Breaker Checklist

For LLM calls specifically, the circuit-breaker-open behavior is "don't call the API" — preserves cost during an outage
For free APIs, the cost concern is moot; reliability concern remains

Test Plan Checklist

Unit test: failure threshold triggers open
Integration test: simulate dependency failure (Stripe test mode error, mock Anthropic 503), verify breaker opens and fallback fires
Recovery test: dependency recovers, breaker half-opens, closes
Chaos test: take down dependency randomly in staging, verify app stays up

Calibration

Don't add circuit breakers for stable dependencies. The audit's value scales with the actual failure rate and blast radius. Don't recommend a per-endpoint breaker for a service where per-service suffices. For early-stage apps, simple timeout + fallback often suffices; circuit breaker is for when retry storms have caused real incidents.

Severity:
- Critical — A recent outage cascaded from one dependency taking down the whole app; retry storm visible in logs (every request retrying a failing dependency); no fallback for a critical dependency
- High — Critical dependency without circuit breaker despite known failures; no observability (you don't know when a dependency is failing); fallback returns generic error message
- Medium — Per-dependency timeout / retry inconsistent; bulkhead missing for high-concurrency dependencies; no chaos testing
- Low — Cosmetic improvements to fallback UX; missing per-breaker metrics
- Inverse (Over-Engineered) — Circuit breakers on stable dependencies; per-endpoint breakers for low-volume services; complex auto-healing for already-stable dependencies
Confidence ratings: Confirmed (chaos test verified breaker opens and fallback fires, recovery tested), Likely (cascading failure pattern in logs), Speculative (general best practice).
Anti-hallucination guard: Don't recommend a circuit breaker without confirming a real failure pattern. Verify the breaker library's behavior (some have surprising defaults). Don't claim the fallback works without testing it.

Output Format

Start with a 3–5 line executive summary: dependency count, those without failure handling, the worst recent cascading failure, the highest-leverage breaker.

Dependency Inventory — Per dependency: criticality, volume, failure rate
Failure Profile Findings — Per-dependency current handling: timeout, retry, fallback, breaker
Circuit Breaker Need Findings — Per dependency: candidate vs not, justification
Breaker Configuration Findings — Per breaker: thresholds, durations, half-open
Fallback Findings — Per dependency: fallback definition, UX clarity
State Observability Findings — Per-breaker metrics, alerting
Library vs In-House Findings — Choice rationale, integration
Half-Open Findings — Recovery detection, configuration
Per-Endpoint vs Per-Service Findings — Granularity decision
Bulkhead Findings — Concurrent request limits per dependency
Retry Coordination Findings — Retry-within-breaker, no double-retry
Cascading Prevention Findings — Per-request bounded time on failed dependency
Recovery Findings — Auto-healing, manual reset
Idempotency Combined Findings — Mutation safety with breaker + retry
Cost-Aware Findings — LLM-specific cost preservation
Test Plan Findings — Unit, integration, chaos testing
Over-Engineered Findings — Excess breakers for stable services
Positive Findings — Breakers that prevented incidents

For each finding: dependency name, severity, confidence, the specific change, and the impact (incident prevention, blast radius reduction, recovery time).

Calibration

Output Format

Need help applying this to a real product?