Resilience Engineering: Scenario-Based Questions

Distributed systems often fail in unpredictable ways due to network latency, service overload, or downstream outages. Circuit breakers and retry strategies help absorb failures and prevent cascading outages.

🧰 What Is a Circuit Breaker?

• Monitors downstream calls for failure patterns (e.g., 5xx errors, timeouts).
• Trips to “open” state to block further calls when failure threshold is exceeded.
• Periodically transitions to “half-open” to test if recovery is possible.
• Returns to “closed” state after success is confirmed.

🔁 Retry Strategies

• Exponential Backoff: Increase delay between retries to reduce pressure on target.
• Jitter: Add randomness to retry intervals to avoid thundering herd effects.
• Max Attempts: Limit total retry count to avoid infinite loops.

🧱 Architecture Integration

• Use libraries like Hystrix, Resilience4j (Java), Polly (.NET), or Envoy retries (service mesh).
• Apply at service mesh or client layer — don’t push retries into the database.
• Use fallback responses (e.g., cached data, default values) when circuit trips.
• Log and alert on circuit breaker state transitions.

✅ Best Practices

• Use circuit breakers on all critical external or slow services.
• Tag metrics and dashboards with breaker status and retry rates.
• Test breaker behavior during chaos drills or simulated outages.

🚫 Common Pitfalls

• Retrying unsafe or state-changing operations (e.g., duplicate payments).
• Not capping retry attempts or failing to add delay.
• Trip breakers too early or fail to reset when recovery happens.

📌 Real-World Insight

Resilient systems expect failure. Circuit breakers and retries allow services to degrade gracefully instead of collapsing. They’re foundational in Netflix, Amazon, and Google architectures for protecting availability.