Microservices that fix themselves
Self-Healing Distributed System
Event-driven platform on AWS that detects and recovers failed microservices automatically.
- Recovery time
- 5-30s
- Automated tests
- 74
- Dashboard
- 20 CloudWatch widgets
the problem
When a microservice dies at 3am, someone gets paged, ssh-es in, and restarts it. Most failures follow predictable patterns and have predictable fixes, yet the industry default is still waking a human up. I wanted to see how far automated recovery could go before a human genuinely needs to be involved.
the solution
An event-driven control loop built on FastAPI, Docker, and AWS. Health signals flow through EventBridge into a recovery engine that classifies failures by severity using sliding-window failure tracking, then applies a per-service strategy: restart, fall back, or escalate. Circuit breakers stop the system from hammering a service that keeps failing, and severity escalates automatically from LOW to CRITICAL as failures accumulate. Typical detection-to-recovery time is 5 to 30 seconds. Everything is observable through a 20-widget CloudWatch dashboard, and crash reports land in S3 for postmortems.
tech stack
- Python + FastAPIHealth endpoints and the recovery control plane
- DockerService isolation and restart primitives
- AWS LambdaServerless recovery actions
- EventBridgeEvent bus routing failure signals to handlers
- CloudWatchMetrics, alarms, and the 20-widget operations dashboard
- S3Durable crash reports for post-incident analysis
- EC2Deployment target for the running system
key engineering decisions
Sliding-window failure tracking instead of simple thresholds
A service failing 3 times in 10 seconds is a very different situation from 3 times in an hour. Tracking failures over a sliding window lets severity escalate from LOW to CRITICAL based on failure density, which maps much better to how incidents actually unfold.
Circuit breakers around every recovery action
The worst failure mode for a self-healing system is a restart loop that makes things worse. Circuit breakers open after repeated failed recoveries and force escalation instead, so the system knows when to stop trying and hand off.
Per-service recovery strategies
Restart is right for a stateless API but wrong for something mid-write. Each service declares its own strategy (restart, fallback, escalate), so recovery behavior matches what the service can actually tolerate.
74 tests, including simulated failures
Recovery code is code that runs when everything is already going wrong, so it has to be the most tested part of the system. The suite covers circuit-breaker state transitions and injected failure scenarios end to end.