Project case study · mvp
Chaos Sensei
A repository-aware chaos engineering and incident-training system designed to make controlled Kubernetes failure experiments safer and easier to review.
01
Problem and intended users
Teams need realistic failure practice, but ad hoc chaos experiments can be difficult to understand, approve, and reverse safely.
Who it is for
- Site reliability engineers
- Platform engineers
- Application teams running services on Kubernetes
02
Constraints and design considerations
- Experiments must be bounded by explicit safety policy.
- Every mutation needs a clear reversal path before execution.
- Repository context and provider-specific behavior must remain separate.
- The current implementation is a Kubernetes MVP, not a production-wide chaos platform.
03
Architecture overview
A repository analysis layer builds experiment context, a policy layer checks the proposed action, and a provider interface translates an approved experiment into a Kubernetes change. Execution records preserve the original state so the same provider can reverse the mutation.
04
Implementation approach
The provider boundary isolates platform mutations from repository analysis and policy decisions. The Kubernetes MVP focuses on selecting a target, applying a bounded change, recording what changed, and invoking the corresponding rollback operation.
05
Operational decisions
- Reliability
- Experiments are designed around reversible state transitions and an explicit rollback path rather than relying on manual recovery.
- Safety
- Policy checks, scoped targets, bounded changes, and pre-defined reversal behavior are treated as execution prerequisites.
- Security
- The design limits provider actions to the target and capability required by an approved experiment; production authorization and multi-tenant controls remain future work.
- Cost
- The MVP targets existing Kubernetes environments and does not require a separate always-on data platform.
06
Outcomes and current maturity
Current maturity: mvp
- Established a provider-oriented design for adding failure mechanisms without coupling them to repository analysis.
- Implemented the current Kubernetes MVP around reversible changes and explicit safety policy.
- Created a foundation for repeatable incident-training scenarios.
Current limitations
- Only the Kubernetes MVP is currently represented.
- Production authorization, audit retention, and broader provider coverage are not yet complete.
- No production reliability or business-impact metrics are claimed.
Roadmap
- Expand provider coverage after the Kubernetes workflow is validated.
- Add durable experiment audit records and approval integrations.
- Add automated recovery verification and reusable training scenarios.
07
Lessons learned
- Rollback must be designed with the experiment rather than added after execution.
- A provider contract keeps safety policy independent from platform-specific mutation details.
- Maturity labels are essential when presenting reliability tooling that is still an MVP.
08