Skip to content

Schedule quarterly chaos game-days with scenario library and post-mortem template #417

Description

@thlpkee20-wq

Description

Chaos tests exist but are ad-hoc. Schedule recurring game-days with a versioned scenario library (DB primary loss, Redis split-brain, Horizon outage) and an automated post-mortem skeleton generated from observed metrics.

Requirements and context

  • Must be secure, tested, and documented
  • Should be efficient and easy to review
  • Relevant code: ops/chaos/, src/__tests__/
  • Scenarios must declare blast radius and abort criteria

Suggested execution

  • Fork the repo and create a branch
  • git checkout -b feat/chaos-game-day-library
  • Implement changes
    • Add ops/chaos/scenarios/*.yaml
    • CLI npm run chaos:run -- --scenario=...
    • Auto-generate post-mortem skeleton with metric snapshots
  • Validate security and correctness assumptions

Test and commit

  • Run tests
    • npm test
  • Cover edge cases
    • Abort criterion triggers cleanup and rolls back
  • Include test output and notes

Example commit message

feat: chaos game-day scenario library

Guidelines

  • Minimum 95 percent test coverage
  • Clear documentation
  • Timeframe: 96 hours

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions