Skip to content

feat: implement mutation testing for test quality assurance #506

@mostronatorcoder

Description

@mostronatorcoder

Context

Mostro Mobile is a Flutter application with critical security requirements for Bitcoin P2P trading. High-quality tests are essential for ensuring the safety of user funds and the reliability of the trading platform.

What is Mutation Testing?

Mutation testing is a technique to measure the quality and effectiveness of our test suite. It works by:

  • Creating mutants: The tool makes small, controlled changes (mutations) to the source code

Example: Change == to !=, + to -, true to false

  • Running the test suite: Tests are executed against each mutated version

  • Measuring survival rate:

✅ Mutant killed: Test failed → Good! Tests detected the artificial bug

❌ Mutant survived: Test passed → Bad! Tests did not catch the change

Mutation Score

Score = (Mutants killed / Total mutants) × 100

  • 80%: Excellent test coverage

  • 50-80%: Acceptable, room for improvement
  • < 50%: Needs work

Why Mutation Testing for Mostro Mobile?

  • Security-critical code: Trading and key management code needs rigorous verification
  • Detect weak tests: Find tests that "cover" code but do not actually verify correctness
  • Force better assertions: Encourages specific, strict assertions instead of generic ones
  • Find edge cases: Surviving mutants often reveal untested boundary conditions
  • CI integration: Can fail builds if mutation score drops below threshold
  • Documentation: Living documentation of what behavior is actually tested

Implementation Plan

Phase 1: Setup

  • Install mutation_test package

    dart pub add --dev mutation_test
  • Configure mutation testing
    Create mutation_test.yaml:

    name: Mostro Mobile
    
    targets:
      lib/**
    
    exclude:
      - "**/*.g.dart"
      - "**/generated/**"
      - "**/*.freezed.dart"

Phase 2: CI Integration

Add to .github/workflows/flutter.yml:

- name: Run tests
  run: flutter test

- name: Mutation Testing
  run: dart run mutation_test mutation_test.yaml
  continue-on-error: true

Phase 3: Gradual Improvement

  • Measure current baseline mutation score
  • Set threshold to current score (do not decrease)
  • Identify and fix surviving mutants in critical areas:
    • Key management (lib/services/key_derivator.dart)
    • Nost service (lib/services/nostr/)
    • Trade logic (lib/features/trade/)
  • Increase threshold by 5% each sprint
  • Target: 80% mutation score for security-critical modules

Acceptance Criteria

  • Add mutation_test to dev_dependencies in pubspec.yaml
  • Create mutation_test.yaml configuration
  • Add mutation testing step to CI workflow
  • Generate baseline mutation report
  • Document mutation score interpretation in docs/MUTATION_TESTING.md
  • Set initial threshold and enforce in CI (non-blocking initially)

Related

  • Similar issue in Choke project: grunch/choke#37
  • Mostro uses NIP-59 gift wrap, BIP-32 HD keys, and Lightning Network - all areas where mutation testing adds significant value

References

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions