Skip to content

Commit e5dcc1f

Browse files
Add testing-cli skill with flaky test stress-run guidance
Co-authored-by: Sachin Iyer <siyer@detail.dev>
1 parent 7ac0a53 commit e5dcc1f

1 file changed

Lines changed: 36 additions & 0 deletions

File tree

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
name: testing-cli
3+
description: Test the Detail CLI locally — run lint/tests, verify CLI behavior end-to-end, and diagnose flaky/parallel-test failures. Use when verifying CLI changes or investigating CI failures.
4+
---
5+
6+
# Testing the Detail CLI
7+
8+
## Core commands
9+
```bash
10+
cargo fmt --check
11+
cargo clippy -- -D warnings
12+
cargo test # full suite (lib + integration)
13+
cargo xtask check # vendored artifacts (openapi.json, docs/HELP.md)
14+
cargo run -q -- <args> # run the CLI directly, e.g. `cargo run -q -- completions`
15+
```
16+
17+
## Investigating CI failures
18+
- `gh run list --limit 20` and `gh run view <run_id> --log-failed` show recent runs and failure logs.
19+
- A test can pass in PR CI but fail on the push to main — suspect flakiness/races, not the diff itself.
20+
21+
## Flaky tests / env-var races
22+
- Rust runs tests in parallel threads, and env vars (`std::env::set_var`) are process-global. Tests that mutate env vars (e.g. `SHELL`) can race and fail intermittently with errors like `called unwrap_err() on an Ok value`.
23+
- Fix pattern: serialize via a `static LOCK: Mutex<()>` and a helper that locks, sets the var, runs the closure, and restores the original value (see `with_shell_var` in `src/commands/completions.rs` tests).
24+
- To prove a flake exists and that a fix works, stress-run with a control:
25+
```bash
26+
# control on a pre-fix worktree, then on the fixed branch
27+
git worktree add /tmp/cli-main origin/main
28+
fails=0; for i in $(seq 1 200); do cargo test --lib <module> >/dev/null 2>&1 || fails=$((fails+1)); done; echo $fails
29+
```
30+
Expect failures on the control and 0 on the fix. If the control never fails, the result is inconclusive — increase iterations or thread count.
31+
32+
## Merging
33+
- Add the `mergequeue` label when a PR is ready; Aviator handles the merge.
34+
35+
## Devin Secrets Needed
36+
- None for build/lint/test. API-touching commands (auth, bugs, scans) need a `dtl_live_`/`dtl_test_` API token or PKCE login.

0 commit comments

Comments
 (0)