Skip to content

feat: perch coverage says what test to add #348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jbrown9513 wants to merge 1 commit into from
Closed

feat: perch coverage says what test to add #348

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 43 additions & 32 deletions docs/coverage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,23 +28,26 @@ Point it at a repository:
```console
$ perch coverage
Source files Mutation score Survived No coverage
src/cart.ts 71% (10 of 14) 4 0
src/checkout.ts 60% (3 of 5) 2 0
src/inventory.ts 63% (5 of 8) 2 0
src/cart.ts 71% (10 of 14) 4 0
All source 67% (18 of 27) 8 0

Test files Quality Duplicates Checks nothing Live services
test/cart.test.ts 43% (3 of 7) 4 0 1
test/checkout.test.ts 100% (1 of 1) 0 0 0
test/inventory.test.ts 75% (3 of 4) 0 1 0
All tests 58% (7 of 12) 4 1 1

src/cart.ts
ID Line Problem Confidence Test or method Note
0399f832 12 survived 87% applyDiscount With `101` instead of `100`, none of the …
1be27338 12 survived 87% applyDiscount With `>` instead of `>=`, none of the 5 t…
20fb814a 12 survived 87% applyDiscount With `1` instead of `0`, none of the 5 te…
883b4b21 12 survived 84% applyDiscount With `false` as the condition, none of th…
1 test file has nothing to fix.

Where to add tests
Method Where Killed Survived Tests
applyDiscount src/cart.ts:11 6 of 10 4 5
Add a test that asserts on the value `100` at line 12. 3 more edits survive.
placeOrder src/checkout.ts:5 3 of 5 2 1
Add a test in which the condition at line 6 is true, and assert on what follows. 1 more edit
survives.
canFulfil src/inventory.ts:4 5 of 8 2 4
Add a test that asserts on the value `0` at line 6. 1 more edit survives.

test/cart.test.ts
ID Line Problem Confidence Test or method Note
Expand All @@ -57,29 +60,39 @@ test/cart.test.ts
test/inventory.test.ts
ID Line Problem Confidence Test or method Note
98a1a024 23 mocked - canFulfil > returns … Mocks every method it calls: canFul…
shop at commit 0ade512: 4 methods, 12 tests, 14 problems, 10 shown, --all for the rest
shop at commit 0ade512: 4 methods, 12 tests, 14 problems, --all for every mutant
Report: .perch/coverage/index.html
28 requests 0 tokens in $0.0000
```

The mutation score is the share of mutants killed by at least one test. A test
kills a mutant when Perch puts its chance of failing against it at 70% or more.
No coverage counts the mutants in methods no test reaches through the call
graph. They count against the score. Perch lists each survived mutant with the
edit it made and the number of tests that miss it. The confidence is how sure
Perch is that no test kills the mutant and that the mutant changes what a caller
sees. A test that checks nothing kills no mutant in the code it reaches. A
duplicate kills exactly the mutants an earlier test kills. A test that mocks
what it tests calls only methods it has replaced with its own mocks, so it
checks the mocks. That is a fact of the call graph, so it is listed with `-`
for confidence. So is a problem on a test or method the model couldn't be asked
about. Use `--all` to see every row.

Perch also writes an HTML report. It shows each line of source with its problems
under it. For a survived mutant, it shows the original and the mutated version
of the line, and the tests that still pass. The report is one file, unless your
repository has more than 8 MB of source. Then each file gets its own page in
the `files/` directory next to `index.html`.
graph. They count against the score. Source files are listed worst first, and
test files only when something in them needs fixing.

Under the tables, Perch lists the methods to add a test to, most survived
mutants first. Each has a line of numbers and, under it, the test to add, from
its surest survived mutant: the edit a new test has to tell apart, and the line
it is on. `--all` lists every method, and every survived mutant with the id
that `perch close` takes. Test problems follow, by file. The confidence is how
sure Perch is. A test that checks nothing kills no mutant in the code it
reaches. A duplicate kills exactly the mutants an earlier test kills. A test
that mocks what it tests calls only methods it has replaced with its own mocks,
so it checks the mocks. That is a fact of the call graph, so it is listed with
`-` for confidence. So is a problem on a test or method the model couldn't be
asked about.

Perch also writes an HTML report. It opens on the methods to add a test to,
each with its score, how many tests reach it, and the test to add; the mutants
fold out under the row. Source files are a tree of directories, worst first,
with each score coloured against 60% and 80% as Stryker colours its own. A
file's page shows its code with a mark in the gutter on each line where a
mutant survived; press the mark to see each edit, the line both ways, and a
Copy prompt button, or step through the survivors with the Next survived
button. The report is one file, unless your repository has more than 8 MB of
source. Then each file gets its own page in the `files/` directory next to
`index.html`.

## How it differs from a coverage report

Expand Down Expand Up @@ -200,17 +213,15 @@ changed on the branch:

```console
$ perch coverage --since main
src/cart.ts
ID Line Problem Confidence Test or method Note
20fb814a 12 survived 86% applyDiscount With `1` instead of `0`, none of the 5 te…
1be27338 12 survived 85% applyDiscount With `>` instead of `>=`, none of the 5 t…
0399f832 12 survived 84% applyDiscount With `101` instead of `100`, none of the …
883b4b21 12 survived 81% applyDiscount With `false` as the condition, none of th…
Where to add tests
Method Where Killed Survived Tests
applyDiscount src/cart.ts:11 6 of 10 4 5
Add a test that asserts on what line 12 returns. 3 more edits survive.

test/cart.test.ts
ID Line Problem Confidence Test or method Note
427fce7a 22 redundant 76% applyDiscount > tak… Kills the same mutants as applyDis…
shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere
shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere, --all for every mutant
Report: .perch/coverage/index.html
28 requests 0 tokens in $0.0000
```
Expand Down
40 changes: 40 additions & 0 deletions src/coverage-advice.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/**
* What test to add for a survived mutant, in one sentence, from the kind of edit it is. The sentence is fixed wording around the
* report's own facts: the line, and the text the edit changed. It says what a test has to tell apart, not how to write it.
*/

/** A piece of code in a sentence, cut to one line of it. */
const edit = text => {
const one = String(text ?? '').trim().replace(/\s+/g, ' ');
return `\`${one.length > 60 ? `${one.slice(0, 57)}…` : one}\``;
};

/** The other value of a forced condition: a test is missing for the case the mutant removed. */
const otherBranch = to => (/^(true|True)$/.test(String(to).trim()) ? 'false' : 'true');

export function adviceFor(mutant) {
const line = mutant.line;
switch (mutant.kind) {
case 'body': return 'Add a test that fails when the method does nothing.';
case 'removal': return `Add a test that fails when ${edit(mutant.from)} is not called.`;
case 'block': return `Add a test that fails when the block at line ${line} is skipped.`;
case 'condition': return `Add a test in which the condition at line ${line} is ${otherBranch(mutant.to)}, and assert on what follows.`;
case 'boundary': return `Add a test at the boundary of ${edit(mutant.from)} on line ${line}, where ${edit(mutant.from)} and ${edit(mutant.to)} give different results.`;
case 'logic': return `Add a test in which only one side of ${edit(mutant.from)} on line ${line} holds.`;
case 'string': return `Add a test that asserts on the text ${edit(mutant.from)} at line ${line}.`;
case 'number': return `Add a test that asserts on the value ${edit(mutant.from)} at line ${line}.`;
case 'return': return `Add a test that asserts on what line ${line} returns.`;
case 'arithmetic': case 'update': return `Add a test that asserts on the arithmetic at line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)}.`;
default: return `Add a test that asserts on the result of line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)} unnoticed.`;
}
}

/**
* The test to add for a method, from its surest survived mutant, and how many more edits survive beside it. `mutants` are the
* method's listed survived mutants, surest first.
*/
export function methodAdvice(mutants) {
if (!mutants.length) return '';
const more = mutants.length - 1;
return `${adviceFor(mutants[0])}${more ? ` ${more} more ${more === 1 ? 'edit survives' : 'edits survive'}.` : ''}`;
}
Loading
Loading