From 44c73ee1643de4c5f01a244363ee6098054b5fe1 Mon Sep 17 00:00:00 2001 From: Joshua Brown Date: Thu, 8 Oct 2026 17:33:05 -0700 Subject: [PATCH] feat: the coverage report ranks the methods to add a test to and says what to add The summary, each file's page and the terminal listed one row per survived mutant, with the same eight test names under every one, and the code came after the rows. They now rank methods with survived mutants, most first, each with a sentence from its surest mutant saying what a new test has to tell apart, and the mutants fold under the row. A file's page shows its code first with a count in the gutter where mutants survived, one note open at a time, and next and previous survived. Source files are a tree worst first with scores coloured against 60 and 80 as Stryker colours them; test files are listed only when something needs fixing. --- docs/coverage.md | 75 ++++--- src/coverage-advice.js | 40 ++++ src/coverage-html.js | 372 +++++++++++++++++++++++++++-------- src/coverage-report.js | 78 ++++++-- test/coverage-advice.test.js | 35 ++++ test/coverage-html.test.js | 123 ++++++++---- test/coverage-report.test.js | 70 ++++++- 7 files changed, 625 insertions(+), 168 deletions(-) create mode 100644 src/coverage-advice.js create mode 100644 test/coverage-advice.test.js diff --git a/docs/coverage.md b/docs/coverage.md index efeadd3..5da62a9 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -28,23 +28,26 @@ Point it at a repository: ```console $ perch coverage Source files Mutation score Survived No coverage -src/cart.ts 71% (10 of 14) 4 0 src/checkout.ts 60% (3 of 5) 2 0 src/inventory.ts 63% (5 of 8) 2 0 +src/cart.ts 71% (10 of 14) 4 0 All source 67% (18 of 27) 8 0 Test files Quality Duplicates Checks nothing Live services test/cart.test.ts 43% (3 of 7) 4 0 1 -test/checkout.test.ts 100% (1 of 1) 0 0 0 test/inventory.test.ts 75% (3 of 4) 0 1 0 All tests 58% (7 of 12) 4 1 1 - -src/cart.ts - ID Line Problem Confidence Test or method Note - 0399f832 12 survived 87% applyDiscount With `101` instead of `100`, none of the … - 1be27338 12 survived 87% applyDiscount With `>` instead of `>=`, none of the 5 t… - 20fb814a 12 survived 87% applyDiscount With `1` instead of `0`, none of the 5 te… - 883b4b21 12 survived 84% applyDiscount With `false` as the condition, none of th… +1 test file has nothing to fix. + +Where to add tests + Method Where Killed Survived Tests + applyDiscount src/cart.ts:11 6 of 10 4 5 + Add a test that asserts on the value `100` at line 12. 3 more edits survive. + placeOrder src/checkout.ts:5 3 of 5 2 1 + Add a test in which the condition at line 6 is true, and assert on what follows. 1 more edit + survives. + canFulfil src/inventory.ts:4 5 of 8 2 4 + Add a test that asserts on the value `0` at line 6. 1 more edit survives. test/cart.test.ts ID Line Problem Confidence Test or method Note @@ -57,7 +60,7 @@ test/cart.test.ts test/inventory.test.ts ID Line Problem Confidence Test or method Note 98a1a024 23 mocked - canFulfil > returns … Mocks every method it calls: canFul… -shop at commit 0ade512: 4 methods, 12 tests, 14 problems, 10 shown, --all for the rest +shop at commit 0ade512: 4 methods, 12 tests, 14 problems, --all for every mutant Report: .perch/coverage/index.html 28 requests 0 tokens in $0.0000 ``` @@ -65,21 +68,31 @@ Report: .perch/coverage/index.html The mutation score is the share of mutants killed by at least one test. A test kills a mutant when Perch puts its chance of failing against it at 70% or more. No coverage counts the mutants in methods no test reaches through the call -graph. They count against the score. Perch lists each survived mutant with the -edit it made and the number of tests that miss it. The confidence is how sure -Perch is that no test kills the mutant and that the mutant changes what a caller -sees. A test that checks nothing kills no mutant in the code it reaches. A -duplicate kills exactly the mutants an earlier test kills. A test that mocks -what it tests calls only methods it has replaced with its own mocks, so it -checks the mocks. That is a fact of the call graph, so it is listed with `-` -for confidence. So is a problem on a test or method the model couldn't be asked -about. Use `--all` to see every row. - -Perch also writes an HTML report. It shows each line of source with its problems -under it. For a survived mutant, it shows the original and the mutated version -of the line, and the tests that still pass. The report is one file, unless your -repository has more than 8 MB of source. Then each file gets its own page in -the `files/` directory next to `index.html`. +graph. They count against the score. Source files are listed worst first, and +test files only when something in them needs fixing. + +Under the tables, Perch lists the methods to add a test to, most survived +mutants first. Each has a line of numbers and, under it, the test to add, from +its surest survived mutant: the edit a new test has to tell apart, and the line +it is on. `--all` lists every method, and every survived mutant with the id +that `perch close` takes. Test problems follow, by file. The confidence is how +sure Perch is. A test that checks nothing kills no mutant in the code it +reaches. A duplicate kills exactly the mutants an earlier test kills. A test +that mocks what it tests calls only methods it has replaced with its own mocks, +so it checks the mocks. That is a fact of the call graph, so it is listed with +`-` for confidence. So is a problem on a test or method the model couldn't be +asked about. + +Perch also writes an HTML report. It opens on the methods to add a test to, +each with its score, how many tests reach it, and the test to add; the mutants +fold out under the row. Source files are a tree of directories, worst first, +with each score coloured against 60% and 80% as Stryker colours its own. A +file's page shows its code with a mark in the gutter on each line where a +mutant survived; press the mark to see each edit, the line both ways, and a +Copy prompt button, or step through the survivors with the Next survived +button. The report is one file, unless your repository has more than 8 MB of +source. Then each file gets its own page in the `files/` directory next to +`index.html`. ## How it differs from a coverage report @@ -200,17 +213,15 @@ changed on the branch: ```console $ perch coverage --since main -src/cart.ts - ID Line Problem Confidence Test or method Note - 20fb814a 12 survived 86% applyDiscount With `1` instead of `0`, none of the 5 te… - 1be27338 12 survived 85% applyDiscount With `>` instead of `>=`, none of the 5 t… - 0399f832 12 survived 84% applyDiscount With `101` instead of `100`, none of the … - 883b4b21 12 survived 81% applyDiscount With `false` as the condition, none of th… +Where to add tests + Method Where Killed Survived Tests + applyDiscount src/cart.ts:11 6 of 10 4 5 + Add a test that asserts on what line 12 returns. 3 more edits survive. test/cart.test.ts ID Line Problem Confidence Test or method Note 427fce7a 22 redundant 76% applyDiscount > tak… Kills the same mutants as applyDis… -shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere +shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere, --all for every mutant Report: .perch/coverage/index.html 28 requests 0 tokens in $0.0000 ``` diff --git a/src/coverage-advice.js b/src/coverage-advice.js new file mode 100644 index 0000000..edac7bc --- /dev/null +++ b/src/coverage-advice.js @@ -0,0 +1,40 @@ +/** + * What test to add for a survived mutant, in one sentence, from the kind of edit it is. The sentence is fixed wording around the + * report's own facts: the line, and the text the edit changed. It says what a test has to tell apart, not how to write it. + */ + +/** A piece of code in a sentence, cut to one line of it. */ +const edit = text => { + const one = String(text ?? '').trim().replace(/\s+/g, ' '); + return `\`${one.length > 60 ? `${one.slice(0, 57)}…` : one}\``; +}; + +/** The other value of a forced condition: a test is missing for the case the mutant removed. */ +const otherBranch = to => (/^(true|True)$/.test(String(to).trim()) ? 'false' : 'true'); + +export function adviceFor(mutant) { + const line = mutant.line; + switch (mutant.kind) { + case 'body': return 'Add a test that fails when the method does nothing.'; + case 'removal': return `Add a test that fails when ${edit(mutant.from)} is not called.`; + case 'block': return `Add a test that fails when the block at line ${line} is skipped.`; + case 'condition': return `Add a test in which the condition at line ${line} is ${otherBranch(mutant.to)}, and assert on what follows.`; + case 'boundary': return `Add a test at the boundary of ${edit(mutant.from)} on line ${line}, where ${edit(mutant.from)} and ${edit(mutant.to)} give different results.`; + case 'logic': return `Add a test in which only one side of ${edit(mutant.from)} on line ${line} holds.`; + case 'string': return `Add a test that asserts on the text ${edit(mutant.from)} at line ${line}.`; + case 'number': return `Add a test that asserts on the value ${edit(mutant.from)} at line ${line}.`; + case 'return': return `Add a test that asserts on what line ${line} returns.`; + case 'arithmetic': case 'update': return `Add a test that asserts on the arithmetic at line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)}.`; + default: return `Add a test that asserts on the result of line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)} unnoticed.`; + } +} + +/** + * The test to add for a method, from its surest survived mutant, and how many more edits survive beside it. `mutants` are the + * method's listed survived mutants, surest first. + */ +export function methodAdvice(mutants) { + if (!mutants.length) return ''; + const more = mutants.length - 1; + return `${adviceFor(mutants[0])}${more ? ` ${more} more ${more === 1 ? 'edit survives' : 'edits survive'}.` : ''}`; +} diff --git a/src/coverage-html.js b/src/coverage-html.js index e9856b9..859c7ef 100644 --- a/src/coverage-html.js +++ b/src/coverage-html.js @@ -5,6 +5,7 @@ */ import { createHash } from 'node:crypto'; import { WORDMARK, WORDMARK_LIGHT } from './brand.js'; +import { methodAdvice } from './coverage-advice.js'; /** Source text and names come from the repository being read, so every one of them is escaped before it becomes markup. */ const ESCAPES = { '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }; @@ -27,10 +28,10 @@ const when = stamp => String(stamp ?? '').replace('T', ' ').replace(/:\d\d(\.\d+ const counted = (count, one, many = `${one}s`) => `${escape(count)} ${count === 1 ? one : many}`; /** - * The colour a percentage is drawn in. The bands are the ones Istanbul draws, so a page that looks like a coverage report reads - * like one too. They colour a number the report gave; they do not grade anything the number does not already say. + * The colour a percentage is drawn in: red under 60, amber to 80, green from there, the thresholds Stryker ships with, so a score + * here reads the same as one in its report. They colour a number the report gave; they do not grade anything it does not say. */ -const band = value => (value === null || value === undefined ? 'none' : value < 0.5 ? 'low' : value < 0.8 ? 'mid' : 'high'); +const band = value => (value === null || value === undefined ? 'none' : value < 0.6 ? 'low' : value < 0.8 ? 'mid' : 'high'); /** * A link to a line of a file: a section of this page, or with `page`, the page of its own that file was written to. The path @@ -224,44 +225,92 @@ const copyAll = ' ` + + `${isNew ? 'new' : ''}${escape(method.name)}${escape(method.path)}:${escape(method.line)}` + + `${cell(score, { count: `${escape(method.killed)}/${escape(method.mutants.length)}` })}${escape(survived.length)}${escape(method.tests.length)}` + + `${escape(advice)}
` + + ``; +} + /** - * A first review, in two short lists: the methods with a survived mutant, riskiest first, and the tests that could go or that - * reach outside the process. Each is ranked by what it costs to leave, and shows a few rows - * with a link to every one of them under All problems. The rankings are counted from the report; nothing is added to it. With - * --since the same lists hold only the problems in code the branch changed. + * A first review, in two lists: the methods to add a test to, most survived mutants first, and the tests that could go or that + * reach outside the process. Each row says what to do; the mutants behind a method are one click down. With --since the same + * lists hold only the problems in code the branch changed. */ function actionsView(report, index) { const TOP = 8; - const risk = id => index.methods.get(id)?.risk ?? 0; const onBranch = report.branch ? new Set(report.branch.findings) : null; const scoped = onBranch ? report.findings.filter(finding => onBranch.has(finding.id)) : report.findings; const changedOnly = onBranch ? '&changed=1' : ''; const all = (kinds, count) => `See all ${escape(count)} under All problems →`; - const table = (heads, rows) => `
${heads}${rows.join('')}
`; - const section = (title, lede, body, more) => (body - ? `

${title}

${lede ? `

${lede}

` : ''}
${body}${more ? `
${more}
` : ''}
` : ''); + const table = (heads, rows, classes = '') => `
${heads}${rows.join('')}
`; + const section = (title, body, more) => (body + ? `

${title}

${body}${more ? `
${more}
` : ''}
` : ''); const of = kinds => scoped.filter(finding => kinds.includes(finding.kind)); // What the changes since the compared run brought in is marked new and put first in its list: it is what the person reading // this is working on now. With --since, the lists are the branch's already, and nothing is marked. const fresh = new Set(report.branch ? [] : (report.diff?.findings?.new ?? []).map(finding => finding.id)); const newFirst = (a, b) => Number(fresh.has(b.id)) - Number(fresh.has(a.id)); - // Survived mutants, one row per method, riskiest first. - const missing = of(['survived']); - const perMethod = new Map(); - for (const finding of missing) if (!perMethod.has(finding.unit)) perMethod.set(finding.unit, finding); - const riskiest = [...perMethod.values()].sort((a, b) => newFirst(a, b) || risk(b.unit) - risk(a.unit) || a.path.localeCompare(b.path)); - const methodRows = riskiest.slice(0, TOP).map(finding => `${actionCells(finding, index, fresh.has(finding.id))}` - + `${escape(Math.round(risk(finding.unit)))}${dismissMark}`); - const methodsSection = section(`Survived mutants ${escape(missing.length)}${copyAll}`, '', - methodRows.length ? table('ProblemWhereDetailsRisk', methodRows) : '', - missing.length > TOP ? all(['survived'], missing.length) : ''); + // Methods to add a test to, every one with a survived mutant, paged by the table. + const methods = methodsToTest(report, index, onBranch); + const hasNew = entry => entry.survived.some(finding => fresh.has(finding.id)); + const methodRowsHtml = [...methods].sort((a, b) => Number(hasNew(b)) - Number(hasNew(a))).map(entry => methodRows(entry, index, { fresh })); + const methodsSection = section(`Where to add tests ${escape(methods.length)}${copyAll}`, + methodRowsHtml.length ? table(methodHeads(), methodRowsHtml, ' methods') : '', ''); // Test problems: duplicates, tests that check nothing, and tests that call a live service, the surest first. const testKinds = ['redundant', ...WEAK, 'infra']; const testProblems = of(testKinds).sort((a, b) => newFirst(a, b) || (b.probability ?? 2) - (a.probability ?? 2) || a.path.localeCompare(b.path)); const testRows = testProblems.slice(0, TOP).map(finding => `${actionCells(finding, index, fresh.has(finding.id))}${dismissMark}`); - const testsSection = section(`Test problems ${escape(testProblems.length)}${copyAll}`, '', + const testsSection = section(`Test problems ${escape(testProblems.length)}${copyAll}`, testRows.length ? table('ProblemWhereDetails', testRows) : '', testProblems.length > TOP ? all(testKinds, testProblems.length) : ''); @@ -284,32 +333,68 @@ const th = (name, numeric = true, bars = false, title = '') => `${escape(file.path)}` - + `${scoreCell(totals)}` - + `${survived}` - + `${escape(totals.no_coverage)}`; - }).join(''); - const totals = report.totals; - const foot = `All source files` - + `${scoreCell(totals)}${escape(totals.survived)}${escape(totals.no_coverage)}`; - const heads = th('File', false) - + th('Mutation score', true, true, 'Mutants some test is predicted to fail against, of every mutant in the file, the ones no test reaches included.') + const survivedIn = path => index.kinds(path).survived ?? 0; + const root = { name: '', path: '', dirs: new Map(), files: [] }; + for (const file of files) { + const parts = file.path.split('/'); + let node = root; + for (const part of parts.slice(0, -1)) { + if (!node.dirs.has(part)) node.dirs.set(part, { name: part, path: node.path ? `${node.path}/${part}` : part, dirs: new Map(), files: [] }); + node = node.dirs.get(part); + } + node.files.push(file); + } + const sum = node => { + const totals = { mutants: 0, killed: 0, no_coverage: 0, survived: 0, files: 0 }; + const add = other => { for (const key of Object.keys(totals)) totals[key] += other[key] ?? 0; }; + for (const file of node.files) add({ mutants: file.totals.mutants, killed: file.totals.killed, no_coverage: file.totals.no_coverage, survived: survivedIn(file.path), files: 1 }); + for (const dir of node.dirs.values()) add(sum(dir)); + node.totals = totals; + return totals; + }; + sum(root); + const joined = node => { + while (!node.files.length && node.dirs.size === 1) { const [only] = node.dirs.values(); node = { ...only, name: `${node.name}/${only.name}` }; } + return node; + }; + const score = totals => scoreValue(totals); + const worst = (a, b) => (score(a.totals) ?? 1) - (score(b.totals) ?? 1) || (b.totals.survived - a.totals.survived) || (b.totals.no_coverage - a.totals.no_coverage) || a.path.localeCompare(b.path); + // A large tree opens on its top level alone; a small one is open all the way down. + const closed = files.length > 200; + const counts = totals => `${scoreCell(totals)}${escape(totals.killed)}` + + `${escape(totals.survived)}${escape(totals.no_coverage)}`; + const rows = []; + const walk = (node, depth, parent) => { + const entries = [...[...node.dirs.values()].map(joined), ...node.files.map(file => ({ file, path: file.path, totals: { ...file.totals, survived: survivedIn(file.path) } }))].sort(worst); + for (const entry of entries) { + if (entry.file) { + rows.push(`${escape(entry.path.split('/').at(-1))}${counts(entry.totals)}`); + } else { + rows.push(` 0 ? ' data-closed' : ''}>${escape(entry.name)}/ ${escape(entry.totals.files)}${counts(entry.totals)}`); + walk(entry, depth + 1, entry.path); + } + } + }; + walk(root, 0, ''); + const totals = { ...report.totals, survived: report.totals.survived }; + const foot = `All source files ${escape(files.length)}${counts(totals)}`; + const heads = 'File' + + th('Mutation score', true, true, 'Mutants some test is predicted to fail against, of every mutant, the ones no test reaches included.') + + th('Killed', true, false, 'Mutants some test is predicted to fail against.') + th('Survived', true, false, 'Mutants no test kills, listed under All problems.') + th('No coverage', true, false, 'Mutants in methods no test reaches. Nothing was asked about them; they count against the score.'); return view('sources', 'Source files', '', `
` - + `
${heads}${rows}${foot}
`); + + `
${heads}${rows.join('')}${foot}
`); } -/** The test files: how many tests each has, how many are worth keeping, and why the rest are not. */ +/** The test files with something to fix, most first. The rest are counted under the table and shown on request. */ function testTable(report, index) { const files = report.files.filter(file => file.kind === 'test'); const notKept = file => file.totals.tests - file.totals.useful; @@ -317,16 +402,19 @@ function testTable(report, index) { const totals = file.totals; const kept = ratio(totals.useful, totals.tests); const count = value => `${escape(value ?? 0)}`; - return `${escape(file.path)}` + const fine = !notKept(file) && !totals.infra; + return `${escape(file.path)}` + `${cell(kept, { count: `${escape(totals.useful)}/${escape(totals.tests)}` })}` + count(totals.redundant) + count(totals.weak) + count(totals.infra) + ''; - }).join(''); + }); + const fine = files.filter(file => !notKept(file) && !file.totals.infra).length; const heads = th('File', false) + th('Quality', true, true, 'Tests worth keeping, of all the file\'s tests: ones that check something and repeat no other test.') + th('Duplicates', true, false, 'Tests that check the same thing with the same code as an earlier test.') + th('Checks nothing', true, false, 'Tests that would pass whatever the code they call does.') + th('Live services', true, false, 'Tests that call a real network service or database with nothing mocked.'); + const foot = fine ? `
${counted(fine, 'test file has', 'test files have')} nothing to fix.
` : ''; return view('tests', 'Tests', '', - `
${heads}${rows}
`); + `
${heads}${rows.join('')}
${foot}
`); } /** Every problem, filterable by kind, each linking to the line it is about. */ @@ -393,9 +481,9 @@ function methodState(method) { /** A method's survived mutants that are listed problems: one under the floor has nothing to act on. */ const listedMutants = (method, index) => method.findings.map(id => index.findings.get(id)).filter(finding => finding?.kind === 'survived'); -/** Tint, branch markers and the survived mutants for each line of a file, innermost method last so a nested function wins its lines. */ +/** Tint, branch markers and how many survived mutants each line of a file has, innermost method last so a nested function wins its lines. */ function lineMarks(file, index) { - const marks = file.lines.map(() => ({ state: '', branch: false, gap: false, starts: [] })); + const marks = file.lines.map(() => ({ state: '', branch: false, gaps: 0, starts: [] })); const at = line => marks[line - 1]; const units = file.kind === 'test' ? file.tests.map(id => index.tests.get(id)) : file.methods.map(id => index.methods.get(id)); const spans = units.filter(Boolean).sort((a, b) => (b.end_line - b.line) - (a.end_line - a.line)); @@ -408,13 +496,13 @@ function lineMarks(file, index) { at(unit.line)?.starts.push(unit); if (file.kind === 'test') continue; for (const line of unit.branches) if (at(line)) at(line).branch = true; - for (const finding of listedMutants(unit, index)) if (at(finding.line)) at(finding.line).gap = true; + for (const finding of listedMutants(unit, index)) if (at(finding.line)) at(finding.line).gaps += 1; } return marks; } -/** The tests nearest a method, the ones calling it directly first. Past `REACH_SHOWN`, a count stands for the rest. */ -const REACH_SHOWN = 50; +/** The tests nearest a method, the ones calling it directly first: the eight perch asked about its mutants. A count stands for the rest. */ +const REACH_SHOWN = 8; function reachedBy(method, index) { const useful = new Set(method.useful); const nearest = [...method.tests].sort((a, b) => a.depth - b.depth); @@ -424,7 +512,10 @@ function reachedBy(method, index) { + (rest > 0 ? `
  • ${escape(rest)} more ${rest === 1 ? 'test' : 'tests'}
    • ` : ''); } -/** The line above a method: its name, how many of its mutants its tests kill, and how many tests reach it, with the tests one click away. */ +/** + * The line above a method: its name, how many of its mutants its tests kill, the test to add when some survived, and how many + * tests reach it, with the tests one click away. + */ function methodBar(method, index) { const ran = !method.covered && method.mutants.length ? `${escape(method.mutants.length)} ${method.mutants.length === 1 ? 'mutant' : 'mutants'}, no coverage` : method.mutants.length ? `${escape(method.killed)} of ${escape(method.mutants.length)} mutants killed` : ''; @@ -432,7 +523,9 @@ function methodBar(method, index) { const tests = count ? `
    ${count} ${count === 1 ? 'test reaches' : 'tests reach'} it${method.useful.length !== count ? `, ${method.useful.length} worth keeping` : ''}
    ` : 'No test reaches it'; - return `
    ${escape(method.name)}${ran ? `${ran}` : ''}${tests}
    `; + const survived = survivedOf(method, index); + const advice = survived.length ? `${escape(methodAdvice(survived.map(finding => mutantOf(method, finding)).filter(Boolean)))}` : ''; + return `
    ${escape(method.name)}${ran ? `${ran}` : ''}${tests}${advice}
    `; } /** "a", "a and b", "a, b and c". */ @@ -440,14 +533,10 @@ const listed = items => (items.length < 2 ? items.join('') : `${items.slice(0, - /** How sure perch is, said only when it is not very: a verdict at 95% needs no number, one at 63% does. */ const unsure = probability => (typeof probability === 'number' && probability < 0.8 ? ` ${escape(percent(probability))} sure` : ''); -/** The note under a survived mutant: the line as written and as changed, and which tests miss it, with its buttons. */ -function mutantNote(method, finding, index) { - const mutant = method.mutants.find(item => item.id === finding.mutant); - const diff = mutant ? `
    - ${escape(mutant.original.trim())}\n+ ${escape(mutant.mutated.trim())}
    ` : ''; - const missed = mutant ? mutant.asked.filter(id => !mutant.killed_by.includes(id)).map(id => unitLink(id, index)) : []; - return `

    ${escape(problemName('survived'))} ` - + `${escape(finding.note)}${unsure(finding.probability)}

    ${acts()}${dismissMark}
    ${diff}` - + (missed.length ? `` : '') + '
    '; +/** The note under a line with survived mutants: each one's edit, how sure perch is, the line both ways, and its buttons. Folded until the gutter mark is pressed. */ +function gapNote(method, findings, index, line) { + const head = `
    ${counted(findings.length, 'survived mutant')} on line ${escape(line)}
    `; + return `
    ${head}${findings.map(finding => mutantBlock(method, finding, index)).join('')}
    `; } /** One problem in a note under the code: its name and the fact, with Copy prompt and the close mark on the right. */ @@ -488,7 +577,9 @@ function notesOf(file, index, report) { } } else { for (const method of file.methods.map(id => index.methods.get(id)).filter(Boolean)) { - for (const finding of listedMutants(method, index)) { placed.add(finding.id); add(finding.line, mutantNote(method, finding, index)); } + const byLine = new Map(); + for (const finding of survivedOf(method, index)) { placed.add(finding.id); if (!byLine.has(finding.line)) byLine.set(finding.line, []); byLine.get(finding.line).push(finding); } + for (const [line, findings] of byLine) add(line, gapNote(method, findings, index, line)); } } for (const finding of report.findings) { @@ -514,32 +605,42 @@ function fileStats(file) { + stat('survived', escape(totals.survived), 'Mutants no test kills, listed below.'); } -/** A file's problems at the top of its page, each linking to its line, so they are read without scrolling the source. */ -function fileProblems(file, report, index) { - const own = [...index.findingsIn(file.path)].sort((a, b) => a.line - b.line || a.kind.localeCompare(b.kind)); - return own.length ? `

    Problems ${escape(own.length)}${copyAll}

    ${actionTable(own, index)}
    ` : ''; +/** The methods of a file to add a test to, the few with most survivors, in the shape the summary lists them. */ +function fileMethods(file, report, index) { + const own = new Set(file.methods); + const entries = methodsToTest(report, index).filter(entry => own.has(entry.method.id)); + if (!entries.length) return ''; + const SHOWN = 5; + const rest = entries.length - SHOWN; + return `

    Where to add tests ${escape(entries.length)}${copyAll}

    ` + + `
    ${methodHeads()}${entries.slice(0, SHOWN).map(entry => methodRows(entry, index)).join('')}
    ` + + (rest > 0 ? `
    ${escape(rest)} more ${rest === 1 ? 'method' : 'methods'} in this file under Summary →
    ` : '') + '
    '; } -/** One file's page: its numbers, a legend, and every line of it with tint, markers and the notes that belong under each. */ +/** One file's page: its numbers, the methods to test, a toolbar with the legend and a way through the survivors, then every line with tint, marks and notes. */ function fileView(file, index, report) { const marks = lineMarks(file, index), notes = notesOf(file, index, report); const methods = file.methods.map(id => index.methods.get(id)).filter(Boolean); + const gaps = marks.filter(mark => mark.gaps).length; const legend = file.kind === 'test' ? 'a test with a problem' : (methods.length ? 'every mutant killeda mutant survivedno coverage' : '') - + '◆branch▲survived mutant'; + + '◆branch▲survived mutants, press for the edits'; + const tools = file.kind === 'source' && gaps + ? `${escape(gaps)} ${gaps === 1 ? 'line' : 'lines'}` + + '' + : ''; const rows = file.lines.map((text, at) => { const line = at + 1, mark = marks[at]; const bars = file.kind === 'source' ? mark.starts.map(method => methodBar(method, index)).join('') : ''; - const gutter = mark.gap ? '▲' : mark.branch ? '◆' : ''; - const classes = ['l', mark.state, mark.gap ? 'gapline' : ''].filter(Boolean).join(' '); + const gutter = mark.gaps ? `` : mark.branch ? '◆' : ''; + const classes = ['l', mark.state, mark.gaps ? 'gapline' : ''].filter(Boolean).join(' '); const under = (notes.get(line) ?? []).join(''); return `${bars}
    ${line}${gutter}${escape(text) || ' '}
    ${under}`; }).join(''); return ``; } @@ -842,7 +943,6 @@ ul.reach{margin:6px 0 2px;padding-left:18px;font-size:13px} .verdict{color:var(--body)}.verdict b{margin-right:6px}.verdict.bad b{color:var(--low-ink)}.note.test .verdict.bad b{color:var(--mid)} .unsure{margin-left:6px;color:var(--faint);font-size:12.5px} .facts{margin:6px 0 0;padding-left:18px;color:var(--muted);font-size:13px} -body.nonotes .note{display:none} [data-dismissed]{display:none!important} /* The toast that says what a button did. */ @@ -850,8 +950,37 @@ body.nonotes .note{display:none} .toast code{color:var(--accent);word-break:break-all}.toast button{margin-left:8px} .toast .prompt-text{display:block;width:min(680px,calc(100vw - 64px));margin-top:8px;padding:8px 10px;background:var(--bg);color:var(--ink);border:1px solid var(--line);border-radius:8px;font:12px/1.5 var(--mono)} +/* Methods to add a test to: name and place, numbers, the test to add, and the mutants folded under the row. */ +table.methods td.act{white-space:normal;min-width:220px}table.methods .act{display:grid;grid-template-columns:28px 1fr;align-items:start} +table.methods .act .fold{grid-row:1/3}.act .name{font-family:var(--mono);font-size:13px;color:var(--ink);font-weight:600;overflow-wrap:anywhere} +.act .path{display:block;grid-column:2;font-size:12px;color:var(--muted)}table.methods .act .new-badge{grid-column:2} +.fold{display:inline-flex;align-items:center;justify-content:center;width:22px;height:22px;margin-right:6px;border:0;border-radius:6px;background:transparent;color:var(--faint);font-size:17px;line-height:1;cursor:pointer;transition:transform .15s;vertical-align:-3px} +.fold:hover{background:var(--raised);color:var(--ink)} +tr[data-open] .fold,tr.dir:not([data-closed]) .fold{transform:rotate(90deg)} +table.methods td.do{white-space:normal;min-width:260px;max-width:440px}.advice{color:var(--body)} +tr.detail td{padding:0 18px 12px 46px;background:var(--raised);white-space:normal} +ul.edits{margin:0;padding:10px 0 0 18px;color:var(--body);font-size:13.5px}ul.edits li{margin:2px 0}ul.edits .at{font-family:var(--mono);font-size:12.5px;margin-right:4px} +.mutant{padding:8px 12px;border:1px solid var(--line);border-left:2px solid var(--low);border-radius:8px;background:var(--panel)} +.mutant .at{font-family:var(--mono);font-size:12.5px;margin-right:6px} +.note .mutant{margin-top:8px} + +/* The source tree: directories fold, their numbers are their files' summed, each level sits in from its parent. */ +table.tree td.path{padding-left:calc(18px + var(--depth,0)*20px)} +tr.dir td{color:var(--ink);font-weight:600}tr.dir td.path{color:var(--ink);cursor:pointer}tr.dir .muted{font-weight:500;font-size:12px} + +/* The file page's toolbar: the legend, a way through the lines with survivors, and whether every note is open. */ +.legend .pager-gaps{display:inline-flex;align-items:center;gap:6px;margin-left:auto} +.legend .pager-gaps button,.legend .toggle{padding:3px 10px;border:1px solid var(--line);border-radius:7px;background:transparent;color:var(--muted);font:500 12px/1.4 var(--sans);cursor:pointer} +.legend .pager-gaps button:hover{color:var(--ink);border-color:color-mix(in srgb,var(--accent) 45%,var(--line))} +.legend .toggle{margin-left:0} +.file-methods{width:calc(100% - 48px);max-width:1132px;margin:0 auto 16px} +button.gapmk{padding:0 4px;min-width:18px;border:0;border-radius:5px;background:color-mix(in srgb,var(--low) 16%,transparent);color:var(--low);font:600 10px/1.6 var(--mono);cursor:pointer} +button.gapmk:hover{background:color-mix(in srgb,var(--low) 30%,transparent)} +.l.gapline.noted{background:color-mix(in srgb,var(--low) 10%,transparent)} +.note.closed{display:none}body.allnotes .note.closed{display:block} + @media (max-width:640px){.wrap{padding:0 16px}.top{position:static}.where{margin-left:0}.file-head,.stats,.legend{padding-left:16px;padding-right:16px} -.code,.file-problems{width:calc(100% - 32px)}.note,.mbar{margin-left:16px;margin-right:16px}} +.code,.file-methods{width:calc(100% - 32px)}.note,.mbar{margin-left:16px;margin-right:16px}} `; /** @@ -879,7 +1008,7 @@ const SCRIPT = String.raw` if (target) { onSummary = false; var row = line ? target.querySelector('[data-n="' + line + '"]') : null; - if (row) { row.classList.add('flash'); row.scrollIntoView({ block: 'center' }); } else window.scrollTo(0, 0); + if (row) { row.classList.add('flash'); openNote(target, line, true); row.scrollIntoView({ block: 'center' }); } else window.scrollTo(0, 0); return; } // One view at a time: the one the hash names, or the first. A link into All problems may name the kinds to show. @@ -907,13 +1036,18 @@ const SCRIPT = String.raw` // page of them. A table no longer than a page has no pager, and still filters through render. var SIZES = [25, 50, 100, 0]; function render(state) { - var rows = Array.prototype.slice.call(state.table.tBodies[0].rows); + var every = Array.prototype.slice.call(state.table.tBodies[0].rows); + var rows = every.filter(function (row) { return !row.hasAttribute('data-detail-for'); }); var kept = rows.filter(function (row) { return !row.hasAttribute('data-out'); }); var size = state.size || Math.max(kept.length, 1), pages = Math.max(1, Math.ceil(kept.length / size)); state.page = Math.min(Math.max(1, state.page), pages); var from = (state.page - 1) * size, to = Math.min(from + size, kept.length); rows.forEach(function (row) { row.hidden = true; }); kept.slice(from, to).forEach(function (row) { row.hidden = false; }); + every.forEach(function (row) { + var owner = row.hasAttribute('data-detail-for') && state.table.querySelector('tr.method[data-method="' + row.getAttribute('data-detail-for').replace(/"/g, '\\"') + '"]'); + if (owner) row.hidden = owner.hidden || !owner.hasAttribute('data-open'); + }); if (rows.length <= SIZES[0]) { state.bar.hidden = true; return; } state.bar.hidden = false; var options = SIZES.map(function (value) { return ''; }).join(''); @@ -941,7 +1075,79 @@ const SCRIPT = String.raw` render(state); } function refresh(table) { if (table.pager) { table.pager.page = 1; render(table.pager); } } - document.querySelectorAll('#summary table.grid').forEach(paged); + document.querySelectorAll('#summary table.grid:not(.tree)').forEach(paged); + + // A method's mutants fold under its row. + document.addEventListener('click', function (event) { + var opener = event.target.closest('[data-open-row]'); + if (!opener) return; + var row = opener.closest('tr'); + if (row.hasAttribute('data-open')) row.removeAttribute('data-open'); else row.setAttribute('data-open', ''); + var table = row.closest('table'); + Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (other) { + if (other.getAttribute('data-detail-for') === row.getAttribute('data-method')) other.hidden = !row.hasAttribute('data-open'); + }); + }); + + // The source tree: a directory folds its rows, and a filter shows the files that match with the directories above them. + var treeFilter = ''; + function applyTree(table) { + var rows = Array.prototype.slice.call(table.tBodies[0].rows), byPath = {}; + rows.forEach(function (row) { byPath[row.getAttribute('data-path')] = row; }); + var matches = function (row) { + if (!treeFilter) return true; + if (row.classList.contains('dir')) return rows.some(function (other) { return !other.classList.contains('dir') && other.getAttribute('data-path').indexOf(row.getAttribute('data-path') + '/') === 0 && other.getAttribute('data-path').toLowerCase().indexOf(treeFilter) >= 0; }); + return row.getAttribute('data-path').toLowerCase().indexOf(treeFilter) >= 0; + }; + rows.forEach(function (row) { + var folded = false, parent = row.getAttribute('data-parent'); + while (!treeFilter && parent) { var up = byPath[parent]; if (!up) break; if (up.hasAttribute('data-closed')) folded = true; parent = up.getAttribute('data-parent'); } + row.hidden = folded || !matches(row); + }); + } + document.querySelectorAll('table.tree').forEach(applyTree); + document.addEventListener('click', function (event) { + var toggle = event.target.closest('[data-toggle-dir]') || (event.target.closest('tr.dir td.path') && !event.target.closest('a') ? event.target.closest('tr.dir td.path') : null); + if (!toggle) return; + var row = toggle.closest('tr'); + if (row.hasAttribute('data-closed')) row.removeAttribute('data-closed'); else row.setAttribute('data-closed', ''); + applyTree(row.closest('table')); + }); + + // The test files with nothing to fix, on request. + document.addEventListener('click', function (event) { + var show = event.target.closest('[data-show-fine]'); + if (!show) return; + var table = show.closest('.panel').querySelector('table'); + Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (row) { if (row.hasAttribute('data-fine')) row.removeAttribute('data-out'); }); + refresh(table); + show.parentNode.hidden = true; + }); + + // Notes under the code open from the mark in the gutter, one at a time, or all at once from the toolbar. + function openNote(section, line, open) { + var note = null, row = section.querySelector('[data-n="' + line + '"]'); + Array.prototype.slice.call(section.querySelectorAll('.note[data-line]')).forEach(function (candidate) { if (candidate.getAttribute('data-line') === String(line)) note = candidate; }); + if (!note) return; + var closed = open === undefined ? !note.classList.contains('closed') : !open; + note.classList.toggle('closed', closed); + if (row) row.classList.toggle('noted', !closed); + } + document.addEventListener('click', function (event) { + var mark = event.target.closest('[data-toggle-note]'); + if (mark) { openNote(mark.closest('section.file'), mark.getAttribute('data-toggle-note')); return; } + var step = event.target.closest('[data-gap]'); + if (!step) return; + var section = step.closest('section.file'); + var lines = Array.prototype.slice.call(section.querySelectorAll('.l.gapline')).map(function (row) { return Number(row.getAttribute('data-n')); }); + if (!lines.length) return; + var middle = window.scrollY + window.innerHeight / 2, current = null; + Array.prototype.slice.call(section.querySelectorAll('.l.gapline')).forEach(function (row) { if (row.getBoundingClientRect().top + window.scrollY <= middle) current = Number(row.getAttribute('data-n')); }); + var at = current === null ? -1 : lines.indexOf(current); + var next = lines[Math.min(lines.length - 1, Math.max(0, at + Number(step.getAttribute('data-gap'))))]; + if (current !== null && at + Number(step.getAttribute('data-gap')) < 0) next = lines[0]; + location.hash = 'file=' + encodeURIComponent(section.getAttribute('data-path')) + '&line=' + next; + }); document.querySelectorAll('table.sortable').forEach(function (table) { var heads = Array.prototype.slice.call(table.querySelectorAll('thead th')); @@ -964,7 +1170,9 @@ const SCRIPT = String.raw` document.querySelectorAll('.filter-files').forEach(function (filter) { filter.addEventListener('input', function () { var text = filter.value.toLowerCase(); + treeFilter = text; document.querySelectorAll('table.files').forEach(function (table) { + if (table.classList.contains('tree')) { applyTree(table); return; } Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (row) { if (text && row.getAttribute('data-path').toLowerCase().indexOf(text) < 0) row.setAttribute('data-out', ''); else row.removeAttribute('data-out'); }); @@ -1013,9 +1221,11 @@ const SCRIPT = String.raw` } function shownIn(scope) { var ids = []; + var take = function (id) { if (fixData.steps[id] && ids.indexOf(id) < 0) ids.push(id); }; + scope.querySelectorAll('tr.method[data-ids]').forEach(function (row) { if (row.getClientRects().length) row.getAttribute('data-ids').split(',').forEach(take); }); scope.querySelectorAll('[data-finding]').forEach(function (element) { - var id = element.getAttribute('data-finding'); - if (element.getClientRects().length && fixData.steps[id] && ids.indexOf(id) < 0) ids.push(id); + if (element.getClientRects().length && !element.closest('tr.detail')) take(element.getAttribute('data-finding')); + else if (element.closest('tr.detail') && element.closest('tr.detail').previousElementSibling && element.closest('tr.detail').previousElementSibling.getClientRects().length) take(element.getAttribute('data-finding')); }); return ids; } @@ -1046,6 +1256,12 @@ const SCRIPT = String.raw` document.addEventListener('click', function (event) { var owner = event.target.closest('[data-finding]'); if (event.target.closest('[data-copy]') && owner) { copyPrompt(promptFor([owner.getAttribute('data-finding')])); return; } + var byIds = event.target.closest('[data-copy-ids]'); + if (byIds) { + var ids = byIds.closest('tr').getAttribute('data-ids').split(',').filter(function (id) { return fixData.steps[id] && dismissedIds().indexOf(id) < 0; }); + if (ids.length) copyPrompt(promptFor(ids), ids.length); else say('Every mutant here is dismissed.'); + return; + } var all = event.target.closest('[data-copy-all]'); if (all) { var ids = shownIn(all.closest('.panel') || document); @@ -1105,7 +1321,7 @@ const SCRIPT = String.raw` document.querySelectorAll('.notes-toggle').forEach(function (box) { box.addEventListener('change', function () { - document.body.classList.toggle('nonotes', !box.checked); + document.body.classList.toggle('allnotes', box.checked); document.querySelectorAll('.notes-toggle').forEach(function (other) { other.checked = box.checked; }); }); }); diff --git a/src/coverage-report.js b/src/coverage-report.js index e4ab3a3..08d60de 100644 --- a/src/coverage-report.js +++ b/src/coverage-report.js @@ -5,6 +5,7 @@ */ import { BELIEVED } from './questions.js'; import { bold, COLOR, dim, keepEnd, keepStart, percent, relative, sureness, table, TOP, WIDTH } from './report.js'; +import { methodAdvice } from './coverage-advice.js'; /** Every kind of problem a coverage report can list, and so everything `--filter kind=` accepts. */ export const COVERAGE_KINDS = ['survived', 'redundant', 'checks_nothing', 'mocked', 'infra']; @@ -83,8 +84,10 @@ const painted = (lines, color, total = false) => lines.map((line, index) => (ind function sourceTable(report, onList, { width, color }) { + const score = totals => (totals.mutants ? totals.killed / totals.mutants : 1); + // Worst first: the lowest score, then the most mutants no test reaches, then the path, so the files to work on head the table. const files = report.files.filter(file => file.kind === 'source' && file.totals.methods) - .sort((a, b) => a.path.localeCompare(b.path)); + .sort((a, b) => score(a.totals) - score(b.totals) || b.totals.no_coverage - a.totals.no_coverage || a.path.localeCompare(b.path)); if (!files.length) return ''; // The mutation score is the mutants some test is predicted to kill, of every mutant, the ones no test reaches included, as // Stryker and PIT count it. Survived is counted from the problems listed under the table, at the same floor, so the two agree; @@ -99,18 +102,62 @@ function sourceTable(report, onList, { width, color }) { return painted(fitted(header, rows, ['left', 'right', 'right', 'right'], { width }), color, true).join('\n'); } +/** The test files with something to fix, most first; the ones with nothing to fix are one line. */ function testTable(report, { width, color }) { - const files = report.files.filter(file => file.kind === 'test' && file.totals.tests) - .sort((a, b) => a.path.localeCompare(b.path)); - if (!files.length) return ''; + const all = report.files.filter(file => file.kind === 'test' && file.totals.tests); + if (!all.length) return ''; + const toFix = totals => totals.tests - totals.useful + totals.infra; + const files = all.filter(file => toFix(file.totals)).sort((a, b) => toFix(b.totals) - toFix(a.totals) || a.path.localeCompare(b.path)); + const fine = all.length - files.length; + const nothing = fine ? `${fine === all.length ? 'All ' : ''}${plural(fine, 'test file has', 'test files have')} nothing to fix.` : ''; + if (!files.length) return nothing; // Quality is the tests worth keeping, of all of them; the next three are why the rest are not, or what they touch. const header = ['Test files', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']; const row = (name, totals) => [name, `${percent(totals.useful / totals.tests)} (${totals.useful} of ${totals.tests})`, totals.redundant, totals.weak, totals.infra]; const rows = files.map(file => row(relative(file.path), file.totals)); rows.push(row('All tests', report.totals)); - return painted(fitted(header, rows, ['left', 'right', 'right', 'right', 'right'], { width }), color, true).join('\n'); + return [painted(fitted(header, rows, ['left', 'right', 'right', 'right', 'right'], { width }), color, true).join('\n'), nothing].filter(Boolean).join('\n'); } +/** Text wrapped at a width, each line indented, for a sentence that has to be read whole. */ +function wrapped(text, width, indent) { + const lines = []; + let line = ''; + for (const word of text.split(' ')) { + if (line && line.length + 1 + word.length > width) { lines.push(line); line = word; } else line = line ? `${line} ${word}` : word; + } + if (line) lines.push(line); + return lines.map(part => `${indent}${part}`); +} + +/** + * The methods to add a test to, most survived mutants first: a line of numbers for each, the way a scan lists a method, and under + * it the test to add, from its surest survived mutant, written out whole. A tie goes to the method fewer tests reach, then to the + * file and line. The survived mutants themselves, with the ids `perch close` takes, are listed by file with --all. + */ +function methodBlock(report, findings, { width, color, all }) { + const survived = findings.filter(finding => finding.kind === 'survived'); + if (!survived.length) return ''; + const methods = new Map(report.methods.map(method => [method.id, method])); + const entries = [...Map.groupBy(survived, finding => finding.unit)].map(([id, list]) => ({ method: methods.get(id), + list: [...list].sort((a, b) => (b.probability ?? 0) - (a.probability ?? 0) || (a.line ?? 0) - (b.line ?? 0)) })) + .filter(entry => entry.method) + .sort((a, b) => b.list.length - a.list.length || a.method.tests.length - b.method.tests.length || a.method.path.localeCompare(b.method.path) || a.method.line - b.method.line); + const shown = all ? entries : entries.slice(0, TOP); + const HEAD = ['Method', 'Where', 'Killed', 'Survived', 'Tests']; + const rows = shown.map(({ method, list }) => ({ cells: [method.name, `${relative(method.path)}:${method.line}`, `${method.killed} of ${method.mutants.length}`, String(list.length), String(method.tests.length)], + advice: methodAdvice(list.map(finding => method.mutants.find(mutant => mutant.id === finding.mutant)).filter(Boolean)) })); + const widths = HEAD.map((name, column) => Math.max(name.length, ...rows.map(row => row.cells[column].length))); + const pad = (cells, aligns) => cells.map((cell, column) => (aligns[column] === 'right' ? cell.padStart(widths[column]) : cell.padEnd(widths[column]))).join(' ').trimEnd(); + const aligns = ['left', 'left', 'right', 'right', 'right']; + const lines = [bold('Where to add tests', color), dim(` ${pad(HEAD, aligns)}`, color)]; + for (const row of rows) lines.push(` ${pad(row.cells, aligns)}`, ...wrapped(row.advice, Math.max(40, width - 4), ' ')); + return lines.join('\n'); +} + +/** How many methods have a survived mutant among the findings. */ +const methodsWithSurvivors = findings => new Set(findings.filter(finding => finding.kind === 'survived').map(finding => finding.unit)).size; + /** * The problems, one block per file and one line per problem, laid out the way `perch scan` lays out its own so the two read as one * tool. The note is the last column and the one that gives way to the terminal, since it is the longest and is a sentence the id @@ -147,15 +194,19 @@ function findingBlocks(findings, { width, color }) { } /** - * What a coverage run prints on stdout: the source files, the test files, then the problems grouped by file. An unasked-for list is - * cut to the top ten; `--all` or a filter is the asking, and gets every one. With --since it is what the branch changed and the - * problems in it instead, and the whole repository is one line on stderr. + * What a coverage run prints on stdout: the source files worst first, the test files with something to fix, the methods to add a + * test to, then the test problems by file. An unasked-for list is cut to the top ten; `--all` or a filter is the asking, and gets + * every method and every survived mutant with its id. With --since it is what the branch changed and the problems in it instead, + * and the whole repository is one line on stderr. */ export function formatCoverage(report, { min = BELIEVED, filters = [], all = false, width = WIDTH(), color = COLOR() } = {}) { const listed = listedFindings(report, { min, filters }); - const shown = all || filters.length ? listed : ranked(listed).slice(0, TOP); - if (report.branch) return shown.length ? findingBlocks(shown, { width, color }) : `No problems in code changed since ${report.branch.ref}.`; - const parts = [sourceTable(report, listedFindings(report, { min }), { width, color }), testTable(report, { width, color }), findingBlocks(shown, { width, color })]; + const every = all || filters.length > 0; + const others = listed.filter(finding => finding.kind !== 'survived'); + const blocks = [methodBlock(report, listed, { width, color, all: every }), + findingBlocks(every ? [...others, ...listed.filter(finding => finding.kind === 'survived')] : ranked(others).slice(0, TOP), { width, color })]; + if (report.branch) return listed.length ? blocks.filter(Boolean).join('\n\n') : `No problems in code changed since ${report.branch.ref}.`; + const parts = [sourceTable(report, listedFindings(report, { min }), { width, color }), testTable(report, { width, color }), ...blocks]; const text = parts.filter(Boolean).join('\n\n'); return text || 'No source files or tests found.'; } @@ -202,7 +253,10 @@ export function coverageCount(report, { min = BELIEVED, filters = [], all = fals const others = listedFindings({ ...report, branch: null }, { min, filters }).length - listed; parts.push(`${plural(listed, 'problem')} in changed code${others ? `, ${others} elsewhere` : ''}`); } else parts.push(plural(listed, 'problem')); - if (!all && !filters.length && listed > TOP) parts.push(`${TOP} shown, --all for the rest`); + // Without --all, the methods past the top ten and every survived mutant's id are off the screen. + const survived = listedFindings(report, { min, filters }), methods = methodsWithSurvivors(survived); + if (!all && !filters.length && methods > TOP) parts.push(`${TOP} of ${plural(methods, 'method')} shown, --all for the rest`); + else if (!all && !filters.length && (methods || survived.length - methods > TOP)) parts.push('--all for every mutant'); const failed = report.failed.length; if (failed) parts.push(`${failed} could not be asked (--json)`); // A config that would not load means every test the parser found was read instead, which changes what is counted. diff --git a/test/coverage-advice.test.js b/test/coverage-advice.test.js new file mode 100644 index 0000000..badc0ff --- /dev/null +++ b/test/coverage-advice.test.js @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { adviceFor, methodAdvice } from '../src/coverage-advice.js'; + +const mutant = (kind, from, to, line = 12) => ({ kind, from, to, line }); + +describe('what test to add for a survived mutant', () => { + it('names the edit a test has to tell apart, by its kind', () => { + expect(adviceFor(mutant('body', '{ save(order); }', '{}'))).toBe('Add a test that fails when the method does nothing.'); + expect(adviceFor(mutant('removal', 'save(order);', ''))).toBe('Add a test that fails when `save(order);` is not called.'); + expect(adviceFor(mutant('block', '{ total += 1; }', '{}'))).toBe('Add a test that fails when the block at line 12 is skipped.'); + expect(adviceFor(mutant('condition', 'percent > 100', 'true'))).toBe('Add a test in which the condition at line 12 is false, and assert on what follows.'); + expect(adviceFor(mutant('condition', 'percent > 100', 'false'))).toBe('Add a test in which the condition at line 12 is true, and assert on what follows.'); + expect(adviceFor(mutant('boundary', '>=', '>'))).toBe('Add a test at the boundary of `>=` on line 12, where `>=` and `>` give different results.'); + expect(adviceFor(mutant('logic', '&&', '||'))).toBe('Add a test in which only one side of `&&` on line 12 holds.'); + expect(adviceFor(mutant('string', "'An item is out of stock'", "''"))).toBe("Add a test that asserts on the text `'An item is out of stock'` at line 12."); + expect(adviceFor(mutant('number', '100', '101'))).toBe('Add a test that asserts on the value `100` at line 12.'); + expect(adviceFor(mutant('return', 'total', '0'))).toBe('Add a test that asserts on what line 12 returns.'); + expect(adviceFor(mutant('arithmetic', '*', '/'))).toBe('Add a test that asserts on the arithmetic at line 12, where `*` can become `/`.'); + expect(adviceFor(mutant('update', '+=', '-='))).toBe('Add a test that asserts on the arithmetic at line 12, where `+=` can become `-=`.'); + expect(adviceFor(mutant('not', '!ready', 'ready'))).toBe('Add a test that asserts on the result of line 12, where `!ready` can become `ready` unnoticed.'); + expect(adviceFor(mutant('boolean', 'true', 'false'))).toBe('Add a test that asserts on the result of line 12, where `true` can become `false` unnoticed.'); + }); + + it('cuts a long call to one line of it', () => { + const call = `references.push(...phpUseReferences(node, index, scope, imports, aliases, 'use', 'function'));`; + expect(adviceFor(mutant('removal', ` ${call}\n`, ''))).toBe("Add a test that fails when `references.push(...phpUseReferences(node, index, scope, i…` is not called."); + }); + + it('speaks for the surest mutant and counts the rest', () => { + expect(methodAdvice([])).toBe(''); + expect(methodAdvice([mutant('boundary', '>', '>=', 6)])).toBe('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); + expect(methodAdvice([mutant('boundary', '>', '>=', 6), mutant('number', '100', '101', 6)])).toBe('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results. 1 more edit survives.'); + expect(methodAdvice([mutant('return', 'x', 'null', 9), mutant('number', '1', '0', 9), mutant('string', "'a'", "''", 10)])).toBe('Add a test that asserts on what line 9 returns. 2 more edits survive.'); + }); +}); diff --git a/test/coverage-html.test.js b/test/coverage-html.test.js index c374332..20270a9 100644 --- a/test/coverage-html.test.js +++ b/test/coverage-html.test.js @@ -138,14 +138,21 @@ describe('coverage HTML report', () => { const html = renderCoverageHtml(sampleReport()); it('puts notes under their lines', () => { - // The mutant's note names the problem, the edit, and shows the line both ways; no percentage when perch is sure (82% is sure enough). - expect(under(html, 'src/cart.py', 6)).toContain('Survived mutant With `>=` instead of `>`, none of the 2 tests reaching it fails.

    '); - expect(under(html, 'src/cart.py', 6)).toContain('
    - if percent > 100:\n+ if percent >= 100:
    '); - expect(under(html, 'src/cart.py', 6)).toContain('Still passes: test_discount_10 and test_discount_20.'); - expect(under(html, 'src/cart.py', 6)).not.toContain('% sure'); - // The tests that reach the method are listed once, in its bar above the code, not again in the gap's note. - expect(under(html, 'src/cart.py', 6)).not.toContain('Reached by'); + // The line's note holds its survived mutants, folded until the mark in the gutter is pressed: each one's edit in words and the + // line both ways; no percentage when perch is sure (82% is sure enough). + const note = under(html, 'src/cart.py', 6); + expect(note).toContain(''); + expect(note).toContain('
    1 survived mutant on line 6
    '); + expect(note).toContain('

    With `>=` instead of `>`, none of the 2 tests reaching it fails.

    '); + expect(note).toContain('
    - if percent > 100:\n+ if percent >= 100:
    '); + expect(note).not.toContain('% sure'); + // The tests that reach the method are listed once, in its bar above the code, not again under every mutant: a survived mutant + // is one none of them kills, so a list of which still pass would be the same list. + expect(note).not.toContain('Still passes'); + expect(note).not.toContain('Reached by'); expect(under(html, 'src/cart.py', 2)).toContain('test_discount_10'); + // The bar above the method says what test to add, from its surest survived mutant. + expect(under(html, 'src/cart.py', 2)).toContain('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); expect(under(html, 'src/cart.py', 5)).not.toContain('Survived mutant'); expect(under(html, 'src/cart.py', 7)).not.toContain('Survived mutant'); // A method no test reaches is said so above its code, and is not a problem: the call graph alone lists nothing. @@ -157,12 +164,19 @@ describe('coverage HTML report', () => { expect(under(html, 'tests/test_cart.py', 3)).not.toContain('class="note'); expect(lineClasses(html, 'tests/test_cart.py', 3)).toEqual(['l']); expect(lineClasses(html, 'tests/test_cart.py', 8)).toEqual(['l', 'weak']); - // A test file opens on its problems, each linking to its line. - const file = html.slice(html.indexOf('
    1'); - expect(problems).toContain('tests/test_cart.py:8'); - expect(file.indexOf('file-problems')).toBeLessThan(file.indexOf('class="code"')); + // A source file opens on the methods to add a test to, then its code; a test file opens on its code. + const cart = html.slice(html.indexOf('
    1'); + expect(methods).toContain('apply_discount'); + expect(cart.indexOf('file-methods')).toBeLessThan(cart.indexOf('class="code"')); + expect(cart).not.toContain('file-problems'); + const tests = html.slice(html.indexOf('
    ↓ Next survived1 line'); + expect(legend).toContain(' open every note'); }); it('names each problem on a test', () => { @@ -230,7 +244,7 @@ describe('coverage HTML report', () => { expect(score).toContain('3 of 6 mutants killed, 50% on covered code'); expect(card(html, 'No coverage')).toContain('0 of 6'); expect(card(html, 'Survived')).toContain('1 of 6'); - const cartRow = viewOf(html, 'sources').match(/.*?<\/tr>/s)[0]; + const cartRow = viewOf(html, 'sources').match(/]*>.*?<\/tr>/s)[0]; expect(cartRow).toContain('3/6'); // The tests tile is how many tests could go. const tests = card(html, 'Redundant tests'); @@ -255,15 +269,17 @@ describe('coverage HTML report', () => { const summary = viewOf(html, 'summary'); expect(summary).not.toContain('class="lede"'); // Two lists and no per-file tables: those are the Source files and Tests tabs. - expect([...summary.matchAll(/

    ([^<]+?) match[1])).toEqual(['Survived mutants', 'Test problems']); - // Survived mutants, one row per method. A method no test reaches is not in the list: the call graph alone is not a problem. - const survivedList = summary.slice(summary.indexOf('

    Survived mutants '), summary.indexOf('

    Test problems ')); - expect(survivedList).toContain('Survived mutant apply_discount'); - expect(survivedList).not.toContain('>total<'); - // The mutant in words; the code is one click away, at the line. - expect(survivedList).toContain('With `>=` instead of `>`, none of the 2 tests reaching it fails.'); - expect(survivedList).not.toContain('if percent'); - expect(survivedList).toContain('src/cart.py:6'); + expect([...summary.matchAll(/

    ([^<]+?) match[1])).toEqual(['Where to add tests', 'Test problems']); + // Methods to add a test to, one row each with the test to add. A method no test reaches is not in the list: the call graph + // alone is not a problem. + const methodList = summary.slice(summary.indexOf('

    Where to add tests '), summary.indexOf('

    Test problems ')); + expect(methodList).toContain('apply_discountsrc/cart.py:3'); + expect(methodList).toContain('12Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); + expect(methodList).toContain(''); + expect(methodList).not.toContain('>total<'); + // The mutants fold under the row, each in words with a link to its line; the code is there. + expect(methodList).toContain(`
    • line 6 With \`>=\` instead of \`>\`, none of the 2 tests reaching it fails.
    `); + expect(methodList).not.toContain('if percent'); const testProblems = summary.slice(summary.indexOf('

    Test problems ')); expect(testProblems).toContain('Duplicate test test_discount_20'); // The first review is short lists: each problem once, not every list again. @@ -276,7 +292,7 @@ describe('coverage HTML report', () => { expect(discount).toContain('3 of 6 mutants killed'); expect(lineClasses(html, 'src/cart.py', 4)).toEqual(['l', 'part']); expect(lineClasses(html, 'src/cart.py', 6)).toEqual(['l', 'part', 'gapline']); - expect(under(html, 'src/cart.py', 6)).toContain('Survived mutant'); + expect(under(html, 'src/cart.py', 6)).toContain('1 survived mutant'); // No test reaches total. expect(under(html, 'src/cart.py', 13)).toContain('No test reaches it'); expect(lineClasses(html, 'src/cart.py', 15)).toEqual(['l', 'none']); @@ -291,6 +307,10 @@ describe('coverage HTML report', () => { // No second list of what is new: the problem is where it belongs, marked and first. expect(summary).not.toContain('New since'); expect(summary).toContain('newDuplicate test test_discount_20'); + // A method whose survived mutant is new is marked too. + const fresh = sampleReport(); + fresh.diff.findings.new = [edge]; + expect(viewOf(renderCoverageHtml(fresh), 'summary')).toContain('newapply_discount'); expect(summary.split('test_discount_20')).toHaveLength(2); // Changes opens on them too. const changes = viewOf(html, 'changes'); @@ -305,8 +325,8 @@ describe('coverage HTML report', () => { const summary = viewOf(page, 'summary'); expect(summary).not.toContain('new-badge'); // The lists hold only the changed code's problems. - const lists = summary.slice(summary.indexOf('

    Survived mutants ')); - expect(lists).toContain('Survived mutant apply_discount'); + const lists = summary.slice(summary.indexOf('

    Where to add tests ')); + expect(lists).toContain('apply_discount'); expect(summary).not.toContain('test_discount_20'); // All problems marks which rows are in changed code, and can show those alone. const problems = viewOf(page, 'problems'); @@ -332,13 +352,13 @@ describe('coverage HTML report', () => { const buttons = '
    '; expect(under(html, 'tests/test_cart.py', 8)).toContain(`
    `); expect(under(html, 'tests/test_cart.py', 8)).toContain(buttons); - expect(under(html, 'src/cart.py', 6)).toContain(`
    `); + expect(under(html, 'src/cart.py', 6)).toContain(`
    `); expect(under(html, 'src/cart.py', 6)).toContain(buttons); expect(html).not.toContain('class="dismiss"'); - // Each list of problems has one button for all of it: the file's Problems box, Risk, and All problems. + // Each list of problems has one button for all of it: the file's methods, the summary's, and All problems. const file = html.slice(html.indexOf('
    Copy prompt'); - expect(viewOf(html, 'summary')).toContain('

    Survived mutants 1

    '); + expect(file.slice(file.indexOf('file-methods'), file.indexOf('class="legend"'))).toContain(''); + expect(viewOf(html, 'summary')).toContain('

    Where to add tests 1

    '); expect(viewOf(html, 'problems')).toContain('data-copy-all'); // No app links: the prompt is copied, to paste into whichever agent you use. expect(html).not.toMatch(/claude-cli:|codex:\/\/|cursor:\/\//); @@ -378,12 +398,43 @@ describe('coverage HTML report', () => { it('breaks each test file down by why tests are not worth keeping, and each source file by survived mutants', () => { const heads = view => [...viewOf(html, view).matchAll(/]*>([^<]+) match[1]); expect(heads('tests')).toEqual(['File', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']); - expect(heads('sources')).toEqual(['File', 'Mutation score', 'Survived', 'No coverage']); + expect(heads('sources')).toEqual(['Mutation score', 'Killed', 'Survived', 'No coverage']); // test_cart.py: one duplicate, test_discount_20. const row = viewOf(html, 'tests').match(/.*?<\/tr>/s)[0]; expect(row).toContain('100'); - // cart.py has the one survived mutant, and no mutant without coverage. - expect(viewOf(html, 'sources').match(/.*?<\/tr>/s)[0]).toContain('10'); + // cart.py: three mutants killed, one survived, none without coverage. + expect(viewOf(html, 'sources').match(/]*>.*?<\/tr>/s)[0]).toContain('310'); + }); + + it('lays the source files out as a tree, worst first, and lists only the test files with something to fix', () => { + const sources = viewOf(html, 'sources'); + // One directory, src, with its files' numbers summed: 3 of 6 killed, 1 survived; the worse file first. + expect(sources).toContain('src/ 2'); + expect(sources.match(/]*>.*?<\/tr>/s)[0]).toContain('310'); + const order = [...sources.matchAll(/ [match[1], match[2]]); + expect(order).toEqual([['src', ''], ['src/cart.py', 'src'], ['src/util.py', 'src']]); + expect(sources).toContain('cart.py'); + expect(sources).toContain(''); + expect(sources).not.toContain('data-closed'); + // A chain of directories with nothing but the next in them is one row. + const deep = sampleReport(); + deep.files[1].path = 'lib/shop/util/money.py'; + deep.methods[2].path = deep.files[1].path; + expect(viewOf(renderCoverageHtml(deep), 'sources')).toContain('>lib/shop/util/ 1'); + // Past two hundred files, every directory under the top starts folded. + const many = sampleReport(); + many.files = [...many.files, ...Array.from({ length: 200 }, (_, at) => ({ ...many.files[1], path: `src/more/file${at}.py`, methods: [] }))]; + const big = viewOf(renderCoverageHtml(many), 'sources'); + expect(big).toContain(''); + expect(big).toContain(''); + expect(tests).toContain('1 test file has nothing to fix.'); + expect(viewOf(html, 'tests')).not.toContain('nothing to fix'); }); it('fetches nothing from anywhere', () => { @@ -422,15 +473,15 @@ describe('coverage HTML report', () => { expect(renderCoverageSite(sampleReport()).index.join('')).toBe(renderCoverageHtml(sampleReport())); }); - it('names the nearest fifty tests that reach a method and counts the rest', () => { + it('names the eight nearest tests that reach a method, the ones it asked, and counts the rest', () => { const report = sampleReport(); const method = report.methods.find(item => item.id === 'src/cart.py::apply_discount'); const reaching = Array.from({ length: 53 }, (_, at) => ({ ...report.tests[0], id: `tests/test_cart.py::t${at}`, name: `t${at}` })); report.tests.push(...reaching); method.tests = reaching.map((test, at) => ({ id: test.id, depth: at < 3 ? 3 : 1 })); const bar = renderCoverageHtml(report).match(/
      (.*?)<\/ul>/s)[1]; - expect(bar.match(/
    • /g)).toHaveLength(50); + expect(bar.match(/
    • /g)).toHaveLength(8); expect(bar).not.toContain('>t0<'); - expect(bar).toContain('
    • 3 more tests
      • '); + expect(bar).toContain('
    • 45 more tests
      • '); }); }); diff --git a/test/coverage-report.test.js b/test/coverage-report.test.js index c16c059..978dd1e 100644 --- a/test/coverage-report.test.js +++ b/test/coverage-report.test.js @@ -19,7 +19,15 @@ const report = { { path: 'tests/test_cart.py', kind: 'test', language: 'python', lines: [], methods: [], tests: ['t1', 't2', 't3', 't4', 't5'], totals: totalsOf({ tests: 5, useful: 3, redundant: 1, weak: 1, infra: 0 }) }, ], - methods: [], + // The methods the survived mutants are on, with the mutant each finding names, as buildReport writes them. + methods: [ + { id: 'm1', path: 'src/cart.py', name: 'apply_discount', line: 12, end_line: 20, tests: [{ id: 't1', depth: 1 }, { id: 't2', depth: 1 }], useful: ['t1'], covered: true, killed: 5, findings: ['d4e5f6a7'], + mutants: [{ id: '17:7:>>>=', kind: 'boundary', line: 17, column: 7, from: '>', to: '>=', original: '', mutated: '' }, { id: '50:9:&&>||', kind: 'logic', line: 50, column: 9, from: '&&', to: '||', original: '', mutated: '' }] }, + { id: 'm4', path: 'src/tax.py', name: 'rate_for', line: 5, end_line: 12, tests: [{ id: 't1', depth: 2 }], useful: ['t1'], covered: true, killed: 1, findings: ['c0ffee11'], + mutants: [{ id: '9:7:<><=', kind: 'boundary', line: 9, column: 7, from: '<', to: '<=', original: '', mutated: '' }, { id: '10:4:x', kind: 'return', line: 10, column: 4, from: 'rate', to: 'None', original: '', mutated: '' }] }, + { id: 'm5', path: 'src/tax.py', name: 'round_tax', line: 28, end_line: 32, tests: [{ id: 't1', depth: 3 }], useful: ['t1'], covered: true, killed: 0, findings: ['b7d31e22'], + mutants: [{ id: '30:11:*>/', kind: 'arithmetic', line: 30, column: 11, from: '*', to: '/', original: '', mutated: '' }] }, + ], tests: [ { id: 't1', path: 'tests/test_cart.py' }, { id: 't2', path: 'tests/test_cart.py' }, @@ -66,8 +74,8 @@ describe('coverage report', () => { const text = formatCoverage(report, { min: 0.5, ...plain }); const source = text.split('\n\n')[0].split('\n'); expect(source[0]).toMatch(/^Source files\s+Mutation score\s+Survived\s+No coverage$/); - // Files in path order, whatever order the report had them in. - expect(source.slice(1).map(line => line.split(/\s{2,}/)[0])).toEqual(['src/cart.py', 'src/tax.py', 'All source']); + // Worst first: tax.py at 33% before cart.py at 58%, whatever order the report had them in. + expect(source.slice(1).map(line => line.split(/\s{2,}/)[0])).toEqual(['src/tax.py', 'src/cart.py', 'All source']); // Of cart.py's 12 mutants, 7 are killed, and one survivor is over the floor. expect(rowOf(text, 'src/cart.py').split(/\s{2,}/)).toEqual(['src/cart.py', '58% (7 of 12)', '1', '0']); // tax.py's two listed survivors: one over the floor, one whose method could not be asked. @@ -81,11 +89,48 @@ describe('coverage report', () => { expect(rowOf(text, 'Test files').split(/\s{2,}/)).toEqual(['Test files', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']); expect(rowOf(text, 'tests/test_cart.py ').split(/\s{2,}/)).toEqual(['tests/test_cart.py', '60% (3 of 5)', '1', '1', '0']); expect(rowOf(text, 'All tests').split(/\s{2,}/)).toEqual(['All tests', '60% (3 of 5)', '1', '1', '0']); + // A test file with nothing to fix is not a row; it is counted under the table, and alone it is the whole table. + const fine = { path: 'tests/test_tax.py', kind: 'test', language: 'python', lines: [], methods: [], tests: ['t6'], totals: totalsOf({ tests: 1, useful: 1 }) }; + const withFine = formatCoverage({ ...report, files: [...report.files, fine] }, { min: 0.5, ...plain }); + expect(withFine).not.toContain('tests/test_tax.py'); + expect(withFine.split('\n\n')[1].split('\n').at(-1)).toBe('1 test file has nothing to fix.'); + const allFine = formatCoverage({ ...report, files: [report.files[0], report.files[1], fine] }, { min: 0.5, ...plain }); + expect(allFine.split('\n\n')[1]).toBe('All 1 test file has nothing to fix.'); }); - it('groups problems above the floor by file', () => { + it('ranks the methods to add a test to, and says what to add', () => { const text = formatCoverage(report, { min: 0.5, ...plain }); - const blocks = text.split('\n\n').slice(2); + const block = text.split('\n\n')[2].split('\n'); + expect(block[0]).toBe('Where to add tests'); + expect(block[1]).toMatch(/^ {2}Method\s+Where\s+Killed\s+Survived\s+Tests$/); + // One survivor each, so the method fewer tests reach comes first, then by file and line: rate_for, round_tax, apply_discount. + // Each is a line of numbers and, under it, the test to add, whole. + expect(block.slice(2)).toEqual([ + ' rate_for src/tax.py:5 1 of 2 1 1', + ' Add a test at the boundary of `<` on line 9, where `<` and `<=` give different results.', + ' round_tax src/tax.py:28 0 of 1 1 1', + ' Add a test that asserts on the arithmetic at line 30, where `*` can become `/`.', + ' apply_discount src/cart.py:12 5 of 2 1 2', + ' Add a test at the boundary of `>` on line 17, where `>` and `>=` give different results.', + ]); + // A method with several survivors names the surest and counts the rest; a narrow terminal wraps the sentence rather than cut it. + const two = { ...report, findings: [...report.findings, { ...report.findings[3], id: 'f0f0f0f0', mutant: '50:9:&&>||', line: 50, probability: 0.6 }] }; + const narrow = formatCoverage(two, { min: 0.5, width: 60, color: false }).split('\n\n')[2].split('\n'); + expect(narrow.slice(2, 5)).toEqual([ + ' apply_discount src/cart.py:12 5 of 2 2 2', + ' Add a test at the boundary of `>` on line 17, where `>`', + ' and `>=` give different results. 1 more edit survives.', + ]); + expect(narrow[5]).toMatch(/^ {2}rate_for/); + expect(narrow.every(line => line.length <= 60)).toBe(true); + }); + + it('lists test problems by file, and every survived mutant with its id under --all', () => { + const text = formatCoverage(report, { min: 0.5, ...plain }); + // Without --all, the survived mutants are the methods table; the test problems are listed by file after it. + expect(text.split('\n\n').slice(3).map(block => block.split('\n')[0])).toEqual(['tests/test_cart.py']); + expect(text).not.toContain('d4e5f6a7'); + const blocks = formatCoverage(report, { min: 0.5, all: true, ...plain }).split('\n\n').slice(3); expect(blocks.map(block => block.split('\n')[0])).toEqual(['src/cart.py', 'src/tax.py', 'tests/test_cart.py']); expect(blocks[0].split('\n')[1]).toMatch(/^ {2}ID\s+Line\s+Problem\s+Confidence\s+Test or method\s+Note$/); expect(blocks[0]).toContain('d4e5f6a7'); @@ -98,7 +143,7 @@ describe('coverage report', () => { 'Decides happy_path of apply_discount, like test_discount_10 at line 12.']); expect(listedFindings(report, { min: 0.5 }).map(finding => finding.id)).toEqual(['a91c2e0f', 'b7d31e22', 'c0ffee11', 'd4e5f6a7']); // --min 0 lists the one under the floor as well. - expect(formatCoverage(report, { min: 0, ...plain })).toContain('e1e2e3e4'); + expect(formatCoverage(report, { min: 0, all: true, ...plain })).toContain('e1e2e3e4'); }); it('cuts long notes to the terminal width', () => { @@ -125,9 +170,11 @@ describe('coverage report', () => { it('ends with one count line, the way a scan does', () => { // Three problems over the floor, and one on a method that could not be asked, which no floor can judge. - expect(coverageCount(report, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 4 problems, 1 could not be asked (--json)'); + // Survived mutants are on the screen as methods; their ids, which perch close takes, want --all. + expect(coverageCount(report, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 4 problems, --all for every mutant, 1 could not be asked (--json)'); const many = { ...report, failed: [], findings: Array.from({ length: 12 }, (_, at) => ({ ...report.findings[1], id: `f${at}`, unit: `m${at}` })) }; - expect(coverageCount(many, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, 10 shown, --all for the rest'); + expect(coverageCount(many, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, 10 of 12 methods shown, --all for the rest'); + expect(coverageCount({ ...report, failed: [], findings: [report.findings[0]] }, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem'); expect(coverageCount(many, { min: 0.5, all: true })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems'); const broken = { ...many, scope: { frameworks: [{ name: 'Jest', config: 'jest.config.js', error: 'no jest' }], left_out: 0 } }; expect(coverageCount(broken, { min: 0.5, all: true })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, jest.config.js did not load (--verbose)'); @@ -158,8 +205,11 @@ describe('coverage report with --since', () => { it("prints the changed code's problems", () => { const text = formatCoverage(onBranch, { min: 0.5, ...plain }); - const [problems, ...rest] = text.split('\n\n'); + const [methods, ...rest] = text.split('\n\n'); expect(rest).toEqual([]); + expect(methods.split('\n')[0]).toBe('Where to add tests'); + expect(methods.split('\n').slice(2).filter((line, at) => at % 2 === 0).map(line => line.trim().split(/\s{2,}/)[0])).toEqual(['apply_discount']); + const problems = formatCoverage(onBranch, { min: 0.5, all: true, ...plain }).split('\n\n')[1]; expect(problems.split('\n')[0]).toBe('src/cart.py'); expect(problems).toContain('d4e5f6a7'); // Under the floor, and elsewhere in the repository: neither is listed. @@ -167,7 +217,7 @@ describe('coverage report with --since', () => { expect(text).not.toContain('a91c2e0f'); expect(listedFindings(onBranch, { min: 0.5 }).map(finding => finding.id)).toEqual(['d4e5f6a7']); // The count line says how many problems are in changed code, and counts the rest apart. - expect(coverageCount(onBranch, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem in changed code, 3 elsewhere'); + expect(coverageCount(onBranch, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem in changed code, 3 elsewhere, --all for every mutant'); }); it('says when nothing changed', () => {