diff --git a/docs/coverage.md b/docs/coverage.md index efeadd3..5da62a9 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -28,23 +28,26 @@ Point it at a repository: ```console $ perch coverage Source files Mutation score Survived No coverage -src/cart.ts 71% (10 of 14) 4 0 src/checkout.ts 60% (3 of 5) 2 0 src/inventory.ts 63% (5 of 8) 2 0 +src/cart.ts 71% (10 of 14) 4 0 All source 67% (18 of 27) 8 0 Test files Quality Duplicates Checks nothing Live services test/cart.test.ts 43% (3 of 7) 4 0 1 -test/checkout.test.ts 100% (1 of 1) 0 0 0 test/inventory.test.ts 75% (3 of 4) 0 1 0 All tests 58% (7 of 12) 4 1 1 - -src/cart.ts - ID Line Problem Confidence Test or method Note - 0399f832 12 survived 87% applyDiscount With `101` instead of `100`, none of the … - 1be27338 12 survived 87% applyDiscount With `>` instead of `>=`, none of the 5 t… - 20fb814a 12 survived 87% applyDiscount With `1` instead of `0`, none of the 5 te… - 883b4b21 12 survived 84% applyDiscount With `false` as the condition, none of th… +1 test file has nothing to fix. + +Where to add tests + Method Where Killed Survived Tests + applyDiscount src/cart.ts:11 6 of 10 4 5 + Add a test that asserts on the value `100` at line 12. 3 more edits survive. + placeOrder src/checkout.ts:5 3 of 5 2 1 + Add a test in which the condition at line 6 is true, and assert on what follows. 1 more edit + survives. + canFulfil src/inventory.ts:4 5 of 8 2 4 + Add a test that asserts on the value `0` at line 6. 1 more edit survives. test/cart.test.ts ID Line Problem Confidence Test or method Note @@ -57,7 +60,7 @@ test/cart.test.ts test/inventory.test.ts ID Line Problem Confidence Test or method Note 98a1a024 23 mocked - canFulfil > returns … Mocks every method it calls: canFul… -shop at commit 0ade512: 4 methods, 12 tests, 14 problems, 10 shown, --all for the rest +shop at commit 0ade512: 4 methods, 12 tests, 14 problems, --all for every mutant Report: .perch/coverage/index.html 28 requests 0 tokens in $0.0000 ``` @@ -65,21 +68,31 @@ Report: .perch/coverage/index.html The mutation score is the share of mutants killed by at least one test. A test kills a mutant when Perch puts its chance of failing against it at 70% or more. No coverage counts the mutants in methods no test reaches through the call -graph. They count against the score. Perch lists each survived mutant with the -edit it made and the number of tests that miss it. The confidence is how sure -Perch is that no test kills the mutant and that the mutant changes what a caller -sees. A test that checks nothing kills no mutant in the code it reaches. A -duplicate kills exactly the mutants an earlier test kills. A test that mocks -what it tests calls only methods it has replaced with its own mocks, so it -checks the mocks. That is a fact of the call graph, so it is listed with `-` -for confidence. So is a problem on a test or method the model couldn't be asked -about. Use `--all` to see every row. - -Perch also writes an HTML report. It shows each line of source with its problems -under it. For a survived mutant, it shows the original and the mutated version -of the line, and the tests that still pass. The report is one file, unless your -repository has more than 8 MB of source. Then each file gets its own page in -the `files/` directory next to `index.html`. +graph. They count against the score. Source files are listed worst first, and +test files only when something in them needs fixing. + +Under the tables, Perch lists the methods to add a test to, most survived +mutants first. Each has a line of numbers and, under it, the test to add, from +its surest survived mutant: the edit a new test has to tell apart, and the line +it is on. `--all` lists every method, and every survived mutant with the id +that `perch close` takes. Test problems follow, by file. The confidence is how +sure Perch is. A test that checks nothing kills no mutant in the code it +reaches. A duplicate kills exactly the mutants an earlier test kills. A test +that mocks what it tests calls only methods it has replaced with its own mocks, +so it checks the mocks. That is a fact of the call graph, so it is listed with +`-` for confidence. So is a problem on a test or method the model couldn't be +asked about. + +Perch also writes an HTML report. It opens on the methods to add a test to, +each with its score, how many tests reach it, and the test to add; the mutants +fold out under the row. Source files are a tree of directories, worst first, +with each score coloured against 60% and 80% as Stryker colours its own. A +file's page shows its code with a mark in the gutter on each line where a +mutant survived; press the mark to see each edit, the line both ways, and a +Copy prompt button, or step through the survivors with the Next survived +button. The report is one file, unless your repository has more than 8 MB of +source. Then each file gets its own page in the `files/` directory next to +`index.html`. ## How it differs from a coverage report @@ -200,17 +213,15 @@ changed on the branch: ```console $ perch coverage --since main -src/cart.ts - ID Line Problem Confidence Test or method Note - 20fb814a 12 survived 86% applyDiscount With `1` instead of `0`, none of the 5 te… - 1be27338 12 survived 85% applyDiscount With `>` instead of `>=`, none of the 5 t… - 0399f832 12 survived 84% applyDiscount With `101` instead of `100`, none of the … - 883b4b21 12 survived 81% applyDiscount With `false` as the condition, none of th… +Where to add tests + Method Where Killed Survived Tests + applyDiscount src/cart.ts:11 6 of 10 4 5 + Add a test that asserts on what line 12 returns. 3 more edits survive. test/cart.test.ts ID Line Problem Confidence Test or method Note 427fce7a 22 redundant 76% applyDiscount > tak… Kills the same mutants as applyDis… -shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere +shop at commit 6fe98ce: 4 methods, 12 tests, 5 problems in changed code, 9 elsewhere, --all for every mutant Report: .perch/coverage/index.html 28 requests 0 tokens in $0.0000 ``` diff --git a/src/coverage-advice.js b/src/coverage-advice.js new file mode 100644 index 0000000..edac7bc --- /dev/null +++ b/src/coverage-advice.js @@ -0,0 +1,40 @@ +/** + * What test to add for a survived mutant, in one sentence, from the kind of edit it is. The sentence is fixed wording around the + * report's own facts: the line, and the text the edit changed. It says what a test has to tell apart, not how to write it. + */ + +/** A piece of code in a sentence, cut to one line of it. */ +const edit = text => { + const one = String(text ?? '').trim().replace(/\s+/g, ' '); + return `\`${one.length > 60 ? `${one.slice(0, 57)}…` : one}\``; +}; + +/** The other value of a forced condition: a test is missing for the case the mutant removed. */ +const otherBranch = to => (/^(true|True)$/.test(String(to).trim()) ? 'false' : 'true'); + +export function adviceFor(mutant) { + const line = mutant.line; + switch (mutant.kind) { + case 'body': return 'Add a test that fails when the method does nothing.'; + case 'removal': return `Add a test that fails when ${edit(mutant.from)} is not called.`; + case 'block': return `Add a test that fails when the block at line ${line} is skipped.`; + case 'condition': return `Add a test in which the condition at line ${line} is ${otherBranch(mutant.to)}, and assert on what follows.`; + case 'boundary': return `Add a test at the boundary of ${edit(mutant.from)} on line ${line}, where ${edit(mutant.from)} and ${edit(mutant.to)} give different results.`; + case 'logic': return `Add a test in which only one side of ${edit(mutant.from)} on line ${line} holds.`; + case 'string': return `Add a test that asserts on the text ${edit(mutant.from)} at line ${line}.`; + case 'number': return `Add a test that asserts on the value ${edit(mutant.from)} at line ${line}.`; + case 'return': return `Add a test that asserts on what line ${line} returns.`; + case 'arithmetic': case 'update': return `Add a test that asserts on the arithmetic at line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)}.`; + default: return `Add a test that asserts on the result of line ${line}, where ${edit(mutant.from)} can become ${edit(mutant.to)} unnoticed.`; + } +} + +/** + * The test to add for a method, from its surest survived mutant, and how many more edits survive beside it. `mutants` are the + * method's listed survived mutants, surest first. + */ +export function methodAdvice(mutants) { + if (!mutants.length) return ''; + const more = mutants.length - 1; + return `${adviceFor(mutants[0])}${more ? ` ${more} more ${more === 1 ? 'edit survives' : 'edits survive'}.` : ''}`; +} diff --git a/src/coverage-html.js b/src/coverage-html.js index e9856b9..859c7ef 100644 --- a/src/coverage-html.js +++ b/src/coverage-html.js @@ -5,6 +5,7 @@ */ import { createHash } from 'node:crypto'; import { WORDMARK, WORDMARK_LIGHT } from './brand.js'; +import { methodAdvice } from './coverage-advice.js'; /** Source text and names come from the repository being read, so every one of them is escaped before it becomes markup. */ const ESCAPES = { '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }; @@ -27,10 +28,10 @@ const when = stamp => String(stamp ?? '').replace('T', ' ').replace(/:\d\d(\.\d+ const counted = (count, one, many = `${one}s`) => `${escape(count)} ${count === 1 ? one : many}`; /** - * The colour a percentage is drawn in. The bands are the ones Istanbul draws, so a page that looks like a coverage report reads - * like one too. They colour a number the report gave; they do not grade anything the number does not already say. + * The colour a percentage is drawn in: red under 60, amber to 80, green from there, the thresholds Stryker ships with, so a score + * here reads the same as one in its report. They colour a number the report gave; they do not grade anything it does not say. */ -const band = value => (value === null || value === undefined ? 'none' : value < 0.5 ? 'low' : value < 0.8 ? 'mid' : 'high'); +const band = value => (value === null || value === undefined ? 'none' : value < 0.6 ? 'low' : value < 0.8 ? 'mid' : 'high'); /** * A link to a line of a file: a section of this page, or with `page`, the page of its own that file was written to. The path @@ -224,44 +225,92 @@ const copyAll = ' ` + + `${isNew ? 'new' : ''}${escape(method.name)}${escape(method.path)}:${escape(method.line)}` + + `${cell(score, { count: `${escape(method.killed)}/${escape(method.mutants.length)}` })}${escape(survived.length)}${escape(method.tests.length)}` + + `${escape(advice)}
` + + ``; +} + /** - * A first review, in two short lists: the methods with a survived mutant, riskiest first, and the tests that could go or that - * reach outside the process. Each is ranked by what it costs to leave, and shows a few rows - * with a link to every one of them under All problems. The rankings are counted from the report; nothing is added to it. With - * --since the same lists hold only the problems in code the branch changed. + * A first review, in two lists: the methods to add a test to, most survived mutants first, and the tests that could go or that + * reach outside the process. Each row says what to do; the mutants behind a method are one click down. With --since the same + * lists hold only the problems in code the branch changed. */ function actionsView(report, index) { const TOP = 8; - const risk = id => index.methods.get(id)?.risk ?? 0; const onBranch = report.branch ? new Set(report.branch.findings) : null; const scoped = onBranch ? report.findings.filter(finding => onBranch.has(finding.id)) : report.findings; const changedOnly = onBranch ? '&changed=1' : ''; const all = (kinds, count) => `See all ${escape(count)} under All problems →`; - const table = (heads, rows) => `
${heads}${rows.join('')}
`; - const section = (title, lede, body, more) => (body - ? `

${title}

${lede ? `

${lede}

` : ''}
${body}${more ? `
${more}
` : ''}
` : ''); + const table = (heads, rows, classes = '') => `
${heads}${rows.join('')}
`; + const section = (title, body, more) => (body + ? `

${title}

${body}${more ? `
${more}
` : ''}
` : ''); const of = kinds => scoped.filter(finding => kinds.includes(finding.kind)); // What the changes since the compared run brought in is marked new and put first in its list: it is what the person reading // this is working on now. With --since, the lists are the branch's already, and nothing is marked. const fresh = new Set(report.branch ? [] : (report.diff?.findings?.new ?? []).map(finding => finding.id)); const newFirst = (a, b) => Number(fresh.has(b.id)) - Number(fresh.has(a.id)); - // Survived mutants, one row per method, riskiest first. - const missing = of(['survived']); - const perMethod = new Map(); - for (const finding of missing) if (!perMethod.has(finding.unit)) perMethod.set(finding.unit, finding); - const riskiest = [...perMethod.values()].sort((a, b) => newFirst(a, b) || risk(b.unit) - risk(a.unit) || a.path.localeCompare(b.path)); - const methodRows = riskiest.slice(0, TOP).map(finding => `${actionCells(finding, index, fresh.has(finding.id))}` - + `${escape(Math.round(risk(finding.unit)))}${dismissMark}`); - const methodsSection = section(`Survived mutants ${escape(missing.length)}${copyAll}`, '', - methodRows.length ? table('ProblemWhereDetailsRisk', methodRows) : '', - missing.length > TOP ? all(['survived'], missing.length) : ''); + // Methods to add a test to, every one with a survived mutant, paged by the table. + const methods = methodsToTest(report, index, onBranch); + const hasNew = entry => entry.survived.some(finding => fresh.has(finding.id)); + const methodRowsHtml = [...methods].sort((a, b) => Number(hasNew(b)) - Number(hasNew(a))).map(entry => methodRows(entry, index, { fresh })); + const methodsSection = section(`Where to add tests ${escape(methods.length)}${copyAll}`, + methodRowsHtml.length ? table(methodHeads(), methodRowsHtml, ' methods') : '', ''); // Test problems: duplicates, tests that check nothing, and tests that call a live service, the surest first. const testKinds = ['redundant', ...WEAK, 'infra']; const testProblems = of(testKinds).sort((a, b) => newFirst(a, b) || (b.probability ?? 2) - (a.probability ?? 2) || a.path.localeCompare(b.path)); const testRows = testProblems.slice(0, TOP).map(finding => `${actionCells(finding, index, fresh.has(finding.id))}${dismissMark}`); - const testsSection = section(`Test problems ${escape(testProblems.length)}${copyAll}`, '', + const testsSection = section(`Test problems ${escape(testProblems.length)}${copyAll}`, testRows.length ? table('ProblemWhereDetails', testRows) : '', testProblems.length > TOP ? all(testKinds, testProblems.length) : ''); @@ -284,32 +333,68 @@ const th = (name, numeric = true, bars = false, title = '') => `${escape(file.path)}` - + `${scoreCell(totals)}` - + `${survived}` - + `${escape(totals.no_coverage)}`; - }).join(''); - const totals = report.totals; - const foot = `All source files` - + `${scoreCell(totals)}${escape(totals.survived)}${escape(totals.no_coverage)}`; - const heads = th('File', false) - + th('Mutation score', true, true, 'Mutants some test is predicted to fail against, of every mutant in the file, the ones no test reaches included.') + const survivedIn = path => index.kinds(path).survived ?? 0; + const root = { name: '', path: '', dirs: new Map(), files: [] }; + for (const file of files) { + const parts = file.path.split('/'); + let node = root; + for (const part of parts.slice(0, -1)) { + if (!node.dirs.has(part)) node.dirs.set(part, { name: part, path: node.path ? `${node.path}/${part}` : part, dirs: new Map(), files: [] }); + node = node.dirs.get(part); + } + node.files.push(file); + } + const sum = node => { + const totals = { mutants: 0, killed: 0, no_coverage: 0, survived: 0, files: 0 }; + const add = other => { for (const key of Object.keys(totals)) totals[key] += other[key] ?? 0; }; + for (const file of node.files) add({ mutants: file.totals.mutants, killed: file.totals.killed, no_coverage: file.totals.no_coverage, survived: survivedIn(file.path), files: 1 }); + for (const dir of node.dirs.values()) add(sum(dir)); + node.totals = totals; + return totals; + }; + sum(root); + const joined = node => { + while (!node.files.length && node.dirs.size === 1) { const [only] = node.dirs.values(); node = { ...only, name: `${node.name}/${only.name}` }; } + return node; + }; + const score = totals => scoreValue(totals); + const worst = (a, b) => (score(a.totals) ?? 1) - (score(b.totals) ?? 1) || (b.totals.survived - a.totals.survived) || (b.totals.no_coverage - a.totals.no_coverage) || a.path.localeCompare(b.path); + // A large tree opens on its top level alone; a small one is open all the way down. + const closed = files.length > 200; + const counts = totals => `${scoreCell(totals)}${escape(totals.killed)}` + + `${escape(totals.survived)}${escape(totals.no_coverage)}`; + const rows = []; + const walk = (node, depth, parent) => { + const entries = [...[...node.dirs.values()].map(joined), ...node.files.map(file => ({ file, path: file.path, totals: { ...file.totals, survived: survivedIn(file.path) } }))].sort(worst); + for (const entry of entries) { + if (entry.file) { + rows.push(`${escape(entry.path.split('/').at(-1))}${counts(entry.totals)}`); + } else { + rows.push(` 0 ? ' data-closed' : ''}>${escape(entry.name)}/ ${escape(entry.totals.files)}${counts(entry.totals)}`); + walk(entry, depth + 1, entry.path); + } + } + }; + walk(root, 0, ''); + const totals = { ...report.totals, survived: report.totals.survived }; + const foot = `All source files ${escape(files.length)}${counts(totals)}`; + const heads = 'File' + + th('Mutation score', true, true, 'Mutants some test is predicted to fail against, of every mutant, the ones no test reaches included.') + + th('Killed', true, false, 'Mutants some test is predicted to fail against.') + th('Survived', true, false, 'Mutants no test kills, listed under All problems.') + th('No coverage', true, false, 'Mutants in methods no test reaches. Nothing was asked about them; they count against the score.'); return view('sources', 'Source files', '', `
` - + `
${heads}${rows}${foot}
`); + + `
${heads}${rows.join('')}${foot}
`); } -/** The test files: how many tests each has, how many are worth keeping, and why the rest are not. */ +/** The test files with something to fix, most first. The rest are counted under the table and shown on request. */ function testTable(report, index) { const files = report.files.filter(file => file.kind === 'test'); const notKept = file => file.totals.tests - file.totals.useful; @@ -317,16 +402,19 @@ function testTable(report, index) { const totals = file.totals; const kept = ratio(totals.useful, totals.tests); const count = value => `${escape(value ?? 0)}`; - return `${escape(file.path)}` + const fine = !notKept(file) && !totals.infra; + return `${escape(file.path)}` + `${cell(kept, { count: `${escape(totals.useful)}/${escape(totals.tests)}` })}` + count(totals.redundant) + count(totals.weak) + count(totals.infra) + ''; - }).join(''); + }); + const fine = files.filter(file => !notKept(file) && !file.totals.infra).length; const heads = th('File', false) + th('Quality', true, true, 'Tests worth keeping, of all the file\'s tests: ones that check something and repeat no other test.') + th('Duplicates', true, false, 'Tests that check the same thing with the same code as an earlier test.') + th('Checks nothing', true, false, 'Tests that would pass whatever the code they call does.') + th('Live services', true, false, 'Tests that call a real network service or database with nothing mocked.'); + const foot = fine ? `
${counted(fine, 'test file has', 'test files have')} nothing to fix.
` : ''; return view('tests', 'Tests', '', - `
${heads}${rows}
`); + `
${heads}${rows.join('')}
${foot}
`); } /** Every problem, filterable by kind, each linking to the line it is about. */ @@ -393,9 +481,9 @@ function methodState(method) { /** A method's survived mutants that are listed problems: one under the floor has nothing to act on. */ const listedMutants = (method, index) => method.findings.map(id => index.findings.get(id)).filter(finding => finding?.kind === 'survived'); -/** Tint, branch markers and the survived mutants for each line of a file, innermost method last so a nested function wins its lines. */ +/** Tint, branch markers and how many survived mutants each line of a file has, innermost method last so a nested function wins its lines. */ function lineMarks(file, index) { - const marks = file.lines.map(() => ({ state: '', branch: false, gap: false, starts: [] })); + const marks = file.lines.map(() => ({ state: '', branch: false, gaps: 0, starts: [] })); const at = line => marks[line - 1]; const units = file.kind === 'test' ? file.tests.map(id => index.tests.get(id)) : file.methods.map(id => index.methods.get(id)); const spans = units.filter(Boolean).sort((a, b) => (b.end_line - b.line) - (a.end_line - a.line)); @@ -408,13 +496,13 @@ function lineMarks(file, index) { at(unit.line)?.starts.push(unit); if (file.kind === 'test') continue; for (const line of unit.branches) if (at(line)) at(line).branch = true; - for (const finding of listedMutants(unit, index)) if (at(finding.line)) at(finding.line).gap = true; + for (const finding of listedMutants(unit, index)) if (at(finding.line)) at(finding.line).gaps += 1; } return marks; } -/** The tests nearest a method, the ones calling it directly first. Past `REACH_SHOWN`, a count stands for the rest. */ -const REACH_SHOWN = 50; +/** The tests nearest a method, the ones calling it directly first: the eight perch asked about its mutants. A count stands for the rest. */ +const REACH_SHOWN = 8; function reachedBy(method, index) { const useful = new Set(method.useful); const nearest = [...method.tests].sort((a, b) => a.depth - b.depth); @@ -424,7 +512,10 @@ function reachedBy(method, index) { + (rest > 0 ? `
  • ${escape(rest)} more ${rest === 1 ? 'test' : 'tests'}
    • ` : ''); } -/** The line above a method: its name, how many of its mutants its tests kill, and how many tests reach it, with the tests one click away. */ +/** + * The line above a method: its name, how many of its mutants its tests kill, the test to add when some survived, and how many + * tests reach it, with the tests one click away. + */ function methodBar(method, index) { const ran = !method.covered && method.mutants.length ? `${escape(method.mutants.length)} ${method.mutants.length === 1 ? 'mutant' : 'mutants'}, no coverage` : method.mutants.length ? `${escape(method.killed)} of ${escape(method.mutants.length)} mutants killed` : ''; @@ -432,7 +523,9 @@ function methodBar(method, index) { const tests = count ? `
    ${count} ${count === 1 ? 'test reaches' : 'tests reach'} it${method.useful.length !== count ? `, ${method.useful.length} worth keeping` : ''}
    ` : 'No test reaches it'; - return `
    ${escape(method.name)}${ran ? `${ran}` : ''}${tests}
    `; + const survived = survivedOf(method, index); + const advice = survived.length ? `${escape(methodAdvice(survived.map(finding => mutantOf(method, finding)).filter(Boolean)))}` : ''; + return `
    ${escape(method.name)}${ran ? `${ran}` : ''}${tests}${advice}
    `; } /** "a", "a and b", "a, b and c". */ @@ -440,14 +533,10 @@ const listed = items => (items.length < 2 ? items.join('') : `${items.slice(0, - /** How sure perch is, said only when it is not very: a verdict at 95% needs no number, one at 63% does. */ const unsure = probability => (typeof probability === 'number' && probability < 0.8 ? ` ${escape(percent(probability))} sure` : ''); -/** The note under a survived mutant: the line as written and as changed, and which tests miss it, with its buttons. */ -function mutantNote(method, finding, index) { - const mutant = method.mutants.find(item => item.id === finding.mutant); - const diff = mutant ? `
    - ${escape(mutant.original.trim())}\n+ ${escape(mutant.mutated.trim())}
    ` : ''; - const missed = mutant ? mutant.asked.filter(id => !mutant.killed_by.includes(id)).map(id => unitLink(id, index)) : []; - return `

    ${escape(problemName('survived'))} ` - + `${escape(finding.note)}${unsure(finding.probability)}

    ${acts()}${dismissMark}
    ${diff}` - + (missed.length ? `` : '') + '
    '; +/** The note under a line with survived mutants: each one's edit, how sure perch is, the line both ways, and its buttons. Folded until the gutter mark is pressed. */ +function gapNote(method, findings, index, line) { + const head = `
    ${counted(findings.length, 'survived mutant')} on line ${escape(line)}
    `; + return `
    ${head}${findings.map(finding => mutantBlock(method, finding, index)).join('')}
    `; } /** One problem in a note under the code: its name and the fact, with Copy prompt and the close mark on the right. */ @@ -488,7 +577,9 @@ function notesOf(file, index, report) { } } else { for (const method of file.methods.map(id => index.methods.get(id)).filter(Boolean)) { - for (const finding of listedMutants(method, index)) { placed.add(finding.id); add(finding.line, mutantNote(method, finding, index)); } + const byLine = new Map(); + for (const finding of survivedOf(method, index)) { placed.add(finding.id); if (!byLine.has(finding.line)) byLine.set(finding.line, []); byLine.get(finding.line).push(finding); } + for (const [line, findings] of byLine) add(line, gapNote(method, findings, index, line)); } } for (const finding of report.findings) { @@ -514,32 +605,42 @@ function fileStats(file) { + stat('survived', escape(totals.survived), 'Mutants no test kills, listed below.'); } -/** A file's problems at the top of its page, each linking to its line, so they are read without scrolling the source. */ -function fileProblems(file, report, index) { - const own = [...index.findingsIn(file.path)].sort((a, b) => a.line - b.line || a.kind.localeCompare(b.kind)); - return own.length ? `

    Problems ${escape(own.length)}${copyAll}

    ${actionTable(own, index)}
    ` : ''; +/** The methods of a file to add a test to, the few with most survivors, in the shape the summary lists them. */ +function fileMethods(file, report, index) { + const own = new Set(file.methods); + const entries = methodsToTest(report, index).filter(entry => own.has(entry.method.id)); + if (!entries.length) return ''; + const SHOWN = 5; + const rest = entries.length - SHOWN; + return `

    Where to add tests ${escape(entries.length)}${copyAll}

    ` + + `
    ${methodHeads()}${entries.slice(0, SHOWN).map(entry => methodRows(entry, index)).join('')}
    ` + + (rest > 0 ? `
    ${escape(rest)} more ${rest === 1 ? 'method' : 'methods'} in this file under Summary →
    ` : '') + '
    '; } -/** One file's page: its numbers, a legend, and every line of it with tint, markers and the notes that belong under each. */ +/** One file's page: its numbers, the methods to test, a toolbar with the legend and a way through the survivors, then every line with tint, marks and notes. */ function fileView(file, index, report) { const marks = lineMarks(file, index), notes = notesOf(file, index, report); const methods = file.methods.map(id => index.methods.get(id)).filter(Boolean); + const gaps = marks.filter(mark => mark.gaps).length; const legend = file.kind === 'test' ? 'a test with a problem' : (methods.length ? 'every mutant killeda mutant survivedno coverage' : '') - + '◆branch▲survived mutant'; + + '◆branch▲survived mutants, press for the edits'; + const tools = file.kind === 'source' && gaps + ? `${escape(gaps)} ${gaps === 1 ? 'line' : 'lines'}` + + '' + : ''; const rows = file.lines.map((text, at) => { const line = at + 1, mark = marks[at]; const bars = file.kind === 'source' ? mark.starts.map(method => methodBar(method, index)).join('') : ''; - const gutter = mark.gap ? '▲' : mark.branch ? '◆' : ''; - const classes = ['l', mark.state, mark.gap ? 'gapline' : ''].filter(Boolean).join(' '); + const gutter = mark.gaps ? `` : mark.branch ? '◆' : ''; + const classes = ['l', mark.state, mark.gaps ? 'gapline' : ''].filter(Boolean).join(' '); const under = (notes.get(line) ?? []).join(''); return `${bars}
    ${line}${gutter}${escape(text) || ' '}
    ${under}`; }).join(''); return ``; } @@ -842,7 +943,6 @@ ul.reach{margin:6px 0 2px;padding-left:18px;font-size:13px} .verdict{color:var(--body)}.verdict b{margin-right:6px}.verdict.bad b{color:var(--low-ink)}.note.test .verdict.bad b{color:var(--mid)} .unsure{margin-left:6px;color:var(--faint);font-size:12.5px} .facts{margin:6px 0 0;padding-left:18px;color:var(--muted);font-size:13px} -body.nonotes .note{display:none} [data-dismissed]{display:none!important} /* The toast that says what a button did. */ @@ -850,8 +950,37 @@ body.nonotes .note{display:none} .toast code{color:var(--accent);word-break:break-all}.toast button{margin-left:8px} .toast .prompt-text{display:block;width:min(680px,calc(100vw - 64px));margin-top:8px;padding:8px 10px;background:var(--bg);color:var(--ink);border:1px solid var(--line);border-radius:8px;font:12px/1.5 var(--mono)} +/* Methods to add a test to: name and place, numbers, the test to add, and the mutants folded under the row. */ +table.methods td.act{white-space:normal;min-width:220px}table.methods .act{display:grid;grid-template-columns:28px 1fr;align-items:start} +table.methods .act .fold{grid-row:1/3}.act .name{font-family:var(--mono);font-size:13px;color:var(--ink);font-weight:600;overflow-wrap:anywhere} +.act .path{display:block;grid-column:2;font-size:12px;color:var(--muted)}table.methods .act .new-badge{grid-column:2} +.fold{display:inline-flex;align-items:center;justify-content:center;width:22px;height:22px;margin-right:6px;border:0;border-radius:6px;background:transparent;color:var(--faint);font-size:17px;line-height:1;cursor:pointer;transition:transform .15s;vertical-align:-3px} +.fold:hover{background:var(--raised);color:var(--ink)} +tr[data-open] .fold,tr.dir:not([data-closed]) .fold{transform:rotate(90deg)} +table.methods td.do{white-space:normal;min-width:260px;max-width:440px}.advice{color:var(--body)} +tr.detail td{padding:0 18px 12px 46px;background:var(--raised);white-space:normal} +ul.edits{margin:0;padding:10px 0 0 18px;color:var(--body);font-size:13.5px}ul.edits li{margin:2px 0}ul.edits .at{font-family:var(--mono);font-size:12.5px;margin-right:4px} +.mutant{padding:8px 12px;border:1px solid var(--line);border-left:2px solid var(--low);border-radius:8px;background:var(--panel)} +.mutant .at{font-family:var(--mono);font-size:12.5px;margin-right:6px} +.note .mutant{margin-top:8px} + +/* The source tree: directories fold, their numbers are their files' summed, each level sits in from its parent. */ +table.tree td.path{padding-left:calc(18px + var(--depth,0)*20px)} +tr.dir td{color:var(--ink);font-weight:600}tr.dir td.path{color:var(--ink);cursor:pointer}tr.dir .muted{font-weight:500;font-size:12px} + +/* The file page's toolbar: the legend, a way through the lines with survivors, and whether every note is open. */ +.legend .pager-gaps{display:inline-flex;align-items:center;gap:6px;margin-left:auto} +.legend .pager-gaps button,.legend .toggle{padding:3px 10px;border:1px solid var(--line);border-radius:7px;background:transparent;color:var(--muted);font:500 12px/1.4 var(--sans);cursor:pointer} +.legend .pager-gaps button:hover{color:var(--ink);border-color:color-mix(in srgb,var(--accent) 45%,var(--line))} +.legend .toggle{margin-left:0} +.file-methods{width:calc(100% - 48px);max-width:1132px;margin:0 auto 16px} +button.gapmk{padding:0 4px;min-width:18px;border:0;border-radius:5px;background:color-mix(in srgb,var(--low) 16%,transparent);color:var(--low);font:600 10px/1.6 var(--mono);cursor:pointer} +button.gapmk:hover{background:color-mix(in srgb,var(--low) 30%,transparent)} +.l.gapline.noted{background:color-mix(in srgb,var(--low) 10%,transparent)} +.note.closed{display:none}body.allnotes .note.closed{display:block} + @media (max-width:640px){.wrap{padding:0 16px}.top{position:static}.where{margin-left:0}.file-head,.stats,.legend{padding-left:16px;padding-right:16px} -.code,.file-problems{width:calc(100% - 32px)}.note,.mbar{margin-left:16px;margin-right:16px}} +.code,.file-methods{width:calc(100% - 32px)}.note,.mbar{margin-left:16px;margin-right:16px}} `; /** @@ -879,7 +1008,7 @@ const SCRIPT = String.raw` if (target) { onSummary = false; var row = line ? target.querySelector('[data-n="' + line + '"]') : null; - if (row) { row.classList.add('flash'); row.scrollIntoView({ block: 'center' }); } else window.scrollTo(0, 0); + if (row) { row.classList.add('flash'); openNote(target, line, true); row.scrollIntoView({ block: 'center' }); } else window.scrollTo(0, 0); return; } // One view at a time: the one the hash names, or the first. A link into All problems may name the kinds to show. @@ -907,13 +1036,18 @@ const SCRIPT = String.raw` // page of them. A table no longer than a page has no pager, and still filters through render. var SIZES = [25, 50, 100, 0]; function render(state) { - var rows = Array.prototype.slice.call(state.table.tBodies[0].rows); + var every = Array.prototype.slice.call(state.table.tBodies[0].rows); + var rows = every.filter(function (row) { return !row.hasAttribute('data-detail-for'); }); var kept = rows.filter(function (row) { return !row.hasAttribute('data-out'); }); var size = state.size || Math.max(kept.length, 1), pages = Math.max(1, Math.ceil(kept.length / size)); state.page = Math.min(Math.max(1, state.page), pages); var from = (state.page - 1) * size, to = Math.min(from + size, kept.length); rows.forEach(function (row) { row.hidden = true; }); kept.slice(from, to).forEach(function (row) { row.hidden = false; }); + every.forEach(function (row) { + var owner = row.hasAttribute('data-detail-for') && state.table.querySelector('tr.method[data-method="' + row.getAttribute('data-detail-for').replace(/"/g, '\\"') + '"]'); + if (owner) row.hidden = owner.hidden || !owner.hasAttribute('data-open'); + }); if (rows.length <= SIZES[0]) { state.bar.hidden = true; return; } state.bar.hidden = false; var options = SIZES.map(function (value) { return ''; }).join(''); @@ -941,7 +1075,79 @@ const SCRIPT = String.raw` render(state); } function refresh(table) { if (table.pager) { table.pager.page = 1; render(table.pager); } } - document.querySelectorAll('#summary table.grid').forEach(paged); + document.querySelectorAll('#summary table.grid:not(.tree)').forEach(paged); + + // A method's mutants fold under its row. + document.addEventListener('click', function (event) { + var opener = event.target.closest('[data-open-row]'); + if (!opener) return; + var row = opener.closest('tr'); + if (row.hasAttribute('data-open')) row.removeAttribute('data-open'); else row.setAttribute('data-open', ''); + var table = row.closest('table'); + Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (other) { + if (other.getAttribute('data-detail-for') === row.getAttribute('data-method')) other.hidden = !row.hasAttribute('data-open'); + }); + }); + + // The source tree: a directory folds its rows, and a filter shows the files that match with the directories above them. + var treeFilter = ''; + function applyTree(table) { + var rows = Array.prototype.slice.call(table.tBodies[0].rows), byPath = {}; + rows.forEach(function (row) { byPath[row.getAttribute('data-path')] = row; }); + var matches = function (row) { + if (!treeFilter) return true; + if (row.classList.contains('dir')) return rows.some(function (other) { return !other.classList.contains('dir') && other.getAttribute('data-path').indexOf(row.getAttribute('data-path') + '/') === 0 && other.getAttribute('data-path').toLowerCase().indexOf(treeFilter) >= 0; }); + return row.getAttribute('data-path').toLowerCase().indexOf(treeFilter) >= 0; + }; + rows.forEach(function (row) { + var folded = false, parent = row.getAttribute('data-parent'); + while (!treeFilter && parent) { var up = byPath[parent]; if (!up) break; if (up.hasAttribute('data-closed')) folded = true; parent = up.getAttribute('data-parent'); } + row.hidden = folded || !matches(row); + }); + } + document.querySelectorAll('table.tree').forEach(applyTree); + document.addEventListener('click', function (event) { + var toggle = event.target.closest('[data-toggle-dir]') || (event.target.closest('tr.dir td.path') && !event.target.closest('a') ? event.target.closest('tr.dir td.path') : null); + if (!toggle) return; + var row = toggle.closest('tr'); + if (row.hasAttribute('data-closed')) row.removeAttribute('data-closed'); else row.setAttribute('data-closed', ''); + applyTree(row.closest('table')); + }); + + // The test files with nothing to fix, on request. + document.addEventListener('click', function (event) { + var show = event.target.closest('[data-show-fine]'); + if (!show) return; + var table = show.closest('.panel').querySelector('table'); + Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (row) { if (row.hasAttribute('data-fine')) row.removeAttribute('data-out'); }); + refresh(table); + show.parentNode.hidden = true; + }); + + // Notes under the code open from the mark in the gutter, one at a time, or all at once from the toolbar. + function openNote(section, line, open) { + var note = null, row = section.querySelector('[data-n="' + line + '"]'); + Array.prototype.slice.call(section.querySelectorAll('.note[data-line]')).forEach(function (candidate) { if (candidate.getAttribute('data-line') === String(line)) note = candidate; }); + if (!note) return; + var closed = open === undefined ? !note.classList.contains('closed') : !open; + note.classList.toggle('closed', closed); + if (row) row.classList.toggle('noted', !closed); + } + document.addEventListener('click', function (event) { + var mark = event.target.closest('[data-toggle-note]'); + if (mark) { openNote(mark.closest('section.file'), mark.getAttribute('data-toggle-note')); return; } + var step = event.target.closest('[data-gap]'); + if (!step) return; + var section = step.closest('section.file'); + var lines = Array.prototype.slice.call(section.querySelectorAll('.l.gapline')).map(function (row) { return Number(row.getAttribute('data-n')); }); + if (!lines.length) return; + var middle = window.scrollY + window.innerHeight / 2, current = null; + Array.prototype.slice.call(section.querySelectorAll('.l.gapline')).forEach(function (row) { if (row.getBoundingClientRect().top + window.scrollY <= middle) current = Number(row.getAttribute('data-n')); }); + var at = current === null ? -1 : lines.indexOf(current); + var next = lines[Math.min(lines.length - 1, Math.max(0, at + Number(step.getAttribute('data-gap'))))]; + if (current !== null && at + Number(step.getAttribute('data-gap')) < 0) next = lines[0]; + location.hash = 'file=' + encodeURIComponent(section.getAttribute('data-path')) + '&line=' + next; + }); document.querySelectorAll('table.sortable').forEach(function (table) { var heads = Array.prototype.slice.call(table.querySelectorAll('thead th')); @@ -964,7 +1170,9 @@ const SCRIPT = String.raw` document.querySelectorAll('.filter-files').forEach(function (filter) { filter.addEventListener('input', function () { var text = filter.value.toLowerCase(); + treeFilter = text; document.querySelectorAll('table.files').forEach(function (table) { + if (table.classList.contains('tree')) { applyTree(table); return; } Array.prototype.slice.call(table.tBodies[0].rows).forEach(function (row) { if (text && row.getAttribute('data-path').toLowerCase().indexOf(text) < 0) row.setAttribute('data-out', ''); else row.removeAttribute('data-out'); }); @@ -1013,9 +1221,11 @@ const SCRIPT = String.raw` } function shownIn(scope) { var ids = []; + var take = function (id) { if (fixData.steps[id] && ids.indexOf(id) < 0) ids.push(id); }; + scope.querySelectorAll('tr.method[data-ids]').forEach(function (row) { if (row.getClientRects().length) row.getAttribute('data-ids').split(',').forEach(take); }); scope.querySelectorAll('[data-finding]').forEach(function (element) { - var id = element.getAttribute('data-finding'); - if (element.getClientRects().length && fixData.steps[id] && ids.indexOf(id) < 0) ids.push(id); + if (element.getClientRects().length && !element.closest('tr.detail')) take(element.getAttribute('data-finding')); + else if (element.closest('tr.detail') && element.closest('tr.detail').previousElementSibling && element.closest('tr.detail').previousElementSibling.getClientRects().length) take(element.getAttribute('data-finding')); }); return ids; } @@ -1046,6 +1256,12 @@ const SCRIPT = String.raw` document.addEventListener('click', function (event) { var owner = event.target.closest('[data-finding]'); if (event.target.closest('[data-copy]') && owner) { copyPrompt(promptFor([owner.getAttribute('data-finding')])); return; } + var byIds = event.target.closest('[data-copy-ids]'); + if (byIds) { + var ids = byIds.closest('tr').getAttribute('data-ids').split(',').filter(function (id) { return fixData.steps[id] && dismissedIds().indexOf(id) < 0; }); + if (ids.length) copyPrompt(promptFor(ids), ids.length); else say('Every mutant here is dismissed.'); + return; + } var all = event.target.closest('[data-copy-all]'); if (all) { var ids = shownIn(all.closest('.panel') || document); @@ -1105,7 +1321,7 @@ const SCRIPT = String.raw` document.querySelectorAll('.notes-toggle').forEach(function (box) { box.addEventListener('change', function () { - document.body.classList.toggle('nonotes', !box.checked); + document.body.classList.toggle('allnotes', box.checked); document.querySelectorAll('.notes-toggle').forEach(function (other) { other.checked = box.checked; }); }); }); diff --git a/src/coverage-report.js b/src/coverage-report.js index e4ab3a3..08d60de 100644 --- a/src/coverage-report.js +++ b/src/coverage-report.js @@ -5,6 +5,7 @@ */ import { BELIEVED } from './questions.js'; import { bold, COLOR, dim, keepEnd, keepStart, percent, relative, sureness, table, TOP, WIDTH } from './report.js'; +import { methodAdvice } from './coverage-advice.js'; /** Every kind of problem a coverage report can list, and so everything `--filter kind=` accepts. */ export const COVERAGE_KINDS = ['survived', 'redundant', 'checks_nothing', 'mocked', 'infra']; @@ -83,8 +84,10 @@ const painted = (lines, color, total = false) => lines.map((line, index) => (ind function sourceTable(report, onList, { width, color }) { + const score = totals => (totals.mutants ? totals.killed / totals.mutants : 1); + // Worst first: the lowest score, then the most mutants no test reaches, then the path, so the files to work on head the table. const files = report.files.filter(file => file.kind === 'source' && file.totals.methods) - .sort((a, b) => a.path.localeCompare(b.path)); + .sort((a, b) => score(a.totals) - score(b.totals) || b.totals.no_coverage - a.totals.no_coverage || a.path.localeCompare(b.path)); if (!files.length) return ''; // The mutation score is the mutants some test is predicted to kill, of every mutant, the ones no test reaches included, as // Stryker and PIT count it. Survived is counted from the problems listed under the table, at the same floor, so the two agree; @@ -99,18 +102,62 @@ function sourceTable(report, onList, { width, color }) { return painted(fitted(header, rows, ['left', 'right', 'right', 'right'], { width }), color, true).join('\n'); } +/** The test files with something to fix, most first; the ones with nothing to fix are one line. */ function testTable(report, { width, color }) { - const files = report.files.filter(file => file.kind === 'test' && file.totals.tests) - .sort((a, b) => a.path.localeCompare(b.path)); - if (!files.length) return ''; + const all = report.files.filter(file => file.kind === 'test' && file.totals.tests); + if (!all.length) return ''; + const toFix = totals => totals.tests - totals.useful + totals.infra; + const files = all.filter(file => toFix(file.totals)).sort((a, b) => toFix(b.totals) - toFix(a.totals) || a.path.localeCompare(b.path)); + const fine = all.length - files.length; + const nothing = fine ? `${fine === all.length ? 'All ' : ''}${plural(fine, 'test file has', 'test files have')} nothing to fix.` : ''; + if (!files.length) return nothing; // Quality is the tests worth keeping, of all of them; the next three are why the rest are not, or what they touch. const header = ['Test files', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']; const row = (name, totals) => [name, `${percent(totals.useful / totals.tests)} (${totals.useful} of ${totals.tests})`, totals.redundant, totals.weak, totals.infra]; const rows = files.map(file => row(relative(file.path), file.totals)); rows.push(row('All tests', report.totals)); - return painted(fitted(header, rows, ['left', 'right', 'right', 'right', 'right'], { width }), color, true).join('\n'); + return [painted(fitted(header, rows, ['left', 'right', 'right', 'right', 'right'], { width }), color, true).join('\n'), nothing].filter(Boolean).join('\n'); } +/** Text wrapped at a width, each line indented, for a sentence that has to be read whole. */ +function wrapped(text, width, indent) { + const lines = []; + let line = ''; + for (const word of text.split(' ')) { + if (line && line.length + 1 + word.length > width) { lines.push(line); line = word; } else line = line ? `${line} ${word}` : word; + } + if (line) lines.push(line); + return lines.map(part => `${indent}${part}`); +} + +/** + * The methods to add a test to, most survived mutants first: a line of numbers for each, the way a scan lists a method, and under + * it the test to add, from its surest survived mutant, written out whole. A tie goes to the method fewer tests reach, then to the + * file and line. The survived mutants themselves, with the ids `perch close` takes, are listed by file with --all. + */ +function methodBlock(report, findings, { width, color, all }) { + const survived = findings.filter(finding => finding.kind === 'survived'); + if (!survived.length) return ''; + const methods = new Map(report.methods.map(method => [method.id, method])); + const entries = [...Map.groupBy(survived, finding => finding.unit)].map(([id, list]) => ({ method: methods.get(id), + list: [...list].sort((a, b) => (b.probability ?? 0) - (a.probability ?? 0) || (a.line ?? 0) - (b.line ?? 0)) })) + .filter(entry => entry.method) + .sort((a, b) => b.list.length - a.list.length || a.method.tests.length - b.method.tests.length || a.method.path.localeCompare(b.method.path) || a.method.line - b.method.line); + const shown = all ? entries : entries.slice(0, TOP); + const HEAD = ['Method', 'Where', 'Killed', 'Survived', 'Tests']; + const rows = shown.map(({ method, list }) => ({ cells: [method.name, `${relative(method.path)}:${method.line}`, `${method.killed} of ${method.mutants.length}`, String(list.length), String(method.tests.length)], + advice: methodAdvice(list.map(finding => method.mutants.find(mutant => mutant.id === finding.mutant)).filter(Boolean)) })); + const widths = HEAD.map((name, column) => Math.max(name.length, ...rows.map(row => row.cells[column].length))); + const pad = (cells, aligns) => cells.map((cell, column) => (aligns[column] === 'right' ? cell.padStart(widths[column]) : cell.padEnd(widths[column]))).join(' ').trimEnd(); + const aligns = ['left', 'left', 'right', 'right', 'right']; + const lines = [bold('Where to add tests', color), dim(` ${pad(HEAD, aligns)}`, color)]; + for (const row of rows) lines.push(` ${pad(row.cells, aligns)}`, ...wrapped(row.advice, Math.max(40, width - 4), ' ')); + return lines.join('\n'); +} + +/** How many methods have a survived mutant among the findings. */ +const methodsWithSurvivors = findings => new Set(findings.filter(finding => finding.kind === 'survived').map(finding => finding.unit)).size; + /** * The problems, one block per file and one line per problem, laid out the way `perch scan` lays out its own so the two read as one * tool. The note is the last column and the one that gives way to the terminal, since it is the longest and is a sentence the id @@ -147,15 +194,19 @@ function findingBlocks(findings, { width, color }) { } /** - * What a coverage run prints on stdout: the source files, the test files, then the problems grouped by file. An unasked-for list is - * cut to the top ten; `--all` or a filter is the asking, and gets every one. With --since it is what the branch changed and the - * problems in it instead, and the whole repository is one line on stderr. + * What a coverage run prints on stdout: the source files worst first, the test files with something to fix, the methods to add a + * test to, then the test problems by file. An unasked-for list is cut to the top ten; `--all` or a filter is the asking, and gets + * every method and every survived mutant with its id. With --since it is what the branch changed and the problems in it instead, + * and the whole repository is one line on stderr. */ export function formatCoverage(report, { min = BELIEVED, filters = [], all = false, width = WIDTH(), color = COLOR() } = {}) { const listed = listedFindings(report, { min, filters }); - const shown = all || filters.length ? listed : ranked(listed).slice(0, TOP); - if (report.branch) return shown.length ? findingBlocks(shown, { width, color }) : `No problems in code changed since ${report.branch.ref}.`; - const parts = [sourceTable(report, listedFindings(report, { min }), { width, color }), testTable(report, { width, color }), findingBlocks(shown, { width, color })]; + const every = all || filters.length > 0; + const others = listed.filter(finding => finding.kind !== 'survived'); + const blocks = [methodBlock(report, listed, { width, color, all: every }), + findingBlocks(every ? [...others, ...listed.filter(finding => finding.kind === 'survived')] : ranked(others).slice(0, TOP), { width, color })]; + if (report.branch) return listed.length ? blocks.filter(Boolean).join('\n\n') : `No problems in code changed since ${report.branch.ref}.`; + const parts = [sourceTable(report, listedFindings(report, { min }), { width, color }), testTable(report, { width, color }), ...blocks]; const text = parts.filter(Boolean).join('\n\n'); return text || 'No source files or tests found.'; } @@ -202,7 +253,10 @@ export function coverageCount(report, { min = BELIEVED, filters = [], all = fals const others = listedFindings({ ...report, branch: null }, { min, filters }).length - listed; parts.push(`${plural(listed, 'problem')} in changed code${others ? `, ${others} elsewhere` : ''}`); } else parts.push(plural(listed, 'problem')); - if (!all && !filters.length && listed > TOP) parts.push(`${TOP} shown, --all for the rest`); + // Without --all, the methods past the top ten and every survived mutant's id are off the screen. + const survived = listedFindings(report, { min, filters }), methods = methodsWithSurvivors(survived); + if (!all && !filters.length && methods > TOP) parts.push(`${TOP} of ${plural(methods, 'method')} shown, --all for the rest`); + else if (!all && !filters.length && (methods || survived.length - methods > TOP)) parts.push('--all for every mutant'); const failed = report.failed.length; if (failed) parts.push(`${failed} could not be asked (--json)`); // A config that would not load means every test the parser found was read instead, which changes what is counted. diff --git a/test/coverage-advice.test.js b/test/coverage-advice.test.js new file mode 100644 index 0000000..badc0ff --- /dev/null +++ b/test/coverage-advice.test.js @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { adviceFor, methodAdvice } from '../src/coverage-advice.js'; + +const mutant = (kind, from, to, line = 12) => ({ kind, from, to, line }); + +describe('what test to add for a survived mutant', () => { + it('names the edit a test has to tell apart, by its kind', () => { + expect(adviceFor(mutant('body', '{ save(order); }', '{}'))).toBe('Add a test that fails when the method does nothing.'); + expect(adviceFor(mutant('removal', 'save(order);', ''))).toBe('Add a test that fails when `save(order);` is not called.'); + expect(adviceFor(mutant('block', '{ total += 1; }', '{}'))).toBe('Add a test that fails when the block at line 12 is skipped.'); + expect(adviceFor(mutant('condition', 'percent > 100', 'true'))).toBe('Add a test in which the condition at line 12 is false, and assert on what follows.'); + expect(adviceFor(mutant('condition', 'percent > 100', 'false'))).toBe('Add a test in which the condition at line 12 is true, and assert on what follows.'); + expect(adviceFor(mutant('boundary', '>=', '>'))).toBe('Add a test at the boundary of `>=` on line 12, where `>=` and `>` give different results.'); + expect(adviceFor(mutant('logic', '&&', '||'))).toBe('Add a test in which only one side of `&&` on line 12 holds.'); + expect(adviceFor(mutant('string', "'An item is out of stock'", "''"))).toBe("Add a test that asserts on the text `'An item is out of stock'` at line 12."); + expect(adviceFor(mutant('number', '100', '101'))).toBe('Add a test that asserts on the value `100` at line 12.'); + expect(adviceFor(mutant('return', 'total', '0'))).toBe('Add a test that asserts on what line 12 returns.'); + expect(adviceFor(mutant('arithmetic', '*', '/'))).toBe('Add a test that asserts on the arithmetic at line 12, where `*` can become `/`.'); + expect(adviceFor(mutant('update', '+=', '-='))).toBe('Add a test that asserts on the arithmetic at line 12, where `+=` can become `-=`.'); + expect(adviceFor(mutant('not', '!ready', 'ready'))).toBe('Add a test that asserts on the result of line 12, where `!ready` can become `ready` unnoticed.'); + expect(adviceFor(mutant('boolean', 'true', 'false'))).toBe('Add a test that asserts on the result of line 12, where `true` can become `false` unnoticed.'); + }); + + it('cuts a long call to one line of it', () => { + const call = `references.push(...phpUseReferences(node, index, scope, imports, aliases, 'use', 'function'));`; + expect(adviceFor(mutant('removal', ` ${call}\n`, ''))).toBe("Add a test that fails when `references.push(...phpUseReferences(node, index, scope, i…` is not called."); + }); + + it('speaks for the surest mutant and counts the rest', () => { + expect(methodAdvice([])).toBe(''); + expect(methodAdvice([mutant('boundary', '>', '>=', 6)])).toBe('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); + expect(methodAdvice([mutant('boundary', '>', '>=', 6), mutant('number', '100', '101', 6)])).toBe('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results. 1 more edit survives.'); + expect(methodAdvice([mutant('return', 'x', 'null', 9), mutant('number', '1', '0', 9), mutant('string', "'a'", "''", 10)])).toBe('Add a test that asserts on what line 9 returns. 2 more edits survive.'); + }); +}); diff --git a/test/coverage-html.test.js b/test/coverage-html.test.js index c374332..20270a9 100644 --- a/test/coverage-html.test.js +++ b/test/coverage-html.test.js @@ -138,14 +138,21 @@ describe('coverage HTML report', () => { const html = renderCoverageHtml(sampleReport()); it('puts notes under their lines', () => { - // The mutant's note names the problem, the edit, and shows the line both ways; no percentage when perch is sure (82% is sure enough). - expect(under(html, 'src/cart.py', 6)).toContain('Survived mutant With `>=` instead of `>`, none of the 2 tests reaching it fails.

    '); - expect(under(html, 'src/cart.py', 6)).toContain('
    - if percent > 100:\n+ if percent >= 100:
    '); - expect(under(html, 'src/cart.py', 6)).toContain('Still passes: test_discount_10 and test_discount_20.'); - expect(under(html, 'src/cart.py', 6)).not.toContain('% sure'); - // The tests that reach the method are listed once, in its bar above the code, not again in the gap's note. - expect(under(html, 'src/cart.py', 6)).not.toContain('Reached by'); + // The line's note holds its survived mutants, folded until the mark in the gutter is pressed: each one's edit in words and the + // line both ways; no percentage when perch is sure (82% is sure enough). + const note = under(html, 'src/cart.py', 6); + expect(note).toContain(''); + expect(note).toContain('
    1 survived mutant on line 6
    '); + expect(note).toContain('

    With `>=` instead of `>`, none of the 2 tests reaching it fails.

    '); + expect(note).toContain('
    - if percent > 100:\n+ if percent >= 100:
    '); + expect(note).not.toContain('% sure'); + // The tests that reach the method are listed once, in its bar above the code, not again under every mutant: a survived mutant + // is one none of them kills, so a list of which still pass would be the same list. + expect(note).not.toContain('Still passes'); + expect(note).not.toContain('Reached by'); expect(under(html, 'src/cart.py', 2)).toContain('test_discount_10'); + // The bar above the method says what test to add, from its surest survived mutant. + expect(under(html, 'src/cart.py', 2)).toContain('Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); expect(under(html, 'src/cart.py', 5)).not.toContain('Survived mutant'); expect(under(html, 'src/cart.py', 7)).not.toContain('Survived mutant'); // A method no test reaches is said so above its code, and is not a problem: the call graph alone lists nothing. @@ -157,12 +164,19 @@ describe('coverage HTML report', () => { expect(under(html, 'tests/test_cart.py', 3)).not.toContain('class="note'); expect(lineClasses(html, 'tests/test_cart.py', 3)).toEqual(['l']); expect(lineClasses(html, 'tests/test_cart.py', 8)).toEqual(['l', 'weak']); - // A test file opens on its problems, each linking to its line. - const file = html.slice(html.indexOf('
    1'); - expect(problems).toContain('tests/test_cart.py:8'); - expect(file.indexOf('file-problems')).toBeLessThan(file.indexOf('class="code"')); + // A source file opens on the methods to add a test to, then its code; a test file opens on its code. + const cart = html.slice(html.indexOf('
    1'); + expect(methods).toContain('apply_discount'); + expect(cart.indexOf('file-methods')).toBeLessThan(cart.indexOf('class="code"')); + expect(cart).not.toContain('file-problems'); + const tests = html.slice(html.indexOf('
    ↓ Next survived1 line'); + expect(legend).toContain(' open every note'); }); it('names each problem on a test', () => { @@ -230,7 +244,7 @@ describe('coverage HTML report', () => { expect(score).toContain('3 of 6 mutants killed, 50% on covered code'); expect(card(html, 'No coverage')).toContain('0 of 6'); expect(card(html, 'Survived')).toContain('1 of 6'); - const cartRow = viewOf(html, 'sources').match(/.*?<\/tr>/s)[0]; + const cartRow = viewOf(html, 'sources').match(/]*>.*?<\/tr>/s)[0]; expect(cartRow).toContain('3/6'); // The tests tile is how many tests could go. const tests = card(html, 'Redundant tests'); @@ -255,15 +269,17 @@ describe('coverage HTML report', () => { const summary = viewOf(html, 'summary'); expect(summary).not.toContain('class="lede"'); // Two lists and no per-file tables: those are the Source files and Tests tabs. - expect([...summary.matchAll(/

    ([^<]+?) match[1])).toEqual(['Survived mutants', 'Test problems']); - // Survived mutants, one row per method. A method no test reaches is not in the list: the call graph alone is not a problem. - const survivedList = summary.slice(summary.indexOf('

    Survived mutants '), summary.indexOf('

    Test problems ')); - expect(survivedList).toContain('Survived mutant apply_discount'); - expect(survivedList).not.toContain('>total<'); - // The mutant in words; the code is one click away, at the line. - expect(survivedList).toContain('With `>=` instead of `>`, none of the 2 tests reaching it fails.'); - expect(survivedList).not.toContain('if percent'); - expect(survivedList).toContain('src/cart.py:6'); + expect([...summary.matchAll(/

    ([^<]+?) match[1])).toEqual(['Where to add tests', 'Test problems']); + // Methods to add a test to, one row each with the test to add. A method no test reaches is not in the list: the call graph + // alone is not a problem. + const methodList = summary.slice(summary.indexOf('

    Where to add tests '), summary.indexOf('

    Test problems ')); + expect(methodList).toContain('apply_discountsrc/cart.py:3'); + expect(methodList).toContain('12Add a test at the boundary of `>` on line 6, where `>` and `>=` give different results.'); + expect(methodList).toContain(''); + expect(methodList).not.toContain('>total<'); + // The mutants fold under the row, each in words with a link to its line; the code is there. + expect(methodList).toContain(`
    • line 6 With \`>=\` instead of \`>\`, none of the 2 tests reaching it fails.
    `); + expect(methodList).not.toContain('if percent'); const testProblems = summary.slice(summary.indexOf('

    Test problems ')); expect(testProblems).toContain('Duplicate test test_discount_20'); // The first review is short lists: each problem once, not every list again. @@ -276,7 +292,7 @@ describe('coverage HTML report', () => { expect(discount).toContain('3 of 6 mutants killed'); expect(lineClasses(html, 'src/cart.py', 4)).toEqual(['l', 'part']); expect(lineClasses(html, 'src/cart.py', 6)).toEqual(['l', 'part', 'gapline']); - expect(under(html, 'src/cart.py', 6)).toContain('Survived mutant'); + expect(under(html, 'src/cart.py', 6)).toContain('1 survived mutant'); // No test reaches total. expect(under(html, 'src/cart.py', 13)).toContain('No test reaches it'); expect(lineClasses(html, 'src/cart.py', 15)).toEqual(['l', 'none']); @@ -291,6 +307,10 @@ describe('coverage HTML report', () => { // No second list of what is new: the problem is where it belongs, marked and first. expect(summary).not.toContain('New since'); expect(summary).toContain('newDuplicate test test_discount_20'); + // A method whose survived mutant is new is marked too. + const fresh = sampleReport(); + fresh.diff.findings.new = [edge]; + expect(viewOf(renderCoverageHtml(fresh), 'summary')).toContain('newapply_discount'); expect(summary.split('test_discount_20')).toHaveLength(2); // Changes opens on them too. const changes = viewOf(html, 'changes'); @@ -305,8 +325,8 @@ describe('coverage HTML report', () => { const summary = viewOf(page, 'summary'); expect(summary).not.toContain('new-badge'); // The lists hold only the changed code's problems. - const lists = summary.slice(summary.indexOf('

    Survived mutants ')); - expect(lists).toContain('Survived mutant apply_discount'); + const lists = summary.slice(summary.indexOf('

    Where to add tests ')); + expect(lists).toContain('apply_discount'); expect(summary).not.toContain('test_discount_20'); // All problems marks which rows are in changed code, and can show those alone. const problems = viewOf(page, 'problems'); @@ -332,13 +352,13 @@ describe('coverage HTML report', () => { const buttons = '
    '; expect(under(html, 'tests/test_cart.py', 8)).toContain(`
    `); expect(under(html, 'tests/test_cart.py', 8)).toContain(buttons); - expect(under(html, 'src/cart.py', 6)).toContain(`
    `); + expect(under(html, 'src/cart.py', 6)).toContain(`
    `); expect(under(html, 'src/cart.py', 6)).toContain(buttons); expect(html).not.toContain('class="dismiss"'); - // Each list of problems has one button for all of it: the file's Problems box, Risk, and All problems. + // Each list of problems has one button for all of it: the file's methods, the summary's, and All problems. const file = html.slice(html.indexOf('
    Copy prompt'); - expect(viewOf(html, 'summary')).toContain('

    Survived mutants 1

    '); + expect(file.slice(file.indexOf('file-methods'), file.indexOf('class="legend"'))).toContain(''); + expect(viewOf(html, 'summary')).toContain('

    Where to add tests 1

    '); expect(viewOf(html, 'problems')).toContain('data-copy-all'); // No app links: the prompt is copied, to paste into whichever agent you use. expect(html).not.toMatch(/claude-cli:|codex:\/\/|cursor:\/\//); @@ -378,12 +398,43 @@ describe('coverage HTML report', () => { it('breaks each test file down by why tests are not worth keeping, and each source file by survived mutants', () => { const heads = view => [...viewOf(html, view).matchAll(/]*>([^<]+) match[1]); expect(heads('tests')).toEqual(['File', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']); - expect(heads('sources')).toEqual(['File', 'Mutation score', 'Survived', 'No coverage']); + expect(heads('sources')).toEqual(['Mutation score', 'Killed', 'Survived', 'No coverage']); // test_cart.py: one duplicate, test_discount_20. const row = viewOf(html, 'tests').match(/.*?<\/tr>/s)[0]; expect(row).toContain('100'); - // cart.py has the one survived mutant, and no mutant without coverage. - expect(viewOf(html, 'sources').match(/.*?<\/tr>/s)[0]).toContain('10'); + // cart.py: three mutants killed, one survived, none without coverage. + expect(viewOf(html, 'sources').match(/]*>.*?<\/tr>/s)[0]).toContain('310'); + }); + + it('lays the source files out as a tree, worst first, and lists only the test files with something to fix', () => { + const sources = viewOf(html, 'sources'); + // One directory, src, with its files' numbers summed: 3 of 6 killed, 1 survived; the worse file first. + expect(sources).toContain('src/ 2'); + expect(sources.match(/]*>.*?<\/tr>/s)[0]).toContain('310'); + const order = [...sources.matchAll(/ [match[1], match[2]]); + expect(order).toEqual([['src', ''], ['src/cart.py', 'src'], ['src/util.py', 'src']]); + expect(sources).toContain('cart.py'); + expect(sources).toContain(''); + expect(sources).not.toContain('data-closed'); + // A chain of directories with nothing but the next in them is one row. + const deep = sampleReport(); + deep.files[1].path = 'lib/shop/util/money.py'; + deep.methods[2].path = deep.files[1].path; + expect(viewOf(renderCoverageHtml(deep), 'sources')).toContain('>lib/shop/util/ 1'); + // Past two hundred files, every directory under the top starts folded. + const many = sampleReport(); + many.files = [...many.files, ...Array.from({ length: 200 }, (_, at) => ({ ...many.files[1], path: `src/more/file${at}.py`, methods: [] }))]; + const big = viewOf(renderCoverageHtml(many), 'sources'); + expect(big).toContain(''); + expect(big).toContain(''); + expect(tests).toContain('1 test file has nothing to fix.'); + expect(viewOf(html, 'tests')).not.toContain('nothing to fix'); }); it('fetches nothing from anywhere', () => { @@ -422,15 +473,15 @@ describe('coverage HTML report', () => { expect(renderCoverageSite(sampleReport()).index.join('')).toBe(renderCoverageHtml(sampleReport())); }); - it('names the nearest fifty tests that reach a method and counts the rest', () => { + it('names the eight nearest tests that reach a method, the ones it asked, and counts the rest', () => { const report = sampleReport(); const method = report.methods.find(item => item.id === 'src/cart.py::apply_discount'); const reaching = Array.from({ length: 53 }, (_, at) => ({ ...report.tests[0], id: `tests/test_cart.py::t${at}`, name: `t${at}` })); report.tests.push(...reaching); method.tests = reaching.map((test, at) => ({ id: test.id, depth: at < 3 ? 3 : 1 })); const bar = renderCoverageHtml(report).match(/
      (.*?)<\/ul>/s)[1]; - expect(bar.match(/
    • /g)).toHaveLength(50); + expect(bar.match(/
    • /g)).toHaveLength(8); expect(bar).not.toContain('>t0<'); - expect(bar).toContain('
    • 3 more tests
      • '); + expect(bar).toContain('
    • 45 more tests
      • '); }); }); diff --git a/test/coverage-report.test.js b/test/coverage-report.test.js index c16c059..978dd1e 100644 --- a/test/coverage-report.test.js +++ b/test/coverage-report.test.js @@ -19,7 +19,15 @@ const report = { { path: 'tests/test_cart.py', kind: 'test', language: 'python', lines: [], methods: [], tests: ['t1', 't2', 't3', 't4', 't5'], totals: totalsOf({ tests: 5, useful: 3, redundant: 1, weak: 1, infra: 0 }) }, ], - methods: [], + // The methods the survived mutants are on, with the mutant each finding names, as buildReport writes them. + methods: [ + { id: 'm1', path: 'src/cart.py', name: 'apply_discount', line: 12, end_line: 20, tests: [{ id: 't1', depth: 1 }, { id: 't2', depth: 1 }], useful: ['t1'], covered: true, killed: 5, findings: ['d4e5f6a7'], + mutants: [{ id: '17:7:>>>=', kind: 'boundary', line: 17, column: 7, from: '>', to: '>=', original: '', mutated: '' }, { id: '50:9:&&>||', kind: 'logic', line: 50, column: 9, from: '&&', to: '||', original: '', mutated: '' }] }, + { id: 'm4', path: 'src/tax.py', name: 'rate_for', line: 5, end_line: 12, tests: [{ id: 't1', depth: 2 }], useful: ['t1'], covered: true, killed: 1, findings: ['c0ffee11'], + mutants: [{ id: '9:7:<><=', kind: 'boundary', line: 9, column: 7, from: '<', to: '<=', original: '', mutated: '' }, { id: '10:4:x', kind: 'return', line: 10, column: 4, from: 'rate', to: 'None', original: '', mutated: '' }] }, + { id: 'm5', path: 'src/tax.py', name: 'round_tax', line: 28, end_line: 32, tests: [{ id: 't1', depth: 3 }], useful: ['t1'], covered: true, killed: 0, findings: ['b7d31e22'], + mutants: [{ id: '30:11:*>/', kind: 'arithmetic', line: 30, column: 11, from: '*', to: '/', original: '', mutated: '' }] }, + ], tests: [ { id: 't1', path: 'tests/test_cart.py' }, { id: 't2', path: 'tests/test_cart.py' }, @@ -66,8 +74,8 @@ describe('coverage report', () => { const text = formatCoverage(report, { min: 0.5, ...plain }); const source = text.split('\n\n')[0].split('\n'); expect(source[0]).toMatch(/^Source files\s+Mutation score\s+Survived\s+No coverage$/); - // Files in path order, whatever order the report had them in. - expect(source.slice(1).map(line => line.split(/\s{2,}/)[0])).toEqual(['src/cart.py', 'src/tax.py', 'All source']); + // Worst first: tax.py at 33% before cart.py at 58%, whatever order the report had them in. + expect(source.slice(1).map(line => line.split(/\s{2,}/)[0])).toEqual(['src/tax.py', 'src/cart.py', 'All source']); // Of cart.py's 12 mutants, 7 are killed, and one survivor is over the floor. expect(rowOf(text, 'src/cart.py').split(/\s{2,}/)).toEqual(['src/cart.py', '58% (7 of 12)', '1', '0']); // tax.py's two listed survivors: one over the floor, one whose method could not be asked. @@ -81,11 +89,48 @@ describe('coverage report', () => { expect(rowOf(text, 'Test files').split(/\s{2,}/)).toEqual(['Test files', 'Quality', 'Duplicates', 'Checks nothing', 'Live services']); expect(rowOf(text, 'tests/test_cart.py ').split(/\s{2,}/)).toEqual(['tests/test_cart.py', '60% (3 of 5)', '1', '1', '0']); expect(rowOf(text, 'All tests').split(/\s{2,}/)).toEqual(['All tests', '60% (3 of 5)', '1', '1', '0']); + // A test file with nothing to fix is not a row; it is counted under the table, and alone it is the whole table. + const fine = { path: 'tests/test_tax.py', kind: 'test', language: 'python', lines: [], methods: [], tests: ['t6'], totals: totalsOf({ tests: 1, useful: 1 }) }; + const withFine = formatCoverage({ ...report, files: [...report.files, fine] }, { min: 0.5, ...plain }); + expect(withFine).not.toContain('tests/test_tax.py'); + expect(withFine.split('\n\n')[1].split('\n').at(-1)).toBe('1 test file has nothing to fix.'); + const allFine = formatCoverage({ ...report, files: [report.files[0], report.files[1], fine] }, { min: 0.5, ...plain }); + expect(allFine.split('\n\n')[1]).toBe('All 1 test file has nothing to fix.'); }); - it('groups problems above the floor by file', () => { + it('ranks the methods to add a test to, and says what to add', () => { const text = formatCoverage(report, { min: 0.5, ...plain }); - const blocks = text.split('\n\n').slice(2); + const block = text.split('\n\n')[2].split('\n'); + expect(block[0]).toBe('Where to add tests'); + expect(block[1]).toMatch(/^ {2}Method\s+Where\s+Killed\s+Survived\s+Tests$/); + // One survivor each, so the method fewer tests reach comes first, then by file and line: rate_for, round_tax, apply_discount. + // Each is a line of numbers and, under it, the test to add, whole. + expect(block.slice(2)).toEqual([ + ' rate_for src/tax.py:5 1 of 2 1 1', + ' Add a test at the boundary of `<` on line 9, where `<` and `<=` give different results.', + ' round_tax src/tax.py:28 0 of 1 1 1', + ' Add a test that asserts on the arithmetic at line 30, where `*` can become `/`.', + ' apply_discount src/cart.py:12 5 of 2 1 2', + ' Add a test at the boundary of `>` on line 17, where `>` and `>=` give different results.', + ]); + // A method with several survivors names the surest and counts the rest; a narrow terminal wraps the sentence rather than cut it. + const two = { ...report, findings: [...report.findings, { ...report.findings[3], id: 'f0f0f0f0', mutant: '50:9:&&>||', line: 50, probability: 0.6 }] }; + const narrow = formatCoverage(two, { min: 0.5, width: 60, color: false }).split('\n\n')[2].split('\n'); + expect(narrow.slice(2, 5)).toEqual([ + ' apply_discount src/cart.py:12 5 of 2 2 2', + ' Add a test at the boundary of `>` on line 17, where `>`', + ' and `>=` give different results. 1 more edit survives.', + ]); + expect(narrow[5]).toMatch(/^ {2}rate_for/); + expect(narrow.every(line => line.length <= 60)).toBe(true); + }); + + it('lists test problems by file, and every survived mutant with its id under --all', () => { + const text = formatCoverage(report, { min: 0.5, ...plain }); + // Without --all, the survived mutants are the methods table; the test problems are listed by file after it. + expect(text.split('\n\n').slice(3).map(block => block.split('\n')[0])).toEqual(['tests/test_cart.py']); + expect(text).not.toContain('d4e5f6a7'); + const blocks = formatCoverage(report, { min: 0.5, all: true, ...plain }).split('\n\n').slice(3); expect(blocks.map(block => block.split('\n')[0])).toEqual(['src/cart.py', 'src/tax.py', 'tests/test_cart.py']); expect(blocks[0].split('\n')[1]).toMatch(/^ {2}ID\s+Line\s+Problem\s+Confidence\s+Test or method\s+Note$/); expect(blocks[0]).toContain('d4e5f6a7'); @@ -98,7 +143,7 @@ describe('coverage report', () => { 'Decides happy_path of apply_discount, like test_discount_10 at line 12.']); expect(listedFindings(report, { min: 0.5 }).map(finding => finding.id)).toEqual(['a91c2e0f', 'b7d31e22', 'c0ffee11', 'd4e5f6a7']); // --min 0 lists the one under the floor as well. - expect(formatCoverage(report, { min: 0, ...plain })).toContain('e1e2e3e4'); + expect(formatCoverage(report, { min: 0, all: true, ...plain })).toContain('e1e2e3e4'); }); it('cuts long notes to the terminal width', () => { @@ -125,9 +170,11 @@ describe('coverage report', () => { it('ends with one count line, the way a scan does', () => { // Three problems over the floor, and one on a method that could not be asked, which no floor can judge. - expect(coverageCount(report, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 4 problems, 1 could not be asked (--json)'); + // Survived mutants are on the screen as methods; their ids, which perch close takes, want --all. + expect(coverageCount(report, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 4 problems, --all for every mutant, 1 could not be asked (--json)'); const many = { ...report, failed: [], findings: Array.from({ length: 12 }, (_, at) => ({ ...report.findings[1], id: `f${at}`, unit: `m${at}` })) }; - expect(coverageCount(many, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, 10 shown, --all for the rest'); + expect(coverageCount(many, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, 10 of 12 methods shown, --all for the rest'); + expect(coverageCount({ ...report, failed: [], findings: [report.findings[0]] }, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem'); expect(coverageCount(many, { min: 0.5, all: true })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems'); const broken = { ...many, scope: { frameworks: [{ name: 'Jest', config: 'jest.config.js', error: 'no jest' }], left_out: 0 } }; expect(coverageCount(broken, { min: 0.5, all: true })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 12 problems, jest.config.js did not load (--verbose)'); @@ -158,8 +205,11 @@ describe('coverage report with --since', () => { it("prints the changed code's problems", () => { const text = formatCoverage(onBranch, { min: 0.5, ...plain }); - const [problems, ...rest] = text.split('\n\n'); + const [methods, ...rest] = text.split('\n\n'); expect(rest).toEqual([]); + expect(methods.split('\n')[0]).toBe('Where to add tests'); + expect(methods.split('\n').slice(2).filter((line, at) => at % 2 === 0).map(line => line.trim().split(/\s{2,}/)[0])).toEqual(['apply_discount']); + const problems = formatCoverage(onBranch, { min: 0.5, all: true, ...plain }).split('\n\n')[1]; expect(problems.split('\n')[0]).toBe('src/cart.py'); expect(problems).toContain('d4e5f6a7'); // Under the floor, and elsewhere in the repository: neither is listed. @@ -167,7 +217,7 @@ describe('coverage report with --since', () => { expect(text).not.toContain('a91c2e0f'); expect(listedFindings(onBranch, { min: 0.5 }).map(finding => finding.id)).toEqual(['d4e5f6a7']); // The count line says how many problems are in changed code, and counts the rest apart. - expect(coverageCount(onBranch, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem in changed code, 3 elsewhere'); + expect(coverageCount(onBranch, { min: 0.5 })).toBe('/repo at commit 9f8e7d6: 6 methods, 5 tests, 1 problem in changed code, 3 elsewhere, --all for every mutant'); }); it('says when nothing changed', () => {