Skip to content

inspect - provide information about output files from input files #12809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
3c1b767
inspect - provide information about output files from input files
cscheid May 22, 2025
4f436fb
Merge branch 'main' into enhance/12805
cderv May 23, 2025
d768449
Merge branch 'main' into enhance/12805
cderv May 26, 2025
4eeb863
update to latest uv
cderv May 16, 2025
9249496
Update to python 3.13.3
cderv May 16, 2025
094992f
3.13.2 is the latest one available on Linux
cderv May 16, 2025
219c4ce
update uv.lock
cderv May 19, 2025
8f4a8d4
update to latest
cderv May 26, 2025
cf0001e
tests - add a test for fig-format with matplotlib
cderv May 26, 2025
21895a9
fix set_matplotlib_formats being removed from Ipython
cderv May 26, 2025
437c2a4
Add to changelog
cderv May 26, 2025
2288789
lua,typst - do not crash if float.content is nil
cscheid May 23, 2025
89836e4
timing for new tests [auto PR]
cderv Jun 2, 2025
6b09cd9
test - Add a test for plotly subfigures
cderv May 27, 2025
dcb001e
Adapt isPlotlyLibrary regex for new plotly.py 6+ inclusion
cderv May 27, 2025
052ccf3
Add to changelog
cderv May 27, 2025
ad97953
also handle non-connected notebook mode we use in dashboard
cderv Jun 3, 2025
ea7fd87
unified light and dark brand
gordonwoodhull May 28, 2025
6601cc8
artifacts
gordonwoodhull Jun 3, 2025
43c141b
Update version.txt
github-actions[bot] Jun 4, 2025
aeac7d0
check, knitr - Add .exe to binary on Windows for correct check of exi…
cderv Jun 4, 2025
e886a6f
Update version.txt
github-actions[bot] Jun 5, 2025
e49a48f
Don't use `copyFileSync` in case source is read-only as future writes…
jkrumbiegel Jun 5, 2025
ff248bc
timing for new tests [auto PR]
cderv Jun 9, 2025
9774d89
Schema update - lualatex is the new default engine
cderv Jun 9, 2025
79547e8
Add missing space with shortcode close
rundel Jun 10, 2025
a3eeab1
Missing {} for multi-arg fenced div
rundel Jun 10, 2025
357e025
Update QuartoNotebookRunner to `0.17.3`
MichaelHatherly Jun 2, 2025
076e3de
Add more verbose about python binary check and Jupyter capabilities
cderv Jun 11, 2025
96de09b
Update version.txt
github-actions[bot] Jun 11, 2025
b529f40
fix: upgrade @fluentui/react-icons from 2.0.298 to 2.0.300
snyk-bot Jun 4, 2025
c1630db
ensure replaceAll() is called with function parameter when string is …
cscheid Jun 11, 2025
09c39d3
add regression test and changelog entry
cscheid Jun 11, 2025
d9a83df
avoid issue with \_ on pandoc attributes
cscheid Jun 11, 2025
af40ed7
revert bad fix
cscheid Jun 12, 2025
fa84a13
work around #9224
cscheid Jun 12, 2025
4a37286
`--log-level critical` is supposed to be supported
cderv Jun 12, 2025
e857346
Add `--log-level=warn` as an alias to `--log-level=warning`
cderv Jun 12, 2025
3db8b5d
Add `--log-level=debug` to the command line help
cderv Jun 12, 2025
d56320f
Add some testing for --log-level, --log and --log-format options
cderv Jun 12, 2025
b7cd0b8
hugo-md - add rendering for non-float layouts (#12676)
cscheid Jun 12, 2025
be710f9
Merge branch 'main' into enhance/12805
cscheid Jun 13, 2025
a6b53c0
fix tests
cscheid Jun 13, 2025
294f1fe
fix tests
cscheid Jun 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/inspect/inspect-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
* Copyright (C) 2020-2022 Posit Software, PBC
*/

import { Format } from "../config/types.ts";
import { Format, FormatIdentifier } from "../config/types.ts";
import { Extension } from "../extension/types.ts";
import {
FileInclusion,
Expand All @@ -25,6 +25,7 @@ export interface InspectedFile {
includeMap: FileInclusion[];
codeCells: InspectedMdCell[];
metadata: Record<string, unknown>;
outputFiles: Record<string, FormatIdentifier>;
}

export interface InspectedConfig {
Expand Down
6 changes: 6 additions & 0 deletions src/inspect/inspect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ import {
import { validateDocumentFromSource } from "../core/schema/validate-document.ts";
import { error } from "../deno_ral/log.ts";
import { ProjectContext } from "../project/types.ts";
import { resolveOutputFileNames } from "../project/project-index.ts";

export function isProjectConfig(
config: InspectedConfig,
Expand Down Expand Up @@ -90,6 +91,7 @@ const inspectProjectConfig = async (context: ProjectContext) => {
return undefined;
}
const fileInformation: Record<string, InspectedFile> = {};
await resolveOutputFileNames(context);
for (const file of context.files.input) {
await populateFileInformation(context, fileInformation, file);
}
Expand Down Expand Up @@ -144,6 +146,7 @@ const populateFileInformation = async (
[],
codeCells: context.fileInformationCache.get(file)?.codeCells ?? [],
metadata: context.fileInformationCache.get(file)?.metadata ?? {},
outputFiles: context.fileInformationCache.get(file)?.outputFiles ?? {},
};
};

Expand Down Expand Up @@ -213,9 +216,11 @@ const inspectDocumentConfig = async (path: string) => {
);
}

path = normalizePath(path);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the normalization that creates the absolute path. As it was added explicitly, I guess you are thinking we should use absolute path in inspect output ?

This will change the content and so the filtering to apply like in our test

const json = JSON.parse(Deno.readTextFileSync(output));
const info = json.fileInformation["docs/websites/issue-9253/index.qmd"];

I guess other tools are doing that too...

I really wonder if this is a good idea that an inspect output from Windows and from Linux / Mac will differ totally. It seems like other tools that will use it will need to handle this.

But this could be discussed because those tools will also run on the OS, and generating an absolute path should give the same result.

I guess using relative paths seems more generic. We could have a first field in inspect for the project path (project.dir today), and then only use a relative path to this project path. An absolute path could then be recreated by joining the two. IDK 🤷

Anyhow, breaking change to handle if we keep this normalization

  • Tests to adapt
  • External tools to warn (like Connect or target - the two I know of)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that because I wanted the output of quarto inspect to be independent of the directory from which it is invoked, and equivalent between single-file calls and project-wide calls. If we're going to use the JSON information to make inferences about the files, it seems that it would be more useful to not have to know which directory the call came from.

Consider a project like this:

find . -type f | sort
./_quarto.yml
./about.qmd
./dir1/index.qmd
./dir2/index.qmd
./index.qmd
./styles.css

Now consider this:

$ cd dir1
$ quarto inspect index.qmd | jq .fileInformation
{
  "index.qmd": {
    "includeMap": [],
    "codeCells": [],
    "metadata": {}
  }
}
$ quarto inspect index.qmd | jq .project.fileInformation
{
  "/Users/cscheid/Desktop/daily-log/2025/06/13/test-dirs/dir2/index.qmd": {
    "includeMap": [],
    "codeCells": [],
    "metadata": {}
  },
  "/Users/cscheid/Desktop/daily-log/2025/06/13/test-dirs/index.qmd": {
    "includeMap": [],
    "codeCells": [],
    "metadata": {
      "title": "test-dirs"
    }
  },
  "/Users/cscheid/Desktop/daily-log/2025/06/13/test-dirs/about.qmd": {
    "includeMap": [],
    "codeCells": [],
    "metadata": {
      "title": "About"
    }
  },
  "/Users/cscheid/Desktop/daily-log/2025/06/13/test-dirs/dir1/index.qmd": {
    "includeMap": [],
    "codeCells": [],
    "metadata": {}
  }
}

Notice two things:

  1. We don't know which of the two dir1/index.qmd or dir2/index.qmd that first pipeline refers to.
  2. The project information is already built with absolute paths.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's another ugly related issue on main which we will need to fix:

$ cd quarto-cli/tests/docs/websites/issue-9253
$  quarto inspect | jq '.fileInformation["/Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/websites/issue-9253/index.qmd"].includeMap[].source'
"/Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/websites/issue-9253/index.qmd"
"_include.qmd"
"/Users/cscheid/repos/github/quarto-dev/quarto-cli/tests/docs/websites/issue-9253/index.qmd"
"_include.qmd"

await context.resolveFullMarkdownForFile(engine, path);
await projectResolveCodeCellsForFile(context, engine, path);
await projectFileMetadata(context, path);
await resolveOutputFileNames(context);
const fileInformation = context.fileInformationCache.get(path);

// data to write
Expand All @@ -231,6 +236,7 @@ const inspectDocumentConfig = async (path: string) => {
includeMap: fileInformation?.includeMap ?? [],
codeCells: fileInformation?.codeCells ?? [],
metadata: fileInformation?.metadata ?? {},
outputFiles: fileInformation?.outputFiles ?? {},
},
},
};
Expand Down
39 changes: 26 additions & 13 deletions src/project/project-index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,11 @@ import {
import { kTitle } from "../config/constants.ts";
import { fileExecutionEngine } from "../execute/engine.ts";

import { projectConfigFile, projectOutputDir } from "./project-shared.ts";
import {
ensureFileInformationCache,
projectConfigFile,
projectOutputDir,
} from "./project-shared.ts";
import { projectScratchPath } from "./project-scratch.ts";
import { parsePandocTitle } from "../core/pandoc/pandoc-partition.ts";
import { readYamlFromString } from "../core/yaml.ts";
Expand Down Expand Up @@ -297,20 +301,13 @@ export async function resolveInputTarget(
}
}

export async function inputFileForOutputFile(
// populates the outputNameIndex field in the project context
export async function resolveOutputFileNames(
project: ProjectContext,
output: string,
): Promise<{ file: string; format: Format } | undefined> {
// compute output dir
): Promise<void> {
if (project.outputNameIndex) return;
const outputDir = projectOutputDir(project);

// full path to output (it's relative to output dir)
output = isAbsolute(output) ? output : join(outputDir, output);

if (project.outputNameIndex !== undefined) {
return project.outputNameIndex.get(output);
}

project.outputNameIndex = new Map();
for (const file of project.files.input) {
const inputRelative = relative(project.dir, file);
Expand All @@ -319,6 +316,8 @@ export async function inputFileForOutputFile(
relative(project.dir, file),
);
if (index) {
const cache = ensureFileInformationCache(project, file);
cache.outputFiles = cache.outputFiles || {};
Object.keys(index.formats).forEach((key) => {
const format = index.formats[key];
const outputFile = formatOutputFile(format);
Expand All @@ -329,11 +328,25 @@ export async function inputFileForOutputFile(
outputFile,
);
project.outputNameIndex!.set(formatOutputPath, { file, format });
cache.outputFiles![formatOutputPath] = format.identifier;
}
});
}
}
return project.outputNameIndex.get(output);
}

export async function inputFileForOutputFile(
project: ProjectContext,
output: string,
): Promise<{ file: string; format: Format } | undefined> {
// compute output dir
const outputDir = projectOutputDir(project);

// full path to output (it's relative to output dir)
output = isAbsolute(output) ? output : join(outputDir, output);

await resolveOutputFileNames(project);
return project.outputNameIndex?.get(output);
}

export async function inputTargetIndexForOutputFile(
Expand Down
3 changes: 2 additions & 1 deletion src/project/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
*/

import { RenderServices } from "../command/render/types.ts";
import { Metadata, PandocFlags } from "../config/types.ts";
import { FormatIdentifier, Metadata, PandocFlags } from "../config/types.ts";
import { Format, FormatExtras } from "../config/types.ts";
import { Brand, LightDarkBrand } from "../core/brand/brand.ts";
import { MappedString } from "../core/mapped-text.ts";
Expand Down Expand Up @@ -57,6 +57,7 @@ export type FileInformation = {
target?: ExecutionTarget;
metadata?: Metadata;
brand?: LightDarkBrand;
outputFiles?: Record<string, FormatIdentifier>;
};

export interface ProjectContext extends Cloneable<ProjectContext> {
Expand Down
2 changes: 1 addition & 1 deletion src/project/types/single-file/single-file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ export async function singleFileProjectContext(
dir: normalizePath(dirname(source)),
engines: [],
files: {
input: [],
input: [normalizePath(source)],
},
notebookContext,
environment: () => environmentMemoizer(result),
Expand Down
3 changes: 2 additions & 1 deletion tests/smoke/inspect/inspect-cleanup.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
*
*/

import { normalizePath } from "../../../src/core/path.ts";
import { existsSync } from "../../../src/deno_ral/fs.ts";
import { } from "../../../src/project/types.ts";
import {
Expand All @@ -25,7 +26,7 @@ import { assert } from "testing/asserts";
verify: async (outputs: ExecuteOutput[]) => {
assert(existsSync(output));
const json = JSON.parse(Deno.readTextFileSync(output));
assert(json.fileInformation["docs/inspect/cleanup-issue-12336/cleanup-bug.qmd"].metadata.engine === "jupyter");
assert(json.fileInformation[normalizePath("docs/inspect/cleanup-issue-12336/cleanup-bug.qmd")].metadata.engine === "jupyter");
assert(!existsSync("docs/inspect/cleanup-issue-12336/cleanup-bug.quarto_ipynb"));
}
}
Expand Down
7 changes: 4 additions & 3 deletions tests/smoke/inspect/inspect-code-cells.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
testQuartoCmd,
} from "../../test.ts";
import { assert } from "testing/asserts";
import { normalizePath } from "../../../src/core/path.ts";

(() => {
const input = "docs/project/book/_include.qmd";
Expand All @@ -26,7 +27,7 @@ import { assert } from "testing/asserts";
verify: async (outputs: ExecuteOutput[]) => {
assert(existsSync(output));
const json = JSON.parse(Deno.readTextFileSync(output));
const info = json.fileInformation["docs/project/book/_include.qmd"];
const info = json.fileInformation[normalizePath("docs/project/book/_include.qmd")];
const codeCells = info.codeCells;
assertObjectMatch(info.codeCells[0], {
start: 0,
Expand Down Expand Up @@ -65,12 +66,12 @@ import { assert } from "testing/asserts";
verify: async (outputs: ExecuteOutput[]) => {
assert(existsSync(output));
const json = JSON.parse(Deno.readTextFileSync(output));
const info = json.fileInformation["docs/inspect/10039.qmd"];
const info = json.fileInformation[normalizePath("docs/inspect/10039.qmd")];
const codeCells = info.codeCells;
assertObjectMatch(info.codeCells[1], {
"start": 14,
"end": 18,
"file": "docs/inspect/10039.qmd",
"file": normalizePath("docs/inspect/10039.qmd"),
"source": "p[[1]]\n",
"language": "r",
"metadata": {
Expand Down
14 changes: 8 additions & 6 deletions tests/smoke/inspect/inspect-include.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
testQuartoCmd,
} from "../../test.ts";
import { assert } from "testing/asserts";
import { normalizePath } from "../../../src/core/path.ts";

(() => {
const input = "docs/inspect/foo.qmd";
Expand All @@ -22,20 +23,21 @@ import { assert } from "testing/asserts";
{
name: "inspect-include",
verify: async (outputs: ExecuteOutput[]) => {
assert(existsSync("docs/inspect/foo.json"));
const json = JSON.parse(Deno.readTextFileSync("docs/inspect/foo.json"));
assertObjectMatch(json.fileInformation["docs/inspect/foo.qmd"].includeMap[0],
assert(existsSync(output));
const normalizedPath = normalizePath(input);
const json = JSON.parse(Deno.readTextFileSync(output));
assertObjectMatch(json.fileInformation[normalizedPath].includeMap[0],
{
source: input,
source: normalizedPath,
target: "_bar.qmd"
});
}
}
],
{
teardown: async () => {
if (existsSync("docs/inspect/foo.json")) {
Deno.removeSync("docs/inspect/foo.json");
if (existsSync(output)) {
Deno.removeSync(output);
}
}
},
Expand Down
3 changes: 2 additions & 1 deletion tests/smoke/inspect/inspect-recursive-include.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
testQuartoCmd,
} from "../../test.ts";
import { assert, assertEquals } from "testing/asserts";
import { normalizePath } from "../../../src/core/path.ts";

(() => {
const input = "docs/websites/issue-9253/index.qmd";
Expand All @@ -25,7 +26,7 @@ import { assert, assertEquals } from "testing/asserts";
verify: async (outputs: ExecuteOutput[]) => {
assert(existsSync(output));
const json = JSON.parse(Deno.readTextFileSync(output));
const info = json.fileInformation["docs/websites/issue-9253/index.qmd"];
const info = json.fileInformation[normalizePath("docs/websites/issue-9253/index.qmd")];
const includeMap: FileInclusion[] = info.includeMap;
assertObjectMatch(info.includeMap[0], { target: "_include.qmd" });
assertObjectMatch(info.includeMap[1], { source: "_include.qmd", target: "_include2.qmd" });
Expand Down
Loading