how do i add additional files to a wheel?

[DetachHead]

i'm having a lot of trouble figuring out how to build the wheel for my package. for context, my project is an npm package wrapped in a python package, so the build process needs to install the npm dependencies and build the npm package, then move it into the wheel.

currently, i have a pdm_build.py script that technically works, but i think i'm doing it completely wrong because it's not using any of the hooks. as a result, i think it makes builds and other unrelated commands extremely slow because it builds the npm package and copies the files every time pdm_build.py gets imported.

to fix this, i tried rewriting it to use pdm_build_update_files, but i couldn't get that to work. i'm not sure i understand what this hook actually does. at first i thought adding files to the files dict would be enough to get pdm to add it to the wheel, but this doesn't seem to actually do anything:

def pdm_build_update_files(context: Context, files: dict[str, Path]):
    run_npm("ci")
    run_npm("run", "build:cli:dev")
    npm_package_dir = Path("packages/pyright")
    pypi_package_dir = Path("basedpyright")
    dist_dir = Path("dist")
    npm_script_paths = cast(PackageJson, loads((npm_package_dir / "package.json").read_text()))[
        "bin"
    ].values()

    for file in (npm_package_dir / dist_dir).rglob("**/*"):
        if not file.is_dir():
            files[str(file.relative_to(npm_package_dir))] = file

    for script_path in npm_script_paths:
        files[script_path] = pypi_package_dir / script_path

so then i rewrote it to be more like what i had originally, where it just manually copies the files into the python package's directory:

def pdm_build_update_files(context: Context, files: dict[str, Path]):
    run_npm("ci")
    run_npm("run", "build:cli:dev")
    npm_package_dir = Path("packages/pyright")
    pypi_package_dir = Path("basedpyright")
    dist_dir = Path("dist")
    npm_script_paths = cast(PackageJson, loads((npm_package_dir / "package.json").read_text()))[
        "bin"
    ].values()

    copytree(npm_package_dir / dist_dir, pypi_package_dir / dist_dir, dirs_exist_ok=True)
    for script_path in npm_script_paths:
        _ = copyfile(npm_package_dir / script_path, pypi_package_dir / script_path)

this works with editable installs, but when installing it in non-editable mode, the files don't seem to get copied into the wheel.

the last thing i tried was using a pdm_build_initialize hook to build the npm package, and the tool.pdm.build.wheel-data option in pyproject.toml to copy the files into the wheel:

def pdm_build_initialize(context: Context) -> None:
    _ = context.ensure_build_dir()
    run_npm("ci")
    run_npm("run", "build:cli:dev")

[tool.pdm.build.wheel-data]
include = [
    { path = "packages/pyright/dist/**/*", relative-to = "packages/pyright/" },
    { path = "packages/pyright/*.js", relative-to = "packages/" },
]

this didn't work either. when attempting to install it, i get this error:

[InvalidWheelSource]: (<pdm.installers.installers.PackageWheelSource object at 0x0000021C5140E660>, 'basedpyright-1.12.6.data/include/dist/pyright-langserver.js is not contained in a valid .data 
subdirectory.')

any guidance would be appreciated as i have no clue what i'm doing lol

[DetachHead]

ok so i got it working. it turns out i need to update the files dict when building the wheel, but copy the files normally when doing an editable install. i also realized the files were being copied to the same level as my package, instead of inside the package itself (ie. it was copying the files to a separate package called dist in site-packages instead of in basedpyright/dist) because i wasn't including pypi_package_dir in the keys when updating the files dict.

here's what my hook looks like now, which seems to work:

def pdm_build_update_files(context: Context, files: dict[str, Path]):
    context.ensure_build_dir()
    run_npm("ci")
    run_npm("run", "build:cli:dev")

    npm_package_dir = Path("packages/pyright")
    pypi_package_dir = Path("basedpyright")
    dist_dir = Path("dist")
    npm_script_paths = cast(PackageJson, loads((npm_package_dir / "package.json").read_text()))[
        "bin"
    ].values()

    if context.target == "editable":
        copytree(npm_package_dir / dist_dir, pypi_package_dir / dist_dir, dirs_exist_ok=True)
        for script_path in npm_script_paths:
            _ = copyfile(npm_package_dir / script_path, pypi_package_dir / script_path)
    else:
        for file in (npm_package_dir / dist_dir).rglob("**/*"):
            if file.is_file():
                files[str(pypi_package_dir / file.relative_to(npm_package_dir))] = file

        for script_path in npm_script_paths:
            files[str(pypi_package_dir / script_path)] = npm_package_dir / script_path

as for tool.pdm.build.wheel-data, i still have no clue what that is or how it works.

i did find out by digging through the source code of the installer package that my error was because the key include is not a valid "scheme", whatever that means. it's briefly mentioned in this PEP but with no explanation at all as to what any of these "schemes" actually mean). despite the fact that the pdm documentation says that scripts, purelib, platlib, include and platinclude are all valid, and also doesn't explain what they are. but i just gave up on that and and went back to the pdm_build_update_files hook above and it seems to work fine.

also, those docs mention have a lot of references to this $prefix variable but it doesn't seem to explain anywhere what that prefix actually means. perhaps the documentation could be updated to clarify some of this.

[KotlinIsland]

Why does this go to the bin directory?

[tool.pdm.build.wheel-data]
# Install all files under scripts/ to the $prefix/bin directory.
scripts = ["scripts/*"]

reference

[frostming]

scripts is a key of the installation schema, it maps to <prefix>/bin like purelib maps to <prefix>/lib/pythonX.Y/site-packages
Run import sysconfig; print(sysconfig.get_paths()) to get a clue.

[KotlinIsland]

What is <prefix>? nothing in the docs explains that.

Is the behavior of the installation schema documented anywhere? I think the current docs could be improved. Trying to follow them without in depth knowledge of the python environment and build mechanics is not easy.

[frostming]

Did you run the command? You'll get what it means.

[KotlinIsland]

Yes I did, and that is useful.

But none of this is explained or linked to from the docs.

additionally, the docs uses $prefix without explaining what is it. from what I know now I assume it means the directory the env is located in.