Skip to content

Specify flit build-system within pyproject.toml #1317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

06kellyjac
Copy link
Contributor

@06kellyjac 06kellyjac commented Aug 1, 2025

Shifted PR to specify filt build system as used by the team.


Old PR Message

Addresses the following warning when building:

Package would be ignored

Python recognizes 'garak.probes' as an importable package(1),
but it is absent from setuptools' packages configuration.

This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'garak.probes' is explicitly added
to the packages configuration field.

Alternatively, you can also rely on setuptools' discovery methods
(for example by using find_namespace_packages(...)/find_namespace:
instead of find_packages(...)/find:).

You can read more about "package discovery" on setuptools documentation page:

If you don't want 'garak.probes' to be distributed and are
already explicitly excluding 'garak.probes' via
find_namespace_packages(...)/find_namespace or find_packages(...)/find,
you can try to use exclude_package_data, or include-package-data=False in
combination with a more fine grained package-data configuration.

You can read more about "package data files" on setuptools documentation page:

1: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any .py files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.


garak.probes is explicitly referenced in the README.md so should be properly included.
If some packages should be excluded LMK.

@jmartin-tech
Copy link
Collaborator

I suspect this is not how we want to expose packages for this project. The library usage pattern interface is still being worked out however most sub-packages for garak are not expected to be directly consumed.

Note the README.md references are about developer testing for creating new probes that are consumed internally in the tool and is not a reference to downstream consumer access and usage. When the project can officially support a bring your own plugin pattern this feedback may change.

Note the project currently uses flit to build and release packages, I have not seen this warning in that process. We can look into the warning from other tooling however maintaining clean build process for all python builders is not in scope for this project.

@06kellyjac, I noticed a number of PRs this week that seem to be targeting a goal. If you would like to reach out on discord, we can coordinate on how to get the right context shared and smooth out the feedback loop.

@06kellyjac
Copy link
Contributor Author

Ah, it's handy to know you're using flit, I can adjust my build to match. I can put in a separate PR or amend this one to have the pyproject.toml explicitly use flit rather than setuptools e.g: https://github.com/pyg-team/pytorch_geometric/blob/master/pyproject.toml#L2-L3

Yeah joining the discord is a good idea, thanks :) I'll find the link

With the team using flit to build and package garak it can be defined in
the pyproject.toml rather than with setuptools.
@06kellyjac 06kellyjac changed the title Adjust how pyproject and setuptools finds packages Specify flit build-system within pyproject.toml Aug 1, 2025
@jmartin-tech
Copy link
Collaborator

How does this impact consumption of MANIFEST.in, the changes here suggest a close comparison is needed as explicit inclusions and exclusions in that file are needed to get a viable install.

@06kellyjac
Copy link
Contributor Author

Well with MANIFEST.in and setuptools I was getting warnings:

reading manifest file 'garak.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: manifest_maker: MANIFEST.in, line 1: unknown action 'garak/analyze'

warning: manifest_maker: MANIFEST.in, line 2: unknown action 'garak/buffs'

warning: manifest_maker: MANIFEST.in, line 3: unknown action 'garak/configs'

warning: manifest_maker: MANIFEST.in, line 4: unknown action 'garak/detectors'

warning: manifest_maker: MANIFEST.in, line 5: unknown action 'garak/evaluators'

warning: manifest_maker: MANIFEST.in, line 6: unknown action 'garak/generators'

warning: manifest_maker: MANIFEST.in, line 7: unknown action 'garak/harnesses'

warning: manifest_maker: MANIFEST.in, line 8: unknown action 'garak/probes'

warning: manifest_maker: MANIFEST.in, line 9: unknown action 'garak/resources'

warning: manifest_maker: MANIFEST.in, line 10: unknown action 'garak/data'

So I'm not sure it was working correctly for a while now.
Going to this link https://packaging.python.org/en/latest/guides/using-manifest-in/ which now points to https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
Looking at the Command table they should probably be prefixed with graft

For filt IDK if it even needs a MANIFEST.in I think it just leverages https://flit.pypa.io/en/stable/pyproject_toml.html#module-section & https://flit.pypa.io/en/stable/pyproject_toml.html#external-data-section

Several items in MANIFEST.in were triggering
`warning: manifest_maker: MANIFEST.in, line 1: unknown action` warnings

With flint the convention for additional resources is to use
`[tool.flit.sdist]` or `[tool.flit.external-data]`
https://flit.pypa.io/en/stable/pyproject_toml.html#external-data-section
@06kellyjac
Copy link
Contributor Author

searching my result built with flit

λ fd . ./result/lib/python3.13/site-packages/garak -tf -E "*.py" -E '__pycache__'
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_about_z.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_detector.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_end_module.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_footer.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_group.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_header.jinja
./result/lib/python3.13/site-packages/garak/analyze/templates/digest_probe.jinja
./result/lib/python3.13/site-packages/garak/configs/bag.yaml
./result/lib/python3.13/site-packages/garak/configs/broad.yaml
./result/lib/python3.13/site-packages/garak/configs/default.yaml
./result/lib/python3.13/site-packages/garak/configs/fast.yaml
./result/lib/python3.13/site-packages/garak/configs/full.yaml
./result/lib/python3.13/site-packages/garak/configs/long_attack_gen.yaml
./result/lib/python3.13/site-packages/garak/configs/notox.yaml
./result/lib/python3.13/site-packages/garak/configs/tox_and_buffs.yaml
./result/lib/python3.13/site-packages/garak/data/autodan/autodan_init.txt
./result/lib/python3.13/site-packages/garak/data/autodan/autodan_prompts.txt
./result/lib/python3.13/site-packages/garak/data/autodan/prompt_group.pth
./result/lib/python3.13/site-packages/garak/data/banners
./result/lib/python3.13/site-packages/garak/data/beast/suffixes.txt
./result/lib/python3.13/site-packages/garak/data/book_cloze.tsv
./result/lib/python3.13/site-packages/garak/data/book_cloze_80.tsv
./result/lib/python3.13/site-packages/garak/data/calibration/bag.md
./result/lib/python3.13/site-packages/garak/data/calibration/calibration-2024-09update.json
./result/lib/python3.13/site-packages/garak/data/calibration/calibration-2024-summer.json
./result/lib/python3.13/site-packages/garak/data/calibration/calibration-2025-05.json
./result/lib/python3.13/site-packages/garak/data/calibration/calibration-alpha-202407.json
./result/lib/python3.13/site-packages/garak/data/calibration/calibration.json
./result/lib/python3.13/site-packages/garak/data/donotanswer/discrimination_exclusion_toxicity_hateful_offensive.txt
./result/lib/python3.13/site-packages/garak/data/donotanswer/human_chatbox.txt
./result/lib/python3.13/site-packages/garak/data/donotanswer/information_hazard.txt
./result/lib/python3.13/site-packages/garak/data/donotanswer/malicious_uses.txt
./result/lib/python3.13/site-packages/garak/data/donotanswer/misinformation_harms.txt
./result/lib/python3.13/site-packages/garak/data/gcg/gcg.txt
./result/lib/python3.13/site-packages/garak/data/graph_connectivity.json
./result/lib/python3.13/site-packages/garak/data/guardian_cloze.tsv
./result/lib/python3.13/site-packages/garak/data/inthewild_jailbreak_llms.txt
./result/lib/python3.13/site-packages/garak/data/ldnoobw-en.txt
./result/lib/python3.13/site-packages/garak/data/misp_descriptions.tsv
./result/lib/python3.13/site-packages/garak/data/nyt_cloze.tsv
./result/lib/python3.13/site-packages/garak/data/ofcom-potentially-offensive.txt
./result/lib/python3.13/site-packages/garak/data/packagehallucination/rust_std_entries-1_84_0
./result/lib/python3.13/site-packages/garak/data/payloads/access_shell_commands.json
./result/lib/python3.13/site-packages/garak/data/payloads/domains_latentinjection.json
./result/lib/python3.13/site-packages/garak/data/payloads/encoded.json
./result/lib/python3.13/site-packages/garak/data/payloads/example_domains_xss.json
./result/lib/python3.13/site-packages/garak/data/payloads/keyedprod_win10.json
./result/lib/python3.13/site-packages/garak/data/payloads/python_code_execution.json
./result/lib/python3.13/site-packages/garak/data/payloads/rude_chinese.json
./result/lib/python3.13/site-packages/garak/data/payloads/rude_french.json
./result/lib/python3.13/site-packages/garak/data/payloads/slur_terms_en.json
./result/lib/python3.13/site-packages/garak/data/payloads/sql_injection.json
./result/lib/python3.13/site-packages/garak/data/payloads/text_en.json
./result/lib/python3.13/site-packages/garak/data/payloads/web_html_js.json
./result/lib/python3.13/site-packages/garak/data/payloads/whois_injection_contexts.json
./result/lib/python3.13/site-packages/garak/data/phrasing/future_tense_en.txt
./result/lib/python3.13/site-packages/garak/data/phrasing/past_tense_en.txt
./result/lib/python3.13/site-packages/garak/data/potter_cloze.tsv
./result/lib/python3.13/site-packages/garak/data/primality_testing.json
./result/lib/python3.13/site-packages/garak/data/profanity_en.csv
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_flirtation.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_identity_attack.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_insult.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_profanity.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_severe_toxicity.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_sexually_explicit.txt
./result/lib/python3.13/site-packages/garak/data/realtoxicityprompts/rtp_threat.txt
./result/lib/python3.13/site-packages/garak/data/safebench_filenames.txt
./result/lib/python3.13/site-packages/garak/data/safebenchtiny_filenames.txt
./result/lib/python3.13/site-packages/garak/data/senator_search.json
./result/lib/python3.13/site-packages/garak/data/slurprompts.jsonl
./result/lib/python3.13/site-packages/garak/data/slurprompts_mini.jsonl
./result/lib/python3.13/site-packages/garak/data/slursreclaimedslurs.txt
./result/lib/python3.13/site-packages/garak/data/tap/tap_jailbreaks.txt
./result/lib/python3.13/site-packages/garak/data/truefalse_falseclaims.txt
./result/lib/python3.13/site-packages/garak/data/truefalse_falseclaims_50.txt
./result/lib/python3.13/site-packages/garak/data/typology_payloads.tsv
./result/lib/python3.13/site-packages/garak/resources/garak.core.yaml
./result/lib/python3.13/site-packages/garak/resources/plugin_cache.json

I did a diff with before (setuptools + manifest) and after (flit and this PR) and had no difference

λ diff build-before.txt build.txt

I'd note the following files I'd probably expect to be there based on MANFIEST.in and the flit config but don't exist for before or after are: CA_DCO.md CONTRIBUTING.md FAQ.md README.md SECURITY.md


(.py or .pyc files don't need any special inclusion, [tool.flit.module] handles that)

@06kellyjac
Copy link
Contributor Author

pip install test seems to have passed which is promising

@jmartin-tech jmartin-tech self-assigned this Aug 4, 2025
@jmartin-tech jmartin-tech added the housekeeping Repository and project maintenance label Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
housekeeping Repository and project maintenance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants