Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hooks for calling arbitrary code around each benchmark #193

Merged
merged 13 commits into from
Jul 28, 2024

Conversation

mdboom
Copy link
Collaborator

@mdboom mdboom commented Jun 24, 2024

This change adds a concept of "hooks" to pyperf for running arbitrary code before and after the actual internal benchmarking code. These are implemented with entry points, so it should be possible to write an external plugin to add new integrations and functionality.

There are a few real use cases that the Faster CPython team already needs this for:

  • Integration with codspeed to turn on/off the instrumentation only around the benchmarking code
  • Doing coverage analysis of benchmarks
  • Adding sys.monitoring hooks around benchmarks to do custom analysis of which specific bytecodes are run during benchmarking

The pystats integration that already exists in pyperf also should have, in hindsight, been implemented this way, so this PR also refactors things to do that, and it serves as a proof-of-concept of the new hook functionality.

This PR needs tests but I thought I would submit a draft first to ensure this seems like the right direction first.

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the concept of hooks.

Would it be possible to use importlib.metadata instead of setuptools pkg_resources? See: https://docs.python.org/dev/library/importlib.metadata.html#entry-points

@mdboom
Copy link
Collaborator Author

mdboom commented Jun 25, 2024

Would it be possible to use importlib.metadata instead of setuptools pkg_resources? See: https://docs.python.org/dev/library/importlib.metadata.html#entry-points

Sure -- I've never used entry points before -- I hadn't realized that's the new canonical way.

@mdboom mdboom marked this pull request as ready for review July 8, 2024 14:53
@mdboom mdboom requested a review from vstinner July 8, 2024 14:53
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. I don't know entry points, so I cannot review this change in details.

Copy link

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@corona10 corona10 merged commit c6c33d9 into psf:main Jul 28, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants