Skip to content

New perf. metrics, stability and other improvements #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Oct 17, 2025
Merged

New perf. metrics, stability and other improvements #184

merged 11 commits into from
Oct 17, 2025

Conversation

Alexsandruss
Copy link
Contributor

@Alexsandruss Alexsandruss commented Apr 28, 2025 •
edited
Loading

Description

Changes:

  • Add support of LightGBM daal4py modelbuilders
  • Add garbage collection and result cleaning in datasets prefetching function to avoid out-of-memory errors
  • Add SKLBENCH_DATA_CACHE env variable as the first default location for datasets cache for convenience ($PWD/data_cache is still working if env variable is not set)
  • Change default dtype to float32
  • Adjust compatibility mode of report generator to work with latest versions of stock sklearn and RAPIDS and for other cases
  • Update collected performance metrics:
    • Add cost metrics counted in microdollars (most readable degree for usual case computation time)
    • Add CPU load profiling
    • Add RAM and VRAM usage profiling
    • Add coefficient of variation for time
    • Add 1st run time
    • Add 1st-mean run ratio
  • Change color scale from RED-YELLOW-GREEN to RED-WHITE-GREEN in perf. report for better readability
  • Add option for cache flushing between case runs
  • Remove Kaggle dataset source
  • Docs:
    • Note about experimental configs content and meaning
    • Move Benchmarking Config Specification to separate file
    • Add benchmarking scopes short explanation

PR should start as a draft, then move to ready for review state after CI is passed and all applicable checkboxes are closed.
This approach ensures that reviewers don't spend extra time asking for regular requirements.

You can remove a checkbox as not applicable only if it doesn't relate to this PR in any way.

Checklist to comply with before moving PR from draft:

PR completeness and readability

  • I have reviewed my changes thoroughly before submitting this pull request.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
  • Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
  • I have added a respective label(s) to PR if I have a permission for that.
  • I have resolved any merge conflicts that might occur with the base branch.

Testing

  • I have run it locally and tested the changes extensively.
  • All CI jobs are green or I have provided justification why they aren't.
  • I have extended testing suite if new functionality was introduced in this PR.

@Alexsandruss Alexsandruss added enhancement New feature or request docs documentation and readme update labels Apr 28, 2025
@Alexsandruss Alexsandruss mentioned this pull request Apr 28, 2025
Merged
9 tasks
@Alexsandruss Alexsandruss changed the title Updates and fixes New perf. metrics, stability improvements and other fixes Apr 28, 2025
@Alexsandruss Alexsandruss marked this pull request as ready for review April 28, 2025 11:45
@Alexsandruss Alexsandruss changed the title New perf. metrics, stability improvements and other fixes New perf. metrics, stability and other improvements Apr 28, 2025
ahuber21
ahuber21 reviewed Apr 30, 2025
View reviewed changes
Copy link
Contributor

@ahuber21 ahuber21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huge pile of changes that I only skimmed, but I don't see any obvious issues. Would you like comments or opinions on specific changes, or anything that needs to be double-checked?

BTW: I don't think the execution with shell=True is a problem here, as the command is hardcoded and can't be changed. But still it would be nice to work around it. Did you try running without shell=True?

david-cortes-intel
david-cortes-intel reviewed May 5, 2025
View reviewed changes
configs/BENCH-CONFIG-SPEC.md
- `INCLUDE` - Other configuration files whose parameter sets to include
- `PARAMETERS_SETS` - Benchmark parameters within each set
- `TEMPLATES` - List different setups with parameters sets template-specific parameters
- `SETS` - List parameters sets to include in the template
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could explain what a "set" is.

configs/BENCH-CONFIG-SPEC.md
|:---------------|:--------------|:--------|:------------|
|<h3>Benchmark workflow parameters</h3>||||
| `bench`:`taskset` | None | | Value for `-c` argument of `taskset` utility used over benchmark subcommand. |
| `bench`:`vtune_profiling` | None | | Analysis type for `collect` argument of Intel(R) VTune* Profiler tool. Linux* OS only. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VTune functionalities are quite undocumented. Could expand on them and add examples and screenshots.

david-cortes-intel
david-cortes-intel reviewed Oct 15, 2025
View reviewed changes
configs/weekly/pca.json
},
{
"data": {
"dataset": ["airline_depdelay", "bosch", "epsilon"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this bosch dataset is what @razdoburdin uses the most in his analyses. Could we obtain it from a different source? Otherwise, does the benchmark have some other dataset with similar characteristics?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we obtain it from a different source?

No, but it still can be loaded as custom dataset if preserved in correct cache format.

Otherwise, does the benchmark have some other dataset with similar characteristics?

I didn't search for that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@razdoburdin Any comments here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't use weekly for xgboost now, so no objections from my side

@Alexsandruss Alexsandruss merged commit d2124b6 into IntelPython:main Oct 17, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ahuber21 ahuber21 ahuber21 left review comments

@david-cortes-intel david-cortes-intel david-cortes-intel approved these changes

@icfaust icfaust Awaiting requested review from icfaust

@ethanglaser ethanglaser Awaiting requested review from ethanglaser ethanglaser is a code owner

@razdoburdin razdoburdin Awaiting requested review from razdoburdin razdoburdin is a code owner

@avolkov-intel avolkov-intel Awaiting requested review from avolkov-intel

Assignees

No one assigned

Labels

docs documentation and readme update enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Alexsandruss @ahuber21 @razdoburdin @david-cortes-intel