Skip to content

Mixed-input Level Manager by Virtualization - #744

Merged
ssoelvsten merged 12 commits into
mainfrom
internal/virtual-level_merger
Jan 29, 2026
Merged

Mixed-input Level Manager by Virtualization#744
ssoelvsten merged 12 commits into
mainfrom
internal/virtual-level_merger

Conversation

@ssoelvsten

@ssoelvsten ssoelvsten commented Jan 13, 2026

Copy link
Copy Markdown
Owner

Resolves #727 by making the input handlers for the level_merger virtual. Doing so provides us with the ability to have mixed inputs (files and/or generators). This is needed for #733 .

In practice, we always know the type of each input handler at compile time, so we should be able to remove the use of virtual. Yet, this is a quick solution to unblock #733 . Assuming the CI benchmarking does not indicate this is too much of an issue, I'll defer the fully templated solution for later. Especially, if a student or group of students would like to take a look at #444 and #733 .

Otherwise, I'll postpone merging this until another refactor to include typename Inputs... in the templated level_merger to derive the individual handlers within it (probably via a std::tuple). For more details, see cppreference on parameter packing.

@ssoelvsten ssoelvsten self-assigned this Jan 13, 2026
@ssoelvsten ssoelvsten added ✨ feature New operation or other feature ✨ optimisation It's all about speed / space ✨ code quality Uncle Bob would be proud labels Jan 13, 2026
@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch from 0f76eec to ef86c89 Compare January 13, 2026 15:46
@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'hex/hein_08_5x5-11_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -0.75% (stdev: 0.48%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 16683.33 16808.00
Standard Deviation 48.99 80.55

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (12-Queens)

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -1.16% (stdev: 0.92%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 11585.33 11719.67
Standard Deviation 34.21 107.87

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'breakthrough/3x4_19_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 1.07% (stdev: 2.65%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 21639.33 21407.67
Standard Deviation 573.73 247.81

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🔴 Regression Test (Picotrav 'adder')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -2.77% (stdev: 0.53%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 10634.33 10929.33
Standard Deviation 56.57 36.47

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'ep_dual/8x8_6_e-8-1_p-2-3_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -0.22% (stdev: 0.36%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 4968.00 4979.00
Standard Deviation 14.93 17.69

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'breakthrough_dual/3x6_10_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 0.92% (stdev: 0.76%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 5191.00 5143.33
Standard Deviation 39.34 26.76

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'connect4/6x6_11_connect4_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 0.64% (stdev: 1.07%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 12052.33 11975.67
Standard Deviation 19.35 127.90

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'httt/4x4_9_tippy_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 0.40% (stdev: 0.35%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 9183.00 9146.33
Standard Deviation 29.72 32.04

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'domineering/5x5_13_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -0.37% (stdev: 0.23%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 15028.33 15083.67
Standard Deviation 14.57 34.03

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (QBF 'ep/8x8_7_e-8-1_p-3-4_bwnib')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 0.36% (stdev: 0.61%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 28453.67 28351.00
Standard Deviation 172.79 163.89

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (14-Queens)

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -0.02% (stdev: 1.01%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 266182.33 266230.00
Standard Deviation 2698.57 2468.55

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (Picotrav 'mem_ctrl')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of 1.37% (stdev: 0.78%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 273112.00 269369.67
Standard Deviation 785.02 2089.21

Number of samples: 3

@github-actions

github-actions Bot commented Jan 13, 2026

Copy link
Copy Markdown

🟡 Regression Test (Picotrav 'arbiter')

'ssoelvsten/adiar/internal/virtual-level_merger' is a change in performance of -3.83% (stdev: 44.40%).

... origin/main ssoelvsten/adiar/internal/virtual-level_merger
Mean 47267.56 49078.00
Standard Deviation 15234.17 21790.18

Number of samples: 9

@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch from ef86c89 to f6e31ba Compare January 14, 2026 10:09
@ssoelvsten

ssoelvsten commented Jan 15, 2026

Copy link
Copy Markdown
Owner Author

There are two things going on right now.

  • The slowdown seems to be between 1% and 2%. This seems pretty reasonable since, according to Stroustrup, virtual functions in general incur an overhead between 10% and 20%. Yet, in this case only a small part of the computation time is spent on querying the next level.
  • While the tests compile and runs without any issues on my machine, the CI is broken due to a failure at the linking stage of the test binary. It's quite hard for me to debug this issue when doing so blindly and with such slow response time.

I'll proceed with reaping the benefits of this approach (see #727) and then take a look at parameter packs and std::apply; the latter would remove the use of virtual which in turn hopefully also fixes the CI.

@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch 2 times, most recently from e09be44 to 6d1c929 Compare January 16, 2026 10:15
This was only used with the 'narc_ifstream' which is currently not
used for anything (and neither planned for anything). Keeping this
feature is (1) maintenance and (2) complicated with the virtualization
of the 'level_merger'
It wasn't really used at all in any of the other places; it only made things
more complicated for no discernable benefit.
Which is the only type of argument it is given.
There is no reason to repeat it at the higher levels (except possibly for
documentation). Having it only at the basic streams still propagates the
information without additional maintenance requirements.
@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch from 6d1c929 to b1a350b Compare January 16, 2026 11:37
@ssoelvsten

ssoelvsten commented Jan 16, 2026

Copy link
Copy Markdown
Owner Author

There we go, #727 hacks in Intercut fully resolved. Now, only onto removing the virtual functions or fix the CI breaking with them.

P.S. Also, thanks to Mathias Rav's hammer; the more I use it, the more I appreciate it.

This new level merger provides us with the ability to use inputs of
different types, e.g. a `levelized_file` and a `generator`. This will
allow us to skip redundant (and expensive) temporary files in the
Intercut operation that were only created to make the types match.
@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch from b1a350b to 8e1da29 Compare January 29, 2026 08:45
@ssoelvsten ssoelvsten changed the title Mixed-input Level Manager Mixed-input Level Manager by Virtualization Jan 29, 2026
This allows us to deal with sequential/random-access data that is
stored in internal memory. For example, this could be a small list
of levels (at most 2MiB) from a generator which needs to be read
more than once, e.g. Intercut.
…nternal, T>'

The 'tpie::internal_vector<T>' has a few pecularities and missing
features. Most importantly, it does not support the ability to iterate
through it in reverse. So, we'll just use the 'tpie::array<T>'
directly.
@ssoelvsten
ssoelvsten force-pushed the internal/virtual-level_merger branch from 8e1da29 to b99bbde Compare January 29, 2026 09:13
@codecov

codecov Bot commented Jan 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.106%. Comparing base (c4e53be) to head (b99bbde).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@              Coverage Diff              @@
##              main      #744       +/-   ##
=============================================
+ Coverage   97.089%   97.106%   +0.017%     
=============================================
  Files           98        99        +1     
  Lines         7215      7257       +42     
=============================================
+ Hits          7005      7047       +42     
  Misses         210       210               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ssoelvsten
ssoelvsten marked this pull request as ready for review January 29, 2026 09:40
@ssoelvsten
ssoelvsten merged commit 1efb1c9 into main Jan 29, 2026
46 of 51 checks passed
@ssoelvsten
ssoelvsten deleted the internal/virtual-level_merger branch January 29, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

✨ code quality Uncle Bob would be proud ✨ feature New operation or other feature ✨ optimisation It's all about speed / space

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for merging levels from different types of sources

1 participant