[ntuple] automatic evolution from fixed-size array to RVec #19935
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
silverweed
reviewed
Sep 22, 2025
Test Results 21 files 21 suites 3d 13h 50m 30s ⏱️ Results for commit 90b5d4b. ♻️ This comment has been updated with latest results. |
jblomer
force-pushed
the
ntuple-evolution-fixed-arrays
branch
6 times, most recently
from
a555c89 to
7de82c0
Compare
jblomer
requested review from
martamaja10,
vepadulano and
bellenot
as code owners
jblomer
force-pushed
the
ntuple-evolution-fixed-arrays
branch
3 times, most recently
from
f0943cb to
0bccc9d
Compare
silverweed
approved these changes
Sep 24, 2025
jblomer
force-pushed
the
ntuple-evolution-fixed-arrays
branch
from
0bccc9d to
4947ca8
Compare
hahnjo
reviewed
Sep 24, 2025
jblomer
force-pushed
the
ntuple-evolution-fixed-arrays
branch
from
4947ca8 to
d737905
Compare
We can safely apply field substitutions in a field hierarchy if we control the time between field creation and connecting the field to the page source, i.e. if we can ensure no field values have been created in-between. This is the case for - the RNTupleReader - views - the RNTupleProcessor - RDF - source member fields used in schema evolution Only imposed models created by users cannot apply field substitutions.
This field used to expect a pre-contructed RVec of a certain size, which we can ensure in RDF but not elsewhere. Factor out the logic to resize RVecs in a type-erased way and use it in both this field and the RRVecField.
jblomer
force-pushed
the
ntuple-evolution-fixed-arrays
branch
from
d737905 to
90b5d4b
Compare
hahnjo
approved these changes
Sep 25, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In contrast to other automatic evolution mechanisms, this one cannot use
ReconcileOnDiskField()because for proper support the type of the field itself must change. In principle, we could stuff all the logic into RRecField, but it would be quite messy. We also discussed adding a "virtual column" for the fixed-size indexes; this doesn't work, however, because the index column stores offsets and not vector lengths, and it is also used to translate between global and local indexing.Thus, this PR adds the infrastructure to replace fields of just created field hierarchies (models). This is legit as long as the fields weren't exposed to the user yet, where the user, e.g., could have already created entries. As a result, the automatic evolution from fixed-size to RVec (and later std::vector) won't work with imposed models. But it does work for views and models reconstructed from disk, which are the most common use cases.
The infrastructure will also be used to switch between streamer field and class field.