Skip to content

Commit 053f392

Browse files
committed
Update the Widen and decimal plan for the PRs since 2026-10-26
WD1 to WD4, CD3, CD3b and the decimal Interleave kernel merged, and CD4, CD5 and WD6 are open drafts. The plan now lists CD3b (#10939) and the Interleave kernel (CD4p, #10940) as dependencies of CD4, and CD3b as a dependency of CD6 and CD7. CD7 merges right after CD5 (#10905). D26 moves to CD6, because CD4 (#10946) adds no serialization lowering: the default writer executes a Widen, and reads of old files are the first ordinary source of one. The CD6 and WR1 briefs now carry that choice. D35 no longer makes WD6 wait for WD5. #10908 joins the follow-up issues, and the plan suggests settling the mechanism of Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> #10748 before WD5.
1 parent f36deae commit 053f392

6 files changed

Lines changed: 56 additions & 29 deletions

File tree

‎research/decimal/02-design.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,7 @@ file hold an invalid child. `vortex.widen` stays in memory only until the mainta
163163
write it to files (decided, D18).
164164

165165
Until then, a writer that meets a `Widen` writes its executed canonical array (proposed, D26,
166-
deferred to CD4). This also holds later for writers whose editions do not permit `vortex.widen`.
166+
deferred to CD6). This also holds later for writers whose editions do not permit `vortex.widen`.
167167
Decimal `Widen` appears in ordinary pipelines, through widening casts and old files, so IPC or a
168168
writer without compression would fail without it. A plugin cannot return the wire form of another
169169
encoding, so the fallback lives in the serializer. The code documents it as a workaround, which
@@ -541,7 +541,7 @@ Writers produce forms that old readers understand. The leaf becomes `vortex.deci
541541
canonical width. DBP becomes DecimalByteParts v1, which the first core edition already contains,
542542
or v2 where an edition permits it. `Widen` is written only where an edition permits
543543
`vortex.widen`, and the compressor produces it only then. Elsewhere the serializer writes the
544-
canonical array of a `Widen` (proposed, D26, deferred to CD4). `vortex.widen` and DBP v2 enter one
544+
canonical array of a `Widen` (proposed, D26, deferred to CD6). `vortex.widen` and DBP v2 enter one
545545
preview edition together, and later `core` together (decided, D18). Constant, dictionary and
546546
run-end arrays of decimals keep their current wire forms.
547547

‎research/decimal/03-roadmap.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -122,9 +122,10 @@ uses `DecimalType`, which already has 128-bit and 256-bit widths.
122122

123123
## Independent changes
124124

125-
These changes are small and independent, and they can land in any order. On 2026-10-23, none of
126-
them is on `develop`. [Part 4](04-implementation.md) splits these changes and the canonical form
127-
into PRs, with one brief for each PR.
125+
These changes are small and independent, and they can land in any order. On 2026-10-29, strict
126+
enum decoding (#10755) and WD1 to WD4 of the integer `Widen` are on `develop`, and N1 is not.
127+
[Part 4](04-implementation.md) splits these changes and the canonical form into PRs, with one
128+
brief for each PR.
128129

129130
1. **Strict enum decoding.** The decimal `values_type` decodes with `DecimalType::try_from`, and
130131
the protobuf `PType` decodes with `PType::try_from`. Both fail on an unknown value instead of a

‎research/decimal/04-implementation.md‎

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -66,19 +66,21 @@ planning review on 2026-10-23.
6666
| FX1 | Strict decoding of stored `DecimalType` and `PType` values, in three commits | None | Medium |
6767
| FX2 | Dropped. #10798 documents the cast-rule contract instead (see the FX2 brief) | None | None |
6868
| FX3 | Dictionary `fill_null` no longer truncates the fill code | None | Small |
69-
| WD1 | In-memory integer `Widen`, with `encode`, the deprecated `encode_values`, the serialization workaround and a flatten rule | None | Large |
69+
| WD1 | In-memory integer `Widen`, with `encode`, the deprecated `encode_values` and a flatten rule | None | Large |
7070
| WD2 | Benchmarks for compute on integer `Widen` | WD1 | Medium |
7171
| WD3 | Aggregates run on the child | WD2 | Medium |
7272
| WD4 | Arithmetic, `between`, `zip` and casts run on the child | WD3 | Large |
7373
| WD5 | Structural encodings move above `Widen`, and dictionary codes use the child | WD4, FX3 | Medium |
74-
| WD6 | The existing narrowing call sites move to `encode_values` | WD1. It merges after WD5 (D35) | Medium |
74+
| WD6 | The existing narrowing call sites move to `encode_values` | WD1 (D35) | Medium |
7575
| CD1 | Decimal benchmarks | None | Medium |
7676
| CD2 | Compat fixtures for old decimal forms | None | Medium |
7777
| CD3 | One width function, imports that check and copy, and the shared helpers | None | Medium |
78-
| CD4 | Decimal `Widen` in memory | WD1, WD3, WD4, CD1, CD3 | Large |
78+
| CD3b | One scan for the range of valid decimal values (`valid_range`), shared by CD4, CD6 and CD7 | CD3 | Small |
79+
| CD4p | Decimal `Interleave` execution, so that CD4 can test its fallbacks | None | Medium |
80+
| CD4 | Decimal `Widen` in memory | WD1, WD3, WD4, CD1, CD3, CD3b, CD4p | Large |
7981
| CD5 | DecimalByteParts decodes to the canonical width | CD1, CD3, released CD2 fixtures | Medium |
80-
| CD6 | The `vortex.decimal` read plugin converts old nodes | FX1, CD3, CD4, released CD2 fixtures | Medium |
81-
| CD7 | One DecimalByteParts scheme on the canonical array | WD1, CD1 | Medium |
82+
| CD6 | The `vortex.decimal` read plugin converts old nodes, and decides D26 | FX1, CD3, CD3b, CD4, released CD2 fixtures | Medium |
83+
| CD7 | One DecimalByteParts scheme on the canonical array. It merges right after CD5 | WD1, CD1, CD3b | Medium |
8284
| CD8a | Every decimal array has the canonical width | CD3, CD5, CD6, CD7 | Large |
8385
| CD8b | `DecimalArray::try_new` checks every valid value, with an `ExecutionCtx` | CD8a | Large |
8486
| CD8c | Remove the code for mixed decimal widths | CD8a | Medium |
@@ -101,7 +103,8 @@ brief or a decision changes.
101103

102104
Each solid arrow is a hard dependency. Some arrows also follow from other arrows. The graph keeps
103105
them, so that the direct needs of each PR show. The dashed arrows are order rules, not code
104-
dependencies: the release gate of the compat fixtures (D31), and WD6 merging after WD5 (D35).
106+
dependencies: the release gate of the compat fixtures (D31), and CD7 merging right after CD5, so
107+
that the old scheme does not split decimals from their storage width for long (#10905).
105108

106109
```mermaid
107110
flowchart TD
@@ -122,6 +125,8 @@ flowchart TD
122125
CD2["CD2 Compat fixtures"]
123126
REL(["Release with CD2"])
124127
CD3["CD3 Width and imports"]
128+
CD3b["CD3b Valid range scan"]
129+
CD4p["CD4p Decimal Interleave"]
125130
CD4["CD4 Decimal Widen"]
126131
CD5["CD5 DBP decoding"]
127132
CD6["CD6 Read plugin"]
@@ -139,12 +144,17 @@ flowchart TD
139144
WD1 --> WD2 --> WD3 --> WD4 --> WD5
140145
FX3 --> WD5
141146
WD1 --> WD6
142-
WD5 -.-> WD6
143147
CD2 --> REL
144148
REL -.-> CD5
145149
REL -.-> CD6
146150
CD1 --> CD4
147151
CD3 --> CD4
152+
CD3 --> CD3b
153+
CD3b --> CD4
154+
CD4p --> CD4
155+
CD3b --> CD6
156+
CD3b --> CD7
157+
CD5 -.-> CD7
148158
WD1 --> CD4
149159
WD3 --> CD4
150160
WD4 --> CD4
@@ -178,10 +188,10 @@ files. The column "Shared files" names the pairs that do not, so run those one a
178188

179189
| Wave | PRs | Note | Shared files |
180190
| --- | --- | --- | --- |
181-
| 1 | FX1, FX3, WD1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts |
182-
| 2 | WD2, WD6, CD7, CD5 | WD6 merges after WD5 (D35). CD7 waits for CD1. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. WD6 and CD7: the golden snapshots of `vortex-btrblocks` |
191+
| 1 | FX1, FX3, WD1, CD1, CD2, CD3, CD4p | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts |
192+
| 2 | WD2, WD6, CD3b, CD7, CD5 | WD6 needs only WD1 (D35). CD7 waits for CD1 and CD3b, and it merges right after CD5. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. WD6 and CD7: the golden snapshots of `vortex-btrblocks` |
183193
| 3 | WD3, then WD4 | One stack. Read the CodSpeed reports with the rule in the NR brief ("CodSpeed on stacked PRs") | None |
184-
| 4 | WD5, CD4 | FX3 is merged (`dcb4f50ae2`) | Both edit `vortex-array/src/arrays/widen/rules.rs` |
194+
| 4 | WD5, CD4 | FX3 is merged (`dcb4f50ae2`). Settle the mechanism question of #10748 before WD5, so that WD5 and phase 4a of the extension notes can share it | Both edit `vortex-array/src/arrays/widen/rules.rs` |
185195
| 5 | CD6 | Waits for FX1, CD4 and the release with CD2 | None |
186196
| 6 | CD8a | The last in-memory step that the wire change needs | None |
187197
| 7 | CD8b, CD8c, WR1 | Each needs only CD8a | CD8b adds a `ctx` parameter to `DecimalArray::new` and `try_new`, which CD8c and WR1 call. CD8b and CD8c both edit `vortex-array/src/builders/decimal.rs` and the same tests. Land CD8b first, or rebase the others on it |
@@ -192,7 +202,7 @@ publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wa
192202

193203
## Rules for every PR
194204

195-
- No PR adds temporary code (D11). The serialization workaround of D26, if CD4 adds it, is the one
205+
- No PR adds temporary code (D11). The serialization workaround of D26, if CD6 adds it, is the one
196206
documented exception. It stays until `vortex.widen` is in `core`.
197207
- Benchmarks land before the changes that they cover (D19): WD2 before WD3 to WD5, and CD1 before
198208
CD4, CD5 and CD7. The dependency table lists these as hard dependencies.
@@ -220,11 +230,12 @@ PRs, and each one has its own issue. Two are closed.
220230
| Issue | Problem |
221231
| --- | --- |
222232
| #10748 | Generic structural encodings always sit above other encodings. WD5 does this for `Widen` only |
223-
| #10749 | Decimal `scalar_at` and `min_max` panic on a stored value outside the precision, which a file read can produce |
233+
| #10749 | Decimal `scalar_at` and `min_max` panic on a stored value outside the precision, which a file read can produce. #10767 fixes both and is in review |
224234
| #10750 | Closed as not planned. The push-down cast rules return no wrong result, and #10798 documents them as a valid form |
225235
| #10751 | Closed by #10757. The FoR scheme now declares and refines its bit-packing child |
226236
| #10754 | The compat harness: no CI job runs `--mode all`, one failing fixture stops the whole check, and the README describes `--git-ref` wrongly |
227237
| #10756 | Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read |
238+
| #10908 | `ZipExecuteAdaptor` dispatches only on the `if_true` branch, so a `Widen` in the `if_false` branch of `zip` does not run on the child. Found in the review of WD4 |
228239

229240
The stale versioning docs needed no issue, because #10747 fixes them.
230241

‎research/decimal/agents/briefs/03-canonical-decimal.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1137,8 +1137,9 @@ than the canonical width returns an error, because the btrblocks scheme never wr
11371137
- Goal: An old `vortex.decimal` node has a stored width that differs from the canonical width of its
11381138
dtype. The reader converts it into the canonical array or a decimal `Widen`. After this PR no
11391139
read produces a decimal leaf with a non-canonical width.
1140-
- Depends on: FX1, CD3, CD4, and the published CD2 fixtures (D31). Soft: CD5. Blocks: CD8a.
1141-
- Decisions: D7, D9, D21, D14, D10, D24, D31, D37.
1140+
- Depends on: FX1, CD3, CD3b (`valid_range`, #10939), CD4, and the published CD2 fixtures (D31).
1141+
Soft: CD5. Blocks: CD8a.
1142+
- Decisions: D7, D9, D21, D14, D10, D24, D26, D31, D37.
11421143
- Branch: `ct/decimal-read-plugin`, based on `develop` after FX1, CD3 and CD4 merge, or stacked on
11431144
CD4.
11441145
- Scope:
@@ -1147,6 +1148,11 @@ than the canonical width returns an error, because the btrblocks scheme never wr
11471148
- Its `is_supported_encoding` override and its registration.
11481149
- A `Decimal` VTable `deserialize` that rejects a non-canonical width.
11491150
- Unit tests with hand-built old nodes.
1151+
- The choice of D26. CD4 (#10946) added no serialization lowering, because the default writer
1152+
executes a `Widen` through its pipeline. After this PR, reads of old files return a `Widen`, so
1153+
IPC and writers without a compressor can meet one. Choose between a lowering specific to
1154+
`Widen`, a generic lowering for arrays that cannot be written, and no lowering, and put that
1155+
choice to the maintainer before writing it.
11501156
- Out of scope:
11511157
- Stored statistics in plugins (rejected, D9).
11521158
- DecimalByteParts (CD5) and strict construction (CD8a).
@@ -1288,8 +1294,8 @@ Review focus:
12881294

12891295
Size: medium. One new module and one registration change.
12901296

1291-
Open questions: None. A read can now return an in-memory `Widen`. Until WR1, a writer handles it as
1292-
CD4 decided for D26.
1297+
Open questions: D26, which this PR decides (see Scope). Until WR1, a writer handles a `Widen` as
1298+
this PR decides.
12931299

12941300
### CD7: One DecimalByteParts scheme on the canonical array
12951301

‎research/decimal/agents/briefs/04-wire.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ The section [Lockstep](#lockstep) gives the reasons.
2525
(D32): the child `PType` for an integer parent, or the child precision for a decimal parent. The
2626
reader builds and checks the child dtype before it decodes the child, so invalid metadata is an
2727
error and never a panic. When the selected editions do not permit `vortex.widen`, the writer
28-
still writes the canonical array (D26).
28+
handles a `Widen` as CD6 decides for D26.
2929
- Depends on: WD1 (encoding, `validate_dtypes`), CD3
3030
(`DecimalDType::canonical_values_type()`), CD4 (decimal rules in `validate_dtypes`, decimal
3131
`Widen`). Soft: FX1, WD3, WD4, WD5, CD6. Blocks: WR2, WR3.
@@ -104,7 +104,7 @@ pub(super) fn child_dtype(parent: &DType, metadata: &[u8]) -> VortexResult<DType
104104
`Widen` always serializes.
105105
- The lowering of D26 also covers a context that does not permit `vortex.widen`. Today the tree
106106
builder has no context, and `try_write_node` rejects a non-permitted ID with "not permitted by
107-
ctx" (`vortex-array/src/serde.rs:251-256`). So the tree builder gets the permitted IDs. CD4
107+
ctx" (`vortex-array/src/serde.rs:251-256`). So the tree builder gets the permitted IDs. CD6
108108
decides whether a lowering exists (D26). If it does, WR1 limits it to a context that does not
109109
permit `vortex.widen`. If it does not, WR1 adds the lowering for that context. The doc keeps the
110110
workaround note and its end at LT1.
@@ -142,7 +142,7 @@ Steps:
142142
contract. Add unit tests for both functions.
143143
2. Replace the `deserialize` stub and change `serialize` in `vtable.rs`. Add `tests/wire.rs`.
144144
3. Pass the permitted IDs to the tree builder, and lower a non-permitted `Widen` in
145-
`ArraySerializationTree::try_new`, at the lowering site of CD4 if CD4 adds one (D26).
145+
`ArraySerializationTree::try_new`, at the lowering site of CD6 if CD6 adds one (D26).
146146
4. Add the `vortex-file` round-trip test.
147147

148148
Tests:

0 commit comments

Comments
 (0)