You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0c062ea
Browse filesBrowse the repository at this point in the historyBrowse files
Update the Narrow and decimal research for the first PRs
Records why FX2 was dropped (#10798 documents the cast-rule contract), the merged CD2 fixtures, the error types without categories (#10080), and the FoR scheme change (#10757). The tracking issues #10701 and #10733 now hold the state of each PR, so part 4 points to them instead of a status table.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Copy file name to clipboardExpand all lines: research/decimal/01-current-state.md
-1Lines changed: 0 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,7 +34,6 @@ gives file and line references for each one.
34
34
| The scalar path and the array path disagree | A decimal scalar casts to `f32` and `f16`. The array cast supports only `f64`, so an array cast to `f32` fails with "No CastKernel". A throwaway test confirmed the `f32` case |
35
35
|`Decimal` duplicates `Primitive`| It has its own canonical variant (51 dispatch sites), builder, compare, arithmetic and aggregates. Decimal-specific files hold about 7,600 non-test lines |
36
36
| DecimalByteParts carries the decimal dtype | It splits wide integers into 64-bit parts, but only decimals can use it |
37
-
| The DecimalByteParts cast rule skips a nullability check | For a cast that changes only the nullability, the rule casts the most significant part. It does not first make sure that the change is safe without execution, as other cast rules do with `Validity::trivially_cast_nullability`|
38
37
| Unknown enum values decode silently | An unknown `values_type` decodes as `I8`. An unknown protobuf `PType` decodes as `U8`, and Variant metadata stores protobuf dtypes in files |
39
38
| Shared code has decimal special cases | Bloom filters, `IN` lists, size statistics, sparse arrays, run-end decoding and the row codec each have a decimal path |
40
39
| No 128-bit integer columns | DuckDB `HUGEINT` cannot be imported, although decimal storage already holds 128-bit values |
| FX2 |Dropped. #10798 documents the cast-rule contract instead (see the FX2 brief) | None |None|
61
66
| FX3 | Dictionary `fill_null` no longer truncates the fill code | None | Small |
62
67
| NR1 | In-memory integer `Narrow`, with `encode`, the deprecated `encode_values`, the serialization workaround and a flatten rule | None | Large |
63
68
| NR2 | Benchmarks for compute on integer `Narrow`| NR1 | Medium |
@@ -83,18 +88,12 @@ Later work: LT1 puts both formats into `core` and decides the end of the seriali
83
88
workaround. LT2 is the max pass in DecimalByteParts decoding (D16). LT3 is the pushdown of sum,
84
89
min and max into the MSP (D19). LT2 and LT3 wait until benchmarks show that they are needed.
85
90
86
-
## Status
91
+
## Progress
87
92
88
-
The first wave opened on 2026-10-23 as draft PRs, all based on `develop` at `3974ee069c`:
89
-
90
-
| ID | PR | Note |
91
-
| --- | --- | --- |
92
-
| FX1 |#10755| Three commits, as planned. Its tests use a new `test_harness::assert_serde_error` helper |
93
-
| FX2 |#10746| As planned |
94
-
| FX3 |#10745| Fixes #10623. Non-nullable codes keep the fast path, because they need no fill code |
95
-
| CD1 |#10752| The compress bench enables the default `core` edition, so the decimal scheme uses its v1 variant |
96
-
| CD2 |#10753| Adds `fixtures_build_canonical_decimal_widths`, and two columns at precisions 1 and 3 |
97
-
| None |#10747| The versioning docs describe the scheme variants that #10076 added |
93
+
The collapsed PR lists in [#10701](https://github.com/vortex-data/vortex/issues/10701) and
94
+
[#10733](https://github.com/vortex-data/vortex/issues/10733) track the state of each PR: its link,
95
+
whether it merged, and short notes. Agents update those lists. This branch changes only when a
96
+
brief or a decision changes.
98
97
99
98
## Graph
100
99
@@ -106,7 +105,6 @@ dependencies: the release gate of the compat fixtures (D31), and NR6 merging aft
106
105
flowchart TD
107
106
subgraph FIX["Independent fixes"]
108
107
FX1["FX1 Strict decoding"]
109
-
FX2["FX2 DBP nullability cast"]
110
108
FX3["FX3 Dict fill code"]
111
109
end
112
110
subgraph NR["Integer Narrow"]
@@ -170,8 +168,6 @@ flowchart TD
170
168
NR5 --> WR3
171
169
```
172
170
173
-
FX2 has no edge. It can land at any time.
174
-
175
171
## Order of work
176
172
177
173
A PR can start when its dependencies have merged. The waves group the PRs whose dependencies merge
@@ -180,7 +176,7 @@ files. The column "Shared files" names the pairs that do not, so run those one a
180
176
181
177
| Wave | PRs | Note | Shared files |
182
178
| --- | --- | --- | --- |
183
-
| 1 | FX1, FX2, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts |
179
+
| 1 | FX1, FX3, NR1, CD1, CD2, CD3 | No dependencies. The next regular release after CD2 publishes the fixtures | CD2 calls `smallest_decimal_value_type`, which CD3 removes. The second of the two to merge adapts |
184
180
| 2 | NR2, NR6, CD7, CD5 | NR6 merges after NR5 (D35). CD7 waits for CD1. CD5 waits for the release with CD2 | CD5 and CD7: the crate doc in `encodings/decimal-byte-parts/src/lib.rs`, the tests in `assemble.rs`, and `benches/dbp_assemble.rs`. CD7 also changes the signature of `split_decimal`. NR6 and CD7: the golden snapshots of `vortex-btrblocks`|
185
181
| 3 | NR3, then NR4 | One stack. Merge NR1 and NR2 before reading the CodSpeed reports of NR3 and NR4 | None |
186
182
| 4 | NR5, CD4 | NR5 waits for FX3 | Both edit `vortex-array/src/arrays/narrow/rules.rs`|
@@ -217,14 +213,14 @@ publishes the CD2 fixtures also gates CD5 and CD6, so CD2 is worth landing in wa
217
213
## Follow-up issues
218
214
219
215
These problems came up during planning and the first wave of PRs. They are outside the planned
220
-
PRs, and each one has its own issue.
216
+
PRs, and each one has its own issue. Two are closed.
221
217
222
218
| Issue | Problem |
223
219
| --- | --- |
224
220
|#10748| Generic structural encodings always sit above other encodings. NR5 does this for `Narrow` only |
225
221
|#10749| Decimal `scalar_at` and `min_max` panic on a stored value outside the precision, which a file read can produce |
226
-
|#10750|Seven cast rules (RunEnd, ZigZag, FoR, RLE, ALP, ALP-RD, DateTimeParts) push nullability casts without the guard that FX2 adds to DecimalByteParts|
227
-
|#10751| The FoR scheme bit-packs its child outside the allowed schemes, so its child skips the edition filter|
222
+
|#10750|Closed as not planned. The push-down cast rules return no wrong result, and #10798 documents them as a valid form|
223
+
|#10751|Closed by #10757. The FoR scheme now declares and refines its bit-packing child|
228
224
|#10754| The compat harness: no CI job runs `--mode all`, one failing fixture stops the whole check, and the README describes `--git-ref` wrongly |
229
225
|#10756| Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read |
0 commit comments