Count distinct values for all columns of a numeric array at once in modality detection - #1255
Open
Innixma wants to merge 4 commits into
Open
Count distinct values for all columns of a numeric array at once in modality detection#1255Innixma wants to merge 4 commits into
Innixma wants to merge 4 commits into
Conversation
…odality detection A numeric array needs no per-column parsing: only the distinct-value count decides, so it is counted block-wise (sort, then adjacent unequal pairs, NaN once) instead of building a Series and calling nunique per column. The per-column early exit is mirrored: columns whose first 1024 rows already clear every threshold keep that count. Object arrays are unchanged. A randomized test checks the decisions against the per-column path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
`_is_numeric_pandas_series` walked every value of an object column in Python (or coerced the whole column) to decide whether it is numeric. A frame with a single non-numeric column arrives as one object array, so every numeric column paid for that. `pd.api.types.infer_dtype` answers the common case in C: a kind whose non-missing values are all numbers settles the column as numeric; string and mixed kinds still take the existing path, since a spelled-out number counts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
`coerce_nullable_dtypes_to_numpy` evaluated two pandas dtype predicates per column; a wide frame has thousands of columns and a handful of dtypes, so the predicates now run once per distinct dtype and the columns are selected by membership. Same columns, same order. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
|
|
…-column test The two existing numeric checks disagree on complex values (`float()` rejects them, `pd.to_numeric` accepts them), so the case's answer depends on the pandas version; the shortcut never settles it, so it is not part of what the test checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
Innixma
added this pull request to stack #1258
September 10, 2026 00:23
Contributor
|
@arthur-priorlabs since you worked quite a bit on preprocessing? |
arthur-priorlabs
approved these changes
Sep 10, 2026
Comment on lines
+250
to
+261
| def _numeric_n_unique_per_column( | ||
| X: np.ndarray, *, decided_at: int | ||
| ) -> np.ndarray | None: | ||
| """Distinct values per column of a numeric or bool array, NaN counted as a value. | ||
|
|
||
| `None` for anything else (an object array is parsed column by column). Mirrors | ||
| the per-column early exit: a column whose first `_EARLY_EXIT_PREFIX_ROWS` rows | ||
| already hold `decided_at` distinct values keeps that prefix count, which lands | ||
| in the same bucket as the full count; only the other columns are counted in | ||
| full. | ||
| """ | ||
| if not isinstance(X, np.ndarray) or X.ndim != 2 or X.dtype.kind not in "biuf": |
Contributor
There was a problem hiding this comment.
Why are we bothering to check the number of distinct values in a bool array?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
detect_feature_modalitiesbuilt apd.Seriesand callednuniquefor every column. When the array is numeric or bool, which it is whenever the input has no object columns, only the distinct-value count decides a column's modality, so it is now counted for all columns at once: sort each column, count adjacent unequal pairs, count NaN once.-0.0equals0.0andinfequalsinf, as undernunique(dropna=False).The per-column early exit is kept: the first 1024 rows are counted for every column, columns that already clear every threshold keep that count (it lands in the same bucket as the full count), and only the remaining columns are counted in full, so tall data does not pay for a full sort of high-cardinality columns. Object arrays go through the per-column path as before, and the numeric decision (constant / categorical / numerical) moved into one shared helper used by both paths.
On a 22k-column float frame with a few hundred rows this takes modality detection from about 0.4 s to 0.02 s and roughly halves
fit. A randomized test compares the resulting schema with the per-column decisions on float, float32, int and bool arrays with NaN, signed zeros, infinities, all-missing and constant columns, row counts on both sides of the prefix, random thresholds and random declared-categorical indices; a second test pins the count againstnunique(dropna=False).Two smaller items in the same spirit:
_is_numeric_pandas_serieswalked every value of an object column in Python (or coerced the whole column) to decide whether it is numeric; a frame with a single non-numeric column arrives as one object array, so every numeric column paid for that.pd.api.types.infer_dtypenow settles the common case in C: a kind whose non-missing values are all numbers (integer,floating,mixed-integer-float,boolean,decimal,empty) makes the column numeric;stringandmixedkinds still take the existing path, since a spelled-out number counts. A parametrized test checks the shortcut against the value walk on object columns of floats, ints, bools, decimals, bytes, spelled numbers, words, mixed values, complex numbers and timestamps.coerce_nullable_dtypes_to_numpyevaluated two pandas dtype predicates per column; they now run once per distinct dtype and the columns are selected by membership. Same columns, same order, with a test over eleven dtypes.🤖 Generated with Claude Code
https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi