Skip to content

Encode numeric categorical columns with a numpy ordinal encoder - #1263

Open
Innixma wants to merge 3 commits into
modality-detection-vectorizedfrom
fast-ordinal-encoder
Open

Encode numeric categorical columns with a numpy ordinal encoder#1263
Innixma wants to merge 3 commits into
modality-detection-vectorizedfrom
fast-ordinal-encoder

Conversation

@Innixma

@Innixma Innixma commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1255.

EncodeCategoricalFeaturesStep's ordinal path wrapped sklearn's OrdinalEncoder. On a small table almost all of a call is sklearn's input validation rather than the encoding, and the step fits once per ensemble member, so on a categorical-heavy table of a few hundred rows the step was the largest single CPU cost of fit. Profiled on a 598 x 31 table with 25 categorical columns and 88 ensemble preprocessing pipelines: 0.54 s of a 0.85 s fit, about 6 ms per pipeline, of which the encoder's own arithmetic is well under a millisecond.

Numeric arrays now go through NumericOrdinalEncoder, a OneToOneFeatureMixin transformer that does one np.unique and one np.searchsorted per column and validates nothing. It gives the same output as OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=np.nan): sorted distinct values become 0..k-1 as float64, NaN stays NaN, a value unseen at fit becomes NaN, and categories_ lists NaN last for a column that had NaN at fit so the per-column category count that the ordinal_shuffled permutation reads is unchanged. Object arrays keep sklearn's encoder. The ColumnTransformer wrapper, output names, schema carry-over and the one-hot path are untouched.

On the table above the step goes from 3.97 ms to 1.20 ms per fit_transform and from 1.59 ms to 0.27 ms per transform; preprocessed outputs are identical. Tests compare the new encoder with sklearn's on plain, missing-value, unknown-at-transform, float32, all-NaN and constant columns, check the encoder choice by dtype, and check the whole step's output, including the shuffled variant, against the step with sklearn's encoder inside. The four test_sklearn_estimator_checks failures for KDITransformerWithNaN in this environment are present on the base branch as well.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi

Innixma and others added 2 commits September 11, 2026 22:11
The ordinal step ran sklearn's OrdinalEncoder, which spends most of a call on input
validation; on a small table that dwarfed the encoding itself, and the step fits once
per ensemble member. Numeric arrays now go through a numpy encoder with the same
output, categories and NaN and unknown-value handling. Object arrays keep sklearn's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
@Innixma
Innixma added this pull request to stack #1264 September 11, 2026 22:14

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9be1cba. Configure here.

Comment thread changelog/1263.changed.md Outdated
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi
@Innixma
Innixma removed this pull request from stack #1264 September 12, 2026 04:47
@Innixma
Innixma added this pull request to stack #1266 September 12, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant