Summary
Training normalizes every row of text before fitting the vectorizer, but no
serving path applies that same normalization before vectorizer.transform. The
shared normalizer is referenced exactly once in the API — to build a cache key —
while the text actually handed to the model is the raw input. The result is a
systematic train–serve skew on the project's core feature path, and nothing in
the artifacts or the API surface would reveal it.
Problems
- The served text is not the text the model was trained on. Training applies
the shared normalizer — NFKC folding, homoglyph mapping, zero-width stripping,
spaced-letter de-obfuscation — and fits the vocabulary on that form. Prediction
transforms the raw string. Precisely the obfuscation the normalizer exists to
defeat (Cyrillic look-alikes, zero-width joiners, f r e e spacing) reaches the
vectorizer intact, drops out of vocabulary, and quietly degrades detection.
- The cache is keyed on a different string than the one scored. The cache key
is derived from the normalized text while the prediction is computed from the
raw text, so two distinct inputs that normalize identically share a single entry
even though scoring them fresh would not agree.
- Every entry point preprocesses differently. Bulk scoring, mailbox scanning,
and the translation step each decide independently what the model sees, so the
same message can be classified differently depending on how it arrived.
- Nothing ties preprocessing to the model. Provenance records checksums for
the model, vectorizer and encoder, but not how text must be prepared for them,
so a future normalizer change silently desynchronizes from artifacts trained
under the old rules.
- A reload does not reach every path. Mailbox scanning reads model objects
pinned on the application at startup and the URL classifier is loaded once as
module state, so both can keep serving pre-reload objects after a hot swap — an
independent second way for paths to disagree.
Proposed work
- Establish one canonical text-preparation contract in a neutral module that both
training and every inference path call, so the string reaching the vectorizer is
produced by the same code in both regimes.
- Route the primary prediction path through it, and key the response cache on the
same prepared text that is actually scored.
- Extend it to the remaining inference paths so a result no longer depends on
which door the message came through.
- Bind preprocessing identity to the model release: record which contract version
an artifact set was trained under, expose it with the existing provenance, and
detect at load time when the serving contract disagrees with the trained one.
- Bring the remaining serving objects under the versioned snapshot so a reload
updates every path at once.
Suggested PR split
Each part stands alone and leaves prediction fully functional.
Part A — Shared preparation contract on the primary path
Extract the canonical preparation step into a neutral module used by training, and
apply it in the main prediction path before vectorization. Derive the response
cache key from the same prepared text that is scored, closing the gap where two
inputs collide on one entry.
Part B — Same contract for bulk and mailbox scanning
Apply the contract to bulk scoring and to mailbox scan classification so batch and
inbox results agree with the primary path for identical input.
Part C — Preprocessing identity in model provenance
Record the preparation contract version alongside the trained artifacts, surface it
with the existing model provenance, and flag at load time when a served artifact
set was trained under a different contract than the one in force.
Part D — Reload coverage for the remaining serving objects
Bring the mailbox-scan model references and the URL classifier under the versioned
serving snapshot so a hot reload swaps every prediction path atomically instead of
leaving some on startup state.
Acceptance criteria
- Obfuscated variants that reduce to the same canonical text are classified
consistently with their plain equivalent on the primary path.
- Bulk scoring and mailbox scanning return the same classification as the primary
path for the same message.
- A cached response corresponds to the text that was actually scored.
- Provenance reports the preparation contract the served artifacts were trained
under, and a mismatch is detectable at load time rather than silently served.
- After a reload, every prediction path serves the newly loaded objects.
- Regression coverage for cross-path parity, obfuscated input, cache-key
correspondence, provenance reporting, and reload propagation.
Summary
Training normalizes every row of text before fitting the vectorizer, but no
serving path applies that same normalization before
vectorizer.transform. Theshared normalizer is referenced exactly once in the API — to build a cache key —
while the text actually handed to the model is the raw input. The result is a
systematic train–serve skew on the project's core feature path, and nothing in
the artifacts or the API surface would reveal it.
Problems
the shared normalizer — NFKC folding, homoglyph mapping, zero-width stripping,
spaced-letter de-obfuscation — and fits the vocabulary on that form. Prediction
transforms the raw string. Precisely the obfuscation the normalizer exists to
defeat (Cyrillic look-alikes, zero-width joiners,
f r e espacing) reaches thevectorizer intact, drops out of vocabulary, and quietly degrades detection.
is derived from the normalized text while the prediction is computed from the
raw text, so two distinct inputs that normalize identically share a single entry
even though scoring them fresh would not agree.
and the translation step each decide independently what the model sees, so the
same message can be classified differently depending on how it arrived.
the model, vectorizer and encoder, but not how text must be prepared for them,
so a future normalizer change silently desynchronizes from artifacts trained
under the old rules.
pinned on the application at startup and the URL classifier is loaded once as
module state, so both can keep serving pre-reload objects after a hot swap — an
independent second way for paths to disagree.
Proposed work
training and every inference path call, so the string reaching the vectorizer is
produced by the same code in both regimes.
same prepared text that is actually scored.
which door the message came through.
an artifact set was trained under, expose it with the existing provenance, and
detect at load time when the serving contract disagrees with the trained one.
updates every path at once.
Suggested PR split
Each part stands alone and leaves prediction fully functional.
Part A — Shared preparation contract on the primary path
Extract the canonical preparation step into a neutral module used by training, and
apply it in the main prediction path before vectorization. Derive the response
cache key from the same prepared text that is scored, closing the gap where two
inputs collide on one entry.
Part B — Same contract for bulk and mailbox scanning
Apply the contract to bulk scoring and to mailbox scan classification so batch and
inbox results agree with the primary path for identical input.
Part C — Preprocessing identity in model provenance
Record the preparation contract version alongside the trained artifacts, surface it
with the existing model provenance, and flag at load time when a served artifact
set was trained under a different contract than the one in force.
Part D — Reload coverage for the remaining serving objects
Bring the mailbox-scan model references and the URL classifier under the versioned
serving snapshot so a hot reload swaps every prediction path atomically instead of
leaving some on startup state.
Acceptance criteria
consistently with their plain equivalent on the primary path.
path for the same message.
under, and a mismatch is detectable at load time rather than silently served.
correspondence, provenance reporting, and reload propagation.