Score predictions on the text the model was trained on

[pavsoss]

Summary

Training prepares text with the shared normalizer before fitting the TF-IDF vocabulary, but the prediction path handed the vectorizer the raw input — the normalizer was reached for only when building a cache key. The obfuscations it exists to undo (Cyrillic look-alikes, zero-width joiners, f r e e spacing) therefore survived into the vectorizer, fell out of vocabulary, and weakened detection on exactly the messages most likely to be evasive.

Part of #1037.

Changes

Both regimes now go through one text-preparation contract. Training calls it, the prediction path calls it before vectorizing, and the response cache is keyed on the same prepared string that is actually scored — previously two different inputs reducing to the same canonical form could share one cache entry while being scored differently.

Preparation runs after translation, so a non-English message is canonicalised in the language the model was trained on.

Test plan

• Canonical-form coverage: zero-width stripping, homoglyph folding, space de-obfuscation, whitespace collapsing, idempotence, and pass-through of empty and non-string input.
• A check that training and inference resolve to the same callable rather than two copies that can drift apart.

[vercel]

@pavsoss is attempting to deploy a commit to the Aditya Sharma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[pavsoss]

@Userunknown84