We ran OpenMasq on Presidio's evaluation corpus, and Presidio on ours — results, method, and where we lose

[openmasq-devteam]

Hi all — disclosure first: I'm the developer of OpenMasq (Apache-2.0, https://github.com/openmasq/openmasq), an on-device redaction engine and desktop chat that swaps detected values for believable substitutes before a prompt reaches an LLM, and restores them in the reply.

We built our detection on top of the problem Presidio defined, so the honest way to measure it was against Presidio itself, both ways. Sharing the results here because this is the community most able to tell us where the method is wrong.

Method. Same scorer for both engines: a truth counts as found when ≥ 60 % of its significant tokens were replaced; false positives are counted by overlap; a detector that finds a value we didn't annotate is not charged for it. Presidio is the default AnalyzerEngine (presidio-analyzer 2.2.364, spaCy 3.8.16, en_core_web_lg), replayed from bench/external/run_presidio.py. Everything is in packages/redact/bench and replays.

On Presidio's own evaluation set (presidio-research, 2 523 truths, English, template + faker):

category	truths	OpenMasq (rules + local NER)	Presidio default
NAME	857	98 %	87 %
ADDRESS	598	38 %	17 %
CITY	411	68 %	58 %
ORG	250	73 %	22 %
CARD	136	100 %	97 %
PHONE	92	57 %	63 %
ID	21	95 %	100 %
EMAIL · URL · IBAN · IP	121	100 %	100 %
global	2 523	74 % · 118 FP	58 % · 196 FP

Where Presidio wins: phone numbers and the ID formats of that set. Where our deterministic pipeline alone (no model) sits: 31 %, with 6 false positives — the recall is the NER's, the precision is the rules'.

On our corpus (18 document families, 14 languages, real layouts, OCR damage, 3 357 truths, all synthetic identities): 95 % · 256 FP for us, 46 % · 847 FP for Presidio default — which is expected, Presidio is tuned for English and we built the corpus around our failure modes; Presidio's 41 % on our English-only cases is the number that says the gap is layouts, not language.

What we'd like from you. If the scorer is unfair to Presidio (recognizer configuration, the 60 % token rule, the FP accounting), say so and we'll change it and republish. If someone wants to add a column for another engine or a tuned Presidio config, the bench takes a small adapter. And if any of the ADDRESS/ORG cases look like annotation errors on our side, please point at them.

Thanks for the corpus — it was the only honest yardstick we had.

[omri374]

Hi, thanks for posting here! Presidio in its vanilla form is optimized for speed, not accuracy. I would suggest to also experiment with a gliner or transformers models to get a more accurate result.