CLIP semantic hashing via Coop HMA

[sahilx13]

Related: UMass ROOST evaluation, UMass tx-extension-clip (CLIPFloatSignal), Coop HMA docs (docs/integrations/hma.md)

Summary

We propose enabling CLIP-based semantic image matching inside Coop’s existing HMA service, alongside today’s PDQ (photos) and VideoMD5 (videos). Inspiration and a working reference stack come from the UMass ROOST evaluation: CLIP embeddings are stored in HMA banks and used to match new content the same way PDQ banks work today.

This is not a new Coop microservice. HMA already hashes at ingest (/h/hash), matches against banks (/m/lookup), and powers IMAGE_SIMILARITY_MATCH / IMAGE_SIMILARITY_DOES_NOT_MATCH. Coop’s server path is largely hash-type agnostic—new signal keys such as clip_float flow through if HMA emits them. The main work is HMA packaging, product semantics (who uses CLIP, how rules treat it, thresholds, seeding), and ops cost (torch).

Background

Today	Gap
PDQ near-duplicate / known-hash matching via HMA	Weak on heavy transforms, different shots of the same subject/series
Exchange banks (NCMEC, etc.) sync PDQ-style hashes	No CLIP hash exchange for those programs
Custom banks created in Coop; content often seeded via HMA UI	Semantic banks need a clear operator seeding story
Boolean bank match; distance-threshold TODO in Coop	CLIP quality depends heavily on distance thresholds

CLIP (via CLIPFloatSignal / clip_float) embeds semantic similarity (cosine distance ~0.0–1.0 in the UMass extension). It complements PDQ; it does not replace verified hash-sharing programs.

Decided: HMA packaging

We will extend the published Coop HMA image (ghcr.io/facebook/threatexchange/hma:1.1.6 today), install a pinned commit of the UMass tx-extension-clip fork, and register CLIPFloatSignal next to PdqSignal and VideoMD5Signal in hma/omm_config.py.

Implications:

• Image size and RAM go up substantially (torch / open-clip). Local compose should plan for higher memory (on the order of multi‑GiB for HMA).
• Optional env already supported by the extension: CLIP_DISABLE_HNSW_INDEX (exact flat search vs default HNSW).
• We are not proposing (for this phase) rebuilding HMA from ThreatExchange source or adopting the evaluation’s /m/lookup_threshold / /m/lookup_topk patches. Prefer Coop-side use of /m/lookup distances unless those APIs prove necessary later.

Proposed architecture (high level)

Items / Report API
  → HMA /h/hash          (pdq + clip_float for photos; md5 for video)
  → store hashes on item
  → IMAGE_SIMILARITY_* rules
  → HMA /m/lookup per signal type against selected org banks

CLIP stays inside the existing HMA container. Coop continues to own bank metadata in Postgres (hash_banks) and rule wiring.

Strong lean (pending confirmation below): do not send clip / clip_float in NCMEC originalFileHash payloads. NCMEC paths should remain PDQ/MD5-style.

---

Questions for the team

1. Who is CLIP for? (product scope)

Not every org may want or need semantic matching, and CLIP has real infra cost (larger HMA image, more RAM, slower hashing than PDQ).

PDQ answers “is this nearly the same bytes/pixels as something we already know?” CLIP answers “is this semantically like something in our bank?” (e.g. another photo from the same series, heavy crop/filter, different shot of the same subject). Many orgs may only need NCMEC/exchange PDQ banks and never build a semantic corpus.

Possible shapes:

• Custom semantic banks for orgs that opt in — CLIP complements PDQ; exchange/NCMEC banks stay PDQ-oriented. Orgs that never opt in keep today’s behavior aside from shared HMA infra cost.
• CLIP as the default story for all custom banks — product leans semantic for anything you seed yourself; simpler mental model, but may surprise teams that expected near-dupe-only custom banks.

Separately: once the HMA image includes CLIP, hashing may run centrally for everyone. Even then we may want product controls—on for every org, opt-in per org, or opt-in per bank—so orgs that only use NCMEC PDQ banks are not surprised by semantic matches or extra latency on ingest.

Please weigh in: who should get CLIP in MVP, and should it be opt-in?

2. How should rules behave when an image has both pdq and clip_float?

Once both signals are enabled, a photo hashed at ingest will typically carry both pdq and clip_float. Today IMAGE_SIMILARITY_MATCH iterates all hashes on the image and treats any bank hit as a match (OR across types).

That is fine if operators understand “bank match = any algorithm.” It is risky if an existing “known hash” routing rule suddenly also fires on a loose CLIP hit against the same bank. Example: bank seeded with a few images for PDQ near-dupe; CLIP later matches a vaguely similar but unrelated image and enqueues to a high-severity queue.

Options:

Approach	Pros	Cons
OR across types (status quo)	Smallest change	Loose CLIP threshold can false-positive “hash match” rules
Rule selects algorithm(s)	Precise control without a new signal	More UI/schema; operators must understand PDQ vs CLIP
Separate semantic signal (e.g. leave IMAGE_SIMILARITY_* PDQ-oriented; add a semantic match signal)	Clearest product story (“hash match” vs “semantic match”)	New SignalType + rule UI + docs

Please weigh in: OR vs picker vs separate signal?

3. Where should match distance thresholds live?

For CLIP, threshold is the product. Too loose → false positives; too tight → misses. PDQ is relatively forgiving to ship with HMA’s defaults; CLIP needs an explicit place for operators (or us) to set “how similar is similar enough?”

Lean for discussion: per hash bank (banks are often purpose-built: “strict PDQ / NCMEC-style bank” vs “semantic series bank”). Threshold lives with the bank you curated.

Alternatives:

• Global defaults + env (e.g. HMA_DISTANCE_THRESHOLDS) — fast for a spike; one knob for the whole deployment; weak if orgs need different strictness.
• Per-rule condition — “match this bank if distance ≤ X” in the rule builder; most flexible for enforcement policy; more builder complexity and easier to misconfigure.
• HMA built-in thresholds only — simplest Coop code; tuning means HMA/extension config, not Coop UI.

Extension default for clip_float distance is on the order of 0.10 (UMass config); real threat corpora need calibration on Roost/customer data, not celebrity face demos.

Please weigh in: per-bank vs per-rule vs env-only for MVP? (We lean per-bank.)

4. How should operators seed custom semantic banks?

A bank only matches after media (or hashes) are indexed into it. Coop can create banks today; adding media into a bank from Coop is not wired (server helper addContentToBank exists; GraphQL/UI do not). Operators can use the HMA UI locally by finding COOP_<ORGID>_… and uploading there.

Without a seeding path, CLIP can be “on” in HMA but unused for custom semantic banks. Exchange sync fills NCMEC-style banks with PDQ hashes—it does not build your CLIP corpus.

Approach	When it fits
Coop “add by URL” on the Hash Bank form	Multi-tenant / prod; operators never touch HMA; aligns with org-scoped auth
HMA UI + docs only	Spike / local eval; weak if HMA UI is not exposed in prod or shared across tenants
Both	Coop as supported path; HMA UI as dev fallback

Note: HMA must be able to fetch the URL (same constraint as ingest hashing—localhost from inside Docker will fail).

Please weigh in: required Coop ingest for MVP, or is HMA UI enough to start?

---

References

• Coop HMA integration: docs/integrations/hma.md
• Evaluation (CLIP + HMA banks + metrics): https://github.com/UMass-Rescue/2025-project-roost-evaluation
• CLIP extension (float signal): https://github.com/UMass-Rescue/tx-extension-clip
• Upstream HMA: https://github.com/facebook/ThreatExchange/tree/main/hasher-matcher-actioner

[juanmrad]

I think we can actually instead of add documentation on how to enable this as well as a guide. We could have the configs on the same HMA folder, but wouldn't have it as part of the docker compose or hooked up to anything other than reference on a guide as that will also affect local development.

From your own writeup adding CLIP increases the deployment needs given how heavy and slow CLIP calculations are, and the image is also way bigger. This helps adding visibility and guidance for anyone interested in exploring CLIP to know how to do it.

[sahilx13]

Yes, this is where I was generally leaning towards as well. So this is what I am looking at:

• Clip will be an optional deployment with steps on how to install/deploy and system needs (GPU).
• Have specific rules to generate Clip hashes and match
• Have a URL where an image can be ingested (for now), and then it generates a Clip hash (unless we want a separate UI)
• Policy rule will dictate which bank will be used to match a specific hash
• Seeding a bank can be via a custom path (unless we decide on a separate UI)