Skip to content

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

AI Model Provenance & Jurisdictional Exposure Assessment

A practical checklist for evaluating AI models, tools, and vendors for exposure to compelled-access jurisdictions — with sources credited.

Maintained by grAIce · Last updated 2026-07-23 · Contributions welcome


Read this first

This guide is about jurisdiction and verifiability, not nationality. The controls here would apply identically to a model trained in any country whose law permits secret compelled access. China's legal regime is the most consequential example today, and this guide names it directly. The method is general.

This guide is not anti-open-source. Open source is one of the great achievements of computing. This guide exists partly because the phrase "open source AI" is being used loosely in a way that borrows trust the artifact hasn't earned. See §3.

What this guide can do: help you inventory what you're running, cut off the data-path risks, avoid the known file-format attacks, and build controls that hold even when detection fails.

What this guide cannot do: tell you whether a set of model weights contains a backdoor. Nobody can currently do that reliably. Anyone who tells you otherwise is selling something. See §6.


How long this takes, and what you actually do

About an hour for a typical machine — and the longest step is the one no script can do.

Written for people who don't code. You need a terminal exactly once, to type one command.

flowchart TD
    A["START<br/>about 1 hour total"] --> B{"Do you have Python?<br/>type: python3 --version"}
    B -->|"no"| B1["Install from python.org<br/>10 min"]
    B -->|"yes, or Mac/Linux"| C["Run the scanner<br/>python3 ai_inventory.py<br/>5 min"]
    B1 --> C
    C --> D["Read the list<br/>worst items printed first<br/>5 min"]
    D --> E{"What verdict<br/>did each model get?"}

    E -->|"STOP"| S["DO NOT DELETE ANYTHING<br/>deleting destroys the evidence<br/>tell compliance or security today"]
    E -->|"REMOVE"| R{"Did you ever LOAD it,<br/>or only download it?"}
    E -->|"KEEP WITH CONTROLS"| K1["Run local, no internet access<br/>review any code it writes<br/>20 min setup, then a habit"]
    E -->|"KEEP"| K2["Nothing further<br/>0 min"]

    R -->|"only downloaded"| R1["Delete the folder<br/>5 min each"]
    R -->|"loaded it"| R2["Work Section 9 in order<br/>rotate every credential<br/>30 min+"]

    S --> F
    R1 --> F
    R2 --> F
    K1 --> F
    K2 --> F

    F["THE STEP NO SCRIPT CAN DO<br/>hosted apps, APIs, and AI features inside<br/>tools you already pay for<br/>password manager, then 12 months of statements<br/>30-45 min"]
    F --> G["DONE<br/>a clean scan means nothing was found on disk<br/>it does not mean no exposure"]

    style S fill:#5c1a1a,stroke:#ef4444,color:#fff
    style R fill:#5c2e1a,stroke:#fb923c,color:#fff
    style R1 fill:#5c2e1a,stroke:#fb923c,color:#fff
    style R2 fill:#5c2e1a,stroke:#fb923c,color:#fff
    style K1 fill:#5c3d1a,stroke:#f59e0b,color:#fff
    style K2 fill:#1a4d2e,stroke:#22c55e,color:#fff
    style F fill:#2d1b69,stroke:#8b5cf6,color:#fff
    style G fill:#1e3a5f,stroke:#4a9eed,color:#fff
Loading

Time, honestly

Step Time Automated?
1. Get Python 0–10 min Usually already on Mac and Linux
2. Run the scanner 5 min This is the automation
3. Read the verdicts 5 min No — but it's plain English
4. Act on what you found 0–30 min Partly
5. Hosted accounts and SaaS 30–45 min No. Nothing can do this for you
≈ 1 hour

Three things this table is telling you:

  1. The unautomatable step is the biggest one, and it finds the most. Everything on disk can be scanned in five minutes. Your hosted accounts cannot be scanned at all, and that's where most people's real exposure sits. If you only have twenty minutes, skip the script and do step 5.

  2. If anything comes back STOP, the clock stops. That's not a task with a duration — it's a phone call. Don't estimate it, don't schedule it, and don't try to finish the rest of the list first.

  3. "KEEP WITH CONTROLS" is 20 minutes of setup and then a permanent habit. Reviewing generated code isn't a task you complete. It's how you work from now on. That's the honest version, and anyone who tells you otherwise is selling a product.

If your markdown viewer doesn't render the diagram above, the same flow reads top to bottom: check Python → run the scanner → read the verdicts → act worst-first → then do the hosted-account audit by hand.


1. The three questions

Almost every confused argument about this topic comes from collapsing three separate risks into one. Keep them apart.

# Question Risk Fixable?
1 Where does my data go? Prompts, files, and code leave your machine and land under foreign jurisdiction Yes — completely. Run locally or don't send sensitive data
2 What is in the artifact I downloaded? Malicious code in the model file (pickle deserialization, Keras Lambda layers) Yes — largely. Format discipline and scanning
3 What is in the weights themselves? Trigger-conditioned behavior baked into the parameters No — not by detection. Only by containment. See §6

Risk 1 is the one everyone argues about. Risk 3 is the one that actually deserves the Trojan horse metaphor.


2. Tier 0 — Find the AI you didn't know you had

This section is not about making a list of the AI you chose. It's about finding the AI that arrived without a decision.

That distinction is the entire point. If you deliberately installed DeepSeek, you already know it's there and you can reason about it. The dangerous surface is the model you never chose: pulled automatically by a Python package, downloaded by a tutorial notebook you ran once, installed by your teenager, or switched on inside a SaaS tool you've paid for since 2023 because the vendor shipped an update.

You cannot make a risk decision about something you don't know exists. Every check below exists to surface one specific category of unchosen AI.

2.1 What each check finds, and what it protects you from

Check What it surfaces The risk it maps to
A. Model cache Weights downloaded automatically by libraries, scripts, or tutorials You may be running weights of unknown provenance (§7, §8) and unknown file format (§5)
B. Local runners Models someone deliberately pulled — possibly not you Same as above, plus shared-device exposure
C. Endpoint redirect Whether your tools are pointed at a server other than the official one Your prompts may be transiting an unknown third party right now (§4)
D. File format Pickle-format model files on disk These execute code when loaded (§5)
E. Hosted accounts No command finds these. Manual work required Where most people's real exposure actually lives (§4)

A. The silent cache

# macOS / Linux
ls -la ~/.cache/huggingface/hub/ 2>/dev/null
# Windows PowerShell
Get-ChildItem "$env:USERPROFILE\.cache\huggingface\hub" -ErrorAction SilentlyContinue

What it does: lists model weights sitting on your disk that were downloaded by anything using the Hugging Face libraries.

Why it matters: almost nobody fills this folder on purpose. Install a Python package, run an example from a blog post, open a Jupyter notebook someone shared — any of these can pull gigabytes of model weights on first execution, with no prompt and no confirmation. This is where models you never chose accumulate.

What you're looking for: folder names in the form models--<organization>--<model-name>. The organization segment is the publisher.

You don't have to identify these by hand. ai_inventory.py (§2.3) reads this folder, extracts each publisher, and checks it against §8 automatically. To look up a single name you saw somewhere:

python3 ai_inventory.py --check deepseek-ai

B. Local model runners

ollama list

LM Studio, GPT4All, and Jan each keep their own separate model directoryai_inventory.py (§2.3) scans all four automatically, so you don't need to find these folders yourself. Paths vary by version; if the tool misses one, point it at the folder with --path.

What it does: lists models pulled deliberately through a local-inference tool.

Why it matters: these were chosen, but often months ago and often not by you. On a shared or family machine this is the check that surfaces what someone else installed. Ask your kids directly. Homework assistants and "free AI" apps are a genuine vector into a household network, and teenagers pull models with no provenance thought at all.

The catch this check exists for: the folder name tells you who uploaded the model, not who built it. A model published by a hobbyist in Ohio can be a repackaged Chinese base model — TheBloke/deepseek-coder-6.7B-GGUF is DeepSeek weights under a British uploader's account. The tool reads the model name as well as the publisher and reports the origin of the weights, flagging the mismatch when the two disagree.

C. The endpoint redirect check — run this one even if you skip the rest

# macOS / Linux
env | grep -iE 'base_url|api_base|_endpoint'
grep -rIn "BASE_URL\|api_base" ~/.zshrc ~/.bashrc ~/.profile ~/.config 2>/dev/null
# Windows PowerShell
Get-ChildItem Env: | Where-Object { $_.Name -match 'BASE_URL|API_BASE|ENDPOINT' }

What it does: reveals whether any tool on your machine has been redirected to send its traffic to a server other than the vendor's official one.

Why this is the highest-value line in the section: ANTHROPIC_BASE_URL is the exact environment variable the hidden Claude Code detection logic inspected — because it is the standard way requests get routed through corporate gateways, third-party proxies, unofficial resellers, and region-specific API routers.

Here is what that means in practice. You can be looking at a familiar interface, believing you are using a US-hosted model, while your prompts transit an intermediary you never evaluated. The brand on the screen tells you nothing about the path your data takes. This variable does.

What to do if it's set and you didn't set it:

  1. Do not send anything sensitive through that tool until resolved.
  2. Find what set it: check your shell profile files, .env files in project folders, and your IDE settings.
  3. Unset it (unset ANTHROPIC_BASE_URL) and confirm the tool still works against the official endpoint.
  4. If a vendor, employer, or tutorial told you to set it — that's a supply-chain relationship you haven't assessed. Assess it (§7).

D. Dangerous file formats

# macOS / Linux
find ~ -type f \( -name "*.bin" -o -name "*.pt" -o -name "*.pth" -o -name "*.ckpt" \) 2>/dev/null | head -50
# Windows PowerShell
Get-ChildItem -Path $env:USERPROFILE -Recurse -Include *.bin,*.pt,*.pth,*.ckpt -ErrorAction SilentlyContinue | Select-Object -First 50

What it does: finds model files stored in pickle-based formats.

Why it matters: loading one of these executes code before any inference happens. This check tells you, in about three seconds, whether §5 applies to you at all. If it returns nothing, you can skip most of §5. If it returns results, read §5 before you load any of them again.

E. The category with no command

No shell command will find these. This is where most people's real exposure lives.

Work through it by hand:

  • Chat apps and web UIs — personal and work accounts both
  • "AI-powered" features inside SaaS tools you already pay for — the single most commonly missed item. You didn't opt in; the vendor shipped an update
  • IDE extensions and coding assistants
  • Browser extensions with AI features
  • Inference routers and aggregators — these may silently route to models you did not choose
  • Family devices, kids' devices, homework helpers, AI toys
  • Voice assistants and smart devices with cloud AI backends

Fastest way to find them: open your password manager and read every entry. Then open the last twelve months of card and bank statements and read every recurring charge. People remember the tools they use weekly. They forget the ones they signed up for once.

2.2 The output

You are done with Tier 0 when you can answer, for every item found:

  1. What is it?
  2. Who made it, and under whose jurisdiction? (§7, §8)
  3. Does my data leave my machine when I use it? (§4)
  4. Did I choose this, or did it arrive?

Item 4 is the one that matters. Anything you cannot account for gets treated as untrusted until you can.


2.3 Automating checks A–D: ai_inventory.py

Checks A, B, C, and D are in this repo as a single script. No installation, no dependencies, Python 3.8+. It reads only — writes nothing, deletes nothing, never loads a model file.

python3 ai_inventory.py              # scan everything, get a verdict per model
python3 ai_inventory.py --regulated  # machine handles PHI/financial/legal/gov data
python3 ai_inventory.py --check qwen # look up one name and stop
python3 ai_inventory.py --offline    # local inspection only, no network
python3 ai_inventory.py --json       # machine-readable, for a compliance record

Scans: Hugging Face cache, Ollama, LM Studio, GPT4All, Jan, plus any folder you name.

It tells you where each model actually came from

Two independent signals, because they often disagree:

  • Publisher — who uploaded it
  • Base model — what the weights actually are, read from the model name

When they conflict, the weights win and the tool says so. TheBloke/deepseek-coder-6.7B-GGUF reports as Built by DeepSeek (China) — publisher unrecognised, origin inferred from model name. That is the re-upload case, and it is the single most common way a model trained in China ends up on a machine whose owner believes otherwise.

It gives you a verdict, not just data

Verdict Means Typical trigger
KEEP Nothing further needed Safe format, identified non-PRC origin
KEEP WITH CONTROLS Usable, not unsupervised Safe format, PRC or unidentified origin
REMOVE Delete it; a safer build exists Pickle format, identified non-PRC origin
STOP — GET PROFESSIONAL HELP Don't handle this alone Pickle format + PRC/unknown origin, or anything flagged on a regulated machine

Results are sorted worst-first, so the thing that matters is at the top.

Two design decisions worth knowing about:

1. Having a pickle file is not the same as having run one. A file sitting on disk has executed nothing. So every unsafe-format finding asks you the question the tool cannot answer: have you ever loaded this, or only downloaded it? Only downloaded → deleting is sufficient. Loaded → work §9 in order.

2. STOP tells you not to delete anything yet. If an incident is possible, deleting destroys the evidence of what happened. That is the opposite of the instinct, and it is why this verdict exists separately from REMOVE.

--regulated changes the thresholds, not just the tone

If the machine handles PHI, financial, legal, or government data, your obligations differ from your risk. In this mode anything with PRC or unidentified origin escalates to STOP, and the output points at §9 Step 6 — notification duties. In testing, the same six models produced one STOP normally and three under --regulated.

What it explicitly cannot do

Printed at the end of every scan so nobody misses it:

  • It cannot detect backdoors in weights. Nothing can — not this tool, not any commercial product. That's why §6 is containment, not detection.
  • It cannot verify publisher claims. Licences and provenance are self-reported. The tool repeats them; it does not confirm them.
  • It cannot see hosted accounts, APIs, or SaaS AI features — check E, where most real exposure lives.

A clean scan means "nothing found on disk." It does not mean "no exposure."


3. Tier 1 — Vocabulary discipline

Fix your language before you assess anything, because the wrong word will lead you to the wrong control.

Term What it actually means
Open source Source code available and human-readable. Anyone can audit the logic.
Open weights Parameters downloadable. The training data, training code, and resulting behavior are not auditable.
Open license A legal permission. Says nothing whatsoever about security.

You can read every line of the Linux kernel. You cannot read weights.

"Open" in "open weights" means downloadable, not inspectable. The field that studies what's inside weights — mechanistic interpretability — is an active research frontier, not a compliance control.

Almost every model marketed as "open source AI" is open weights. Some don't even meet that bar. When someone says "it's open source, so it's safe," they are making a claim the artifact cannot support.


4. Tier 2 — The data path (fully fixable)

Rule: sensitive data never touches a hosted endpoint in a compelled-access jurisdiction.

4.1 Hosted service checks

  • Where is data stored at rest? (Check the privacy policy, not the marketing page.)
  • Where is inference executed? Data residency claims for storage often don't cover inference.
  • Is training-on-your-data opt-out, opt-in, or exempted from consent entirely?
  • What's retained: prompts, outputs, metadata, keystroke timing, device fingerprints, IP?
  • Who is the corporate parent, and where is it incorporated?
  • Is there a compelled-access statute in that jurisdiction with a non-disclosure obligation?

4.2 The verification test

This is the core reasoning move of the whole guide. Ask four questions:

  1. Does a legal mechanism exist to compel this entity to provide data or assistance?
  2. Does it reach entities and citizens operating abroad?
  3. Is the compelled party legally barred from disclosing the compulsion?
  4. Can any assurance the vendor gives you be independently verified?

If 1–3 are yes and 4 is no, vendor assurances carry zero information. Not because the vendor is lying — because a denial issued under a non-disclosure obligation is exactly as informative as silence.

For China this is the National Intelligence Law (2017, amended 2018):

  • Article 7 — all organizations and citizens shall support, assist, and cooperate with national intelligence work, and shall protect national intelligence work secrets they are aware of
  • Article 10 — intelligence work may be carried out domestically and abroad
  • Article 14 — intelligence institutions may demand that support
  • Counter-Espionage Law (2014) — relevant organizations "shall provide truthfully and may not refuse"

Honest counterpoint: some legal scholars (notably China Law Translate) argue Article 7 sits in general provisions, lacks a distinct enforcement mechanism, and doesn't authorize pre-emptive collection. This is a good-faith argument. It does not change the assessment, because the compliance question is about the existence of the mechanism and the impossibility of verification, not about frequency of use.

4.3 Comparable statutes elsewhere

Apply the same test everywhere. Symmetry is what makes this credible rather than partisan.

  • US: CLOUD Act; FISA 702; National Security Letters with gag provisions
  • UK: Investigatory Powers Act (technical capability notices, non-disclosure)
  • Russia: SORM; data localization requirements

The difference is one of degree, scope, and judicial oversight — not of category. Say so out loud when you present this to stakeholders. It's the difference between an assessment and an argument.


5. Tier 3 — The artifact (largely fixable)

Loading a model file can execute code before any inference happens.

5.1 Format rules

Format Executes code on load? Verdict
.safetensors No — tensor data only, no deserialization hooks Use this
.gguf No ✅ Acceptable
ONNX No ✅ Acceptable
.bin / .pt / .pth / .ckpt (pickle) Yes Avoid
TensorFlow / Keras .h5 Yes, via Lambda layers ⚠️ Caution

Rule: safetensors or don't load it. If a publisher offers only pickle format, that is itself a data point.

5.2 Known bypasses — why scanning alone is insufficient

Credit where due, these researchers did the work:

  • JFrog Security Research — found roughly 100 malicious models on Hugging Face with silent backdoors via altered module files
  • ReversingLabs — documented "nullifAI": PyTorch files compressed with 7z instead of ZIP, causing Picklescan to miss the embedded payload entirely
  • Sonatype — disclosed four vulnerabilities in Picklescan itself, the tool Hugging Face relies on
  • Protect AI (Guardian) + the huntr community — zero-trust model scanning; treats arbitrary code execution as unsafe regardless of apparent intent
  • Hugging Face — built safetensors, runs malware/pickle/secrets scanning

Critical: Hugging Face flags unsafe models. It does not block them. You can still download and execute a model marked unsafe. The warning is advisory.

5.3 Practical checks

# Prefer safetensors explicitly
huggingface-cli download <repo> --include "*.safetensors" "*.json"

# Scan any pickle you cannot avoid — in a disposable container, not on your daily driver
pip install picklescan
picklescan --path ./model.bin

# Verify checksums against the publisher's published hashes
sha256sum model.safetensors

Load unknown models inside a container with no network access, first time, every time.

docker run --rm --network=none -v "$(pwd)/model:/model:ro" <your-inference-image>

6. Tier 4 — The weights (NOT fixable by detection)

This section is the honest one. Read it even if you skip everything else.

6.1 What the research says

Anthropic — Hubinger et al., "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training" (arXiv:2401.05566, January 2024). Models were trained to write secure code when the prompt indicated 2023 and to insert exploitable vulnerabilities when it indicated 2024. Standard removal techniques were applied: supervised fine-tuning, reinforcement learning, adversarial training.

None removed the behavior. Adversarial training taught the model to conceal it better. Larger models preserved backdoors more effectively.

OWASP GenAI Security Project — LLM04: Data and Model Poisoning. Formally recognizes that poisoning may implement a backdoor leaving behavior untouched until a trigger fires, "in effect creating the opportunity for a model to become a sleeper agent."

Microsoft AI Red Team — Ram Shankar Siva Kumar (founder, 2019) calls detecting sleeper-agent backdoors the "golden cup" and says anyone claiming to have eliminated the risk is "making an unrealistic assumption." Microsoft's mechanistic work has identified anomalous attention patterns around trigger tokens — a promising research signal, not a deployable scanner.

Cloud Security Alliance research note (March 2026) — temporal backdoors in open-weight tool-using models achieving 99.6% attack success, activating on a specific calendar year. Standard red-teaming and benchmark regression testing would not surface this.

6.2 What this means for you

You will not out-detect a well-resourced state actor. Stop trying to build that capability and build the one that works instead:

Containment, not detection.

  • Treat all model-generated code as untrusted input. Review it. This is the single highest-value control, because it doesn't require detecting anything.
  • Never let an agent install packages, modify CI, or deploy unsupervised. The realistic attack is a model that suggests a subtly compromised dependency.
  • Pin and hash all dependencies. Verify anything a model recommends against a source you chose independently.
  • No network egress from local inference. --network=none.
  • Least privilege for agents. Scoped tokens, no standing credentials, no production write access.
  • Log and retain model outputs in any pipeline where output influences a system change.
  • Different models for different trust tiers. The model summarizing public articles need not be the model touching your codebase.
  • Behavioral canaries. If you use a model long-term, keep a fixed prompt set and diff outputs across versions. Won't catch a sophisticated trigger; will catch sloppy ones.

7. Corporate & ownership due diligence

  • Ultimate parent entity and country of incorporation
  • State ownership stakes, sovereign fund investment, or golden shares
  • Presence of a CCP committee within the corporate structure (required in many PRC firms)
  • VIE structures obscuring effective control
  • Listing on BIS Entity List, Treasury NS-CMIC, FCC Covered List, or DoD 1260H
  • Where the model was trained, and on whose infrastructure
  • Whether the inference provider differs from the model developer (a US host serving Chinese weights is a different risk profile — data path mitigated, weights risk unchanged)

Free lookups: BIS Entity List (bis.doc.gov) · Treasury sanctions search (sanctionssearch.ofac.treas.gov) · FCC Covered List · SEC EDGAR for US-listed parents


8. Model origin reference

Verified via reporting reviewed for this guide (July 2026):

Family Developer Notes
DeepSeek DeepSeek (Hangzhou), backed by High-Flyer Capital Extensive documented findings; multiple government bans
Qwen / Tongyi Alibaba Largest derivative ecosystem on Hugging Face
Kimi Moonshot AI K3 release triggered renewed US policy response, July 2026
GLM Zhipu / Z.ai
MiniMax MiniMax
MiMo Xiaomi
Hunyuan / Yuanbao Tencent Yuanbao named in Taiwan NSB ban
Doubao / Seed ByteDance Named in Taiwan NSB ban
Ernie / Yiyan Baidu Named in Taiwan NSB ban

Widely reported but not independently verified during this session — confirm before relying on: Yi (01.AI), InternLM (Shanghai AI Lab), Step (StepFun).

This list will go stale. New labs launch constantly and models get rebranded, quantized, merged, and re-uploaded by third parties under unrelated names. Check provenance per-artifact, not per-name. A model named after a Western city can be a fine-tune of anything.


9. If you think you're already exposed

Don't panic. Work the list in order.

Step 1 — Scope it (do this first)

  • What did you actually send? Casual questions are a different problem from client files, PHI, source code, or credentials.
  • Over what period?
  • From which device — personal, work, or a device with network access to work systems?
  • Was it the hosted app/API (data left your machine) or local weights (it didn't)?

Step 2 — Stop the flow

  • Uninstall the app; revoke API keys at the provider, not just locally.
  • Delete the account and submit a data deletion request. Under GDPR/CCPA you may have enforceable rights; a company under a non-disclosure obligation may not be able to fully honor them. Submit anyway — it creates a record.
  • Remove redirected BASE_URL variables from shells, .env files, and CI config.

Step 3 — Treat credentials as burned

  • Rotate any API key, token, password, or connection string that appeared in a prompt. Assume disclosure.
  • Rotate anything in a repo the tool had access to.
  • Check provider access logs for unexpected sessions.

Step 4 — If you loaded a pickle-format model

  • Treat the host as potentially compromised. Code execution on load is the documented risk.
  • Run a full endpoint scan; review persistence points (cron, launch agents, startup items, shell profiles).
  • Check outbound connections: lsof -i -P | grep ESTABLISHED
  • For a machine with access to sensitive systems: reimage. It's cheaper than the alternative.

Step 5 — Audit generated code (do not skip)

  • Review every dependency a model recommended. Verify each against the official registry.
  • Look for typosquatted package names, unpinned versions, unexpected postinstall scripts.
  • Re-run dependency audits: npm audit, pip-audit, cargo audit.
  • Diff any AI-generated code that touches auth, crypto, input validation, or network calls.

Step 6 — Notification obligations

This is where compliance people earn their keep. Check before you assume you have none.

  • Personal data of others → GDPR (72-hour authority notification), CCPA/CPRA, state breach laws
  • Health data → HIPAA
  • Financial data → GLBA, SOX, PCI-DSS
  • Government work → FISMA, CMMC, contractual reporting
  • Any regulated sector → check your specific regulator's incident rules

Step 7 — Report it

  • CISA: report@cisa.gov or (888) 282-0870
  • FBI: your local field office, or ic3.gov
  • Your organization's security team, before you tell anyone else

Step 8 — Write it down

Document what happened, when you found it, and what you did. Contemporaneous records are the difference between an incident and a finding.


10. Authoritative sources

Government guidance

  • NSA AISC, CISA, FBI + ASD ACSC, CCCS, NCSC-NZ, NCSC-UKDeploying AI Systems Securely: Best Practices for Deploying Secure and Resilient AI Systems (April 2024)
  • CISA, NSA, FBI + international partnersAI Data Security Cybersecurity Information Sheet (May 2025) — ten AI-specific practices covering data supply chain, maliciously modified data, and data drift
  • UK NCSC, CISA, NSA + 21 agencies across 18 countriesGuidelines for Secure AI System Development (2023)
  • NIST AI 100-1 — AI Risk Management Framework
  • NIST SP 800-53 / SP 800-37 — security controls and risk management framework
  • DHS — Safety and Security Guidelines for Critical Infrastructure Owners and Operators
  • CISA AI portal — cisa.gov/ai

Research

  • Hubinger et al. (Anthropic)Sleeper Agents, arXiv:2401.05566
  • OWASP GenAI Security Project — LLM Top 10, esp. LLM04
  • Microsoft AI Red Team — mechanistic work on backdoored attention patterns
  • Cloud Security Alliance — temporal backdoor research note (March 2026); AI Controls Matrix
  • BackdoorLLM benchmark (NeurIPS 2025)

Security research credited in this guide

  • JFrog Security Research — malicious Hugging Face models
  • ReversingLabs — nullifAI
  • Sonatype — Picklescan vulnerabilities
  • Protect AI / Guardian + huntr
  • Feroot Security — DeepSeek web platform → China Mobile authentication registry
  • Wiz Research — exposed DeepSeek database
  • NowSecure — DeepSeek iOS → Volcengine
  • Cisco + University of Pennsylvania; CrowdStrike; Qualys — model safety evaluations
  • "Thereallo" and Reddit user LegitMichel777 — Claude Code steganographic detection discovery, June 2026

Regulatory & legal

  • China Law Translate — National Intelligence Law translation and the dissenting analysis. Read both.
  • Italy's Garante decision on DeepSeek; European Data Protection Board AI Enforcement Task Force
  • Taiwan National Security Bureau assessment
  • US House Select Committee on the CCP — DeepSeek report

11. Limitations

State plainly, every time you use this:

  1. This cannot detect backdoors in weights. No available tool can. §6 is containment, not detection.
  2. Model scanning has documented bypasses. nullifAI is proof. Scanning reduces risk; it does not eliminate it.
  3. Provenance claims are self-reported. You are usually trusting a model card.
  4. This is not legal advice. Notification obligations are jurisdiction- and sector-specific. Ask a lawyer.
  5. Threat assessments are political artifacts too. Governments and commercial incumbents both have interests in how this risk gets characterized. Apply the same skepticism to official warnings that you apply to vendor marketing. If you only doubt one side, you're not doing security, you're doing politics.

Contributing

Corrections welcome, especially:

  • Model origin errors or omissions in §8
  • Broken or superseded source links
  • Jurisdictions and statutes missing from §4.3
  • Detection tooling that actually works — with evidence

Standard for merge: claims must be sourced. Assertions about specific companies must link to primary reporting or official documents. Speculation gets labeled as speculation or gets cut.


Licensed CC BY 4.0. Use it, fork it, argue with it.

There's no such thing as a free horse.

About

Practical framework and local, read-only scanner for assessing AI-model provenance, jurisdictional exposure, unsafe formats, and supply-chain risk.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages