[integrations] Smart ingest edge function

[alanshurafa]

Summary

• Standalone Supabase Edge Function for LLM-powered atomic thought extraction from raw text
• Ported from ExoCortex production smart-ingest pipeline (1,369 completed jobs) with OB1 adaptations
• Depends on schemas/smart-ingest-tables (PR [schemas] Smart ingest pipeline tables #4) for ingestion_jobs and ingestion_items tables

What It Does

Accepts raw text via HTTP POST, extracts atomic thoughts using an LLM (OpenRouter primary, OpenAI/Anthropic fallback), then deduplicates each thought against existing brain content using both SHA-256 content fingerprinting and pgvector semantic similarity. Four reconciliation actions: add, skip, append_evidence, create_revision.

Key Features

• Dry-run mode — preview extractions without writing to the database
• Job execution — commit dry-run results via /execute endpoint
• Quality gate — minimum 30 chars, minimum importance 3
• Fingerprint + semantic dedup — 0.85 match threshold, 0.92 skip threshold
• Source metadata threading — import_key session dedup, capture provenance
• Text chunking — handles long documents (5000 word limit per LLM call)
• Sensitivity pre-flight — blocks restricted content from cloud processing
• Entity extraction trigger — optional, best-effort (non-fatal if worker not deployed)

OB1 Adaptations

• OpenRouter-first LLM provider order (reversed from ExoCortex)
• Wildcard CORS for generic deployments
• Model constants from _shared/config.ts (consistent with enhanced-mcp)
• _shared/ helpers copied from enhanced-mcp (PR [integrations] Enhanced MCP server with alpha tool suite #6) for consistency

Files

All within integrations/smart-ingest/:

File	Lines	Purpose
index.ts	1094	Edge function with extraction, dedup, and execution logic
_shared/helpers.ts	770	Shared utilities (embedding, fingerprint, sensitivity, payload prep)
_shared/config.ts	204	Constants, types, prompts
README.md	225	Setup guide with prerequisites, steps, API reference, troubleshooting
metadata.json	18	OB1 contribution metadata
deno.json	5	Deno import map

Test plan

• Verify all 15 gate checks pass via gh pr checks
• Validate metadata.json against .github/metadata.schema.json
• Confirm README contains: "prerequisites", numbered steps, "expected outcome"
• Confirm "05-tool-audit" string appears in README
• Confirm all relative links resolve (../../docs/01-getting-started.md, ../../docs/05-tool-audit.md)
• Confirm no files outside integrations/smart-ingest/
• Deploy to test Supabase project and smoke-test dry-run + execute flow

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 661fe55dc6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fall back to OpenAI when OpenRouter embedding fails

embedText advertises OpenRouter-primary/OpenAI-fallback behavior, but this branch throws immediately on any OpenRouter non-2xx response, so the OpenAI branch is never attempted when both keys are configured. In production, transient OpenRouter 429/5xx errors will cause ingestion reconciliation to fail (or lose embeddings) even though a healthy fallback provider is available; catch this failure and continue to the OpenAI path instead of hard-failing here.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Mark skipped dry-run items executed in /execute

In handleExecuteJob, skip actions are counted and immediately continued, but the corresponding ingestion_items row is never updated. Because dry-run persistence stores pending items as ready, these rows stay ready even after the job is marked complete, leaving job state inconsistent and potentially misleading any UI/automation that interprets ready as unprocessed. Update skipped rows to executed before continuing (as the immediate-execution path already does).

Useful? React with 👍 / 👎.