[schemas] Smart ingest pipeline tables

[alanshurafa]

Summary

• Adds ingestion_jobs and ingestion_items tables for tracking the extract-deduplicate-execute lifecycle of bulk text ingestion
• Installs append_thought_evidence RPC for idempotent evidence accumulation on thoughts
• Part of the OB1 alpha milestone (Wave 2 schema — depends on PR [schemas] Enhanced thoughts columns and utility RPCs #1 enhanced-thoughts)

What's included

Object	Type	Purpose
ingestion_jobs	table	One row per ingest invocation with status lifecycle and per-action counters
ingestion_items	table	Individual extracted thoughts with action codes, dedup reasons, and execution results
ingestion_items_job_idx	index	Fast job-to-item lookups
append_thought_evidence()	RPC	Idempotent evidence append using SHA256 identity hashing

Design decisions

• Idempotent DDL: All CREATE TABLE IF NOT EXISTS and CREATE OR REPLACE FUNCTION patterns — safe to run multiple times
• Dry-run lifecycle: Jobs progress through pending → extracting → dry_run_complete → executing → complete, supporting human-in-the-loop review before mutation
• Content-hash dedup: input_hash unique constraint on jobs prevents duplicate processing of the same text
• Evidence identity: append_thought_evidence uses SHA256 of (source_label + excerpt + thought_id) to prevent duplicate evidence entries
• Cascade cleanup: Deleting a job automatically removes its items via ON DELETE CASCADE

Dependencies

• Requires enhanced-thoughts schema (PR [schemas] Enhanced thoughts columns and utility RPCs #1) for the enriched thoughts table columns
• No external services or API keys needed

Test plan

• Paste schema.sql into Supabase SQL Editor and run — no errors
• Confirm ingestion_jobs and ingestion_items tables appear in Table Editor
• Confirm append_thought_evidence function appears in Database > Functions
• Run SELECT count(*) FROM ingestion_jobs; — returns 0
• Re-run the migration — no errors (idempotency check)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3abe49a057

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Serialize evidence updates to avoid lost writes

append_thought_evidence reads metadata->'evidence' into v_current_evidence and later writes back v_current_evidence || ..., so concurrent calls on the same thought can clobber each other: if two workers append different evidence at the same time, the second write can overwrite the first and drop one entry. This breaks the function’s stated idempotent accumulation behavior for multi-item ingestion and should be fixed by locking the row (FOR UPDATE) or performing the append against the current row value inside a single UPDATE expression.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Restrict SECURITY DEFINER RPC from anon/auth roles

This migration grants EXECUTE on a SECURITY DEFINER function to authenticated and anon, which allows callers to mutate public.thoughts through the function even when row-level access is intended to be service-role-only (as configured in docs/01-getting-started.md with a service-role policy on thoughts). In Supabase deployments where anon keys are client-visible, this exposes an authorization bypass for arbitrary thought_id updates and should be limited to service_role (or enforce caller ownership checks inside the function).

Useful? React with 👍 / 👎.