add services (Graph RAG, HITL, Workflow Generator, Hooks) + Docker dev env + Chat UX

[nnnet]

Overview

This PR adds four new backend services, a Docker-based development environment, and several chat UX improvements. It lays the foundation for knowledge graph management,
human-in-the-loop workflow approvals, LLM-powered workflow generation, and lifecycle hooks.

42 files changed | +4,802 lines | 7 new database tables | 30+ new API endpoints

---

New Services

1. Graph RAG — Knowledge Graph + Hybrid Retrieval

Manages entity-relationship knowledge graphs with three search modes: vector similarity (pgvector HNSW), graph traversal (neighbor depth queries), and fuzzy keywo23:04:15 [160/538]
(pg_trgm). Optional LightRAG integration for advanced entity extraction.

• Interface: packages/core/src/services/graph-rag-service.ts
• Implementation: packages/gateway/src/services/graph-rag-service.ts (913 LOC)
• Routes: packages/gateway/src/routes/knowledge-graph.ts

Method	Endpoint	Description
POST	/api/v1/knowledge-graph/ingest-text	Extract entities from text via LLM and store
POST	/api/v1/knowledge-graph/extract	Extract entities/relations without storing
POST	/api/v1/knowledge-graph/ingest	Store pre-extracted knowledge
GET	/api/v1/knowledge-graph/search	Hybrid search (vector + graph + keyword)
GET	/api/v1/knowledge-graph/entities	List entities with pagination and filters
GET	/api/v1/knowledge-graph/entities/:id	Get entity by ID
GET	/api/v1/knowledge-graph/entities/:id/neighbors	Graph traversal (configurable depth)
DELETE	/api/v1/knowledge-graph/entities/:id	Delete entity and its relations
POST	/api/v1/knowledge-graph/collections	Create knowledge collection
GET	/api/v1/knowledge-graph/collections	List collections
DELETE	/api/v1/knowledge-graph/collections/:id	Delete collection
POST	/api/v1/knowledge-graph/decay	Apply memory decay to stale entities
GET	/api/v1/knowledge-graph/lightrag/status	Check LightRAG availability
POST	/api/v1/knowledge-graph/lightrag/insert	Insert document into LightRAG
POST	/api/v1/knowledge-graph/lightrag/query	Query LightRAG directly

DB tables: knowledge_entities, knowledge_entity_embeddings, knowledge_relations, knowledge_collections

2. HITL — Human-in-the-Loop Approvals

Manages approval/rejection requests during workflow execution. Supports multiple interaction types: approve/reject, collect input, review tool calls, multi-turn conversation.
Requests expire automatically after a configurable timeout.

• Interface: packages/core/src/services/hitl-service.ts
• Implementation: packages/gateway/src/services/hitl-service.ts (305 LOC)
• Routes: packages/gateway/src/routes/hitl.ts

Method	Endpoint	Description
POST	/api/v1/hitl/requests	Create approval request
GET	/api/v1/hitl/requests/pending	List pending requests
GET	/api/v1/hitl/requests/:id	Get request details
POST	/api/v1/hitl/requests/:id/resolve	Approve / reject / modify
POST	/api/v1/hitl/requests/cancel-workflow	Cancel all pending for a workflow
POST	/api/v1/hitl/expire	Expire timed-out requests

DB table: hitl_requests

3. Workflow Generator — LLM-Powered DAG Creation

Generates complete workflow definitions from natural-language goals. Three-phase pipeline: decompose goal into subtasks, build DAG with appropriate node types, validate result.
Streams progress to the client via Server-Sent Events.

• Interface: packages/core/src/services/workflow-generator-service.ts
• Implementation: packages/gateway/src/services/workflow-generator-service.ts (556 LOC)
• Routes: packages/gateway/src/routes/workflow-generator.ts

Method	Endpoint	Description
POST	/api/v1/workflow-generator/generate	Generate workflow (SSE stream)
POST	/api/v1/workflow-generator/decompose	Decompose goal into subtasks only
POST	/api/v1/workflow-generator/review	Validate a generated workflow
GET	/api/v1/workflow-generator/history	List generation history

DB table: workflow_generations

4. Workflow Hooks — Lifecycle Event Hooks

Registers and fires hooks on workflow execution events (start, end, node start/end/error/retry, approval events). Five hook types: logging, metrics, notification, webhook, custom.
Fire-and-forget execution with 60s in-memory cache.

• Interface: packages/core/src/services/workflow-hooks-service.ts
• Implementation: packages/gateway/src/services/workflow-hooks-service.ts (247 LOC)
• Routes: packages/gateway/src/routes/workflow-hooks.ts

Method	Endpoint	Description
GET	/api/v1/workflow-hooks/:workflowId	List hooks for a workflow
POST	/api/v1/workflow-hooks/:workflowId	Create/update hook
DELETE	/api/v1/workflow-hooks/hook/:id	Delete a hook
PATCH	/api/v1/workflow-hooks/hook/:id/toggle	Enable/disable a hook

DB table: workflow_hook_configs

---

Docker Development Environment 23:04:15 [75/538]

New files: Dockerfile.dev, docker-compose.dev.yml

• Hot reload: gateway source mounted as volume, tsx watch restarts on changes
• Services: PostgreSQL (pgvector), Gateway (port 8501), optional LightRAG and MQTT
• Local AI connectivity: transparent localhost / 127.0.0.1 → host.docker.internal URL rewriting so containers can reach host-side AI servers (LMStudio, Ollama) without
  manual configuration
• Utility: packages/gateway/src/utils/docker-url.ts — shared rewrite function used by BaseProvider, EmbeddingService, and LocalDiscovery

Production docker-compose.yml also updated with optional LightRAG service.

---

Chat UX Improvements

Arrow-Up Input History

packages/ui/src/components/ChatInput.tsx

• Arrow Up/Down cycles through previously sent messages (like terminal / Claude Code CLI)
• Persisted in localStorage (max 50 entries, key: ownpilot-chat-input-history)
• Only activates when cursor is at the start (Up) or end (Down) of text — no interference with multiline editing
• Current draft preserved when navigating

Persistent LLM Selection

packages/ui/src/hooks/useChatStore.tsx

• Selected provider and model saved to localStorage
• Survives page refresh and container restarts

Cross-Model Conversation Transfer

packages/gateway/src/routes/chat.ts

• Switching models mid-conversation no longer returns 404
• New agent searches the cached agent pool, exports/imports conversation memory
• Falls back to starting a fresh conversation if not found anywhere

Provider Resolution Fallback

packages/gateway/src/routes/providers.ts

• GET /providers/:id and GET /providers/:id/models now fall back to local providers (LMStudio, Ollama) when core preset lookup returns null
• Fixes "Provider not found" error on Dashboard for local provider UUIDs

Toggle Endpoint Fix

packages/gateway/src/routes/local-providers.ts

• PATCH /local-providers/:id/toggle now works without a request body (inverts current state)
• Previously returned 400 when UI sent PATCH with empty body

---

Database Migration

File: packages/gateway/src/db/migrations/postgres/028_knowledge_graph.sql

Creates 7 tables, all idempotent (IF NOT EXISTS):

Table	Purpose
knowledge_entities	Graph nodes with pgvector embeddings, pg_trgm fuzzy search
knowledge_entity_embeddings	Separate vector index for faster similarity search
knowledge_relations	Graph edges with unique constraint per (user, source, target, type)
knowledge_collections	Agent-scoped entity groupings
hitl_requests	Approval queue with JSONB context/response
workflow_hook_configs	Per-workflow hook settings
workflow_generations	Generation history with metrics

All tables enforce multi-tenancy via user_id; optional agent_id for per-agent scoping.

---

Testing & Demo Data

File	Purpose
tests/run-all.sh	Runs all integration test suites
tests/seed-demo.sh	Populates demo data (agents, goals, memories, triggers, plans, workflows, KG, HITL, hooks)
tests/test-health.sh	Health endpoint checks
tests/test-knowledge-graph.sh	Knowledge Graph API tests
tests/test-hitl.sh	HITL API tests
tests/test-workflow-generator.sh	Workflow Generator API tests
tests/test-workflow-hooks.sh	Workflow Hooks API tests
docs/SEED_DATA.md	Guide to seed data and where to find it in the UI

---

Bugs Fixed

• Provider not found on Dashboard for local provider UUIDs (LMStudio, Ollama)
• Conversation not found 404 when switching models mid-chat
• Toggle 400 error when UI sends PATCH without body
• Docker→host connectivity for localhost-bound AI servers
• ON CONFLICT mismatch in knowledge_relations (unique index missing user_id)
• Missing table knowledge_entity_embeddings (referenced by service but not in migration)

[nnnet]

New backend services — Graph RAG (knowledge graph with hybrid vector/graph/keyword search), HITL (human-in-the-loop approvals for workflows), Workflow Generator
(LLM-powered DAG creation from natural language), and Workflow Hooks (lifecycle event hooks: logging, metrics, webhooks).

Also includes: Docker dev environment with hot reload, chat UX improvements (input history, persistent LLM selection, cross-model conversation transfer), 7 new DB tables, 30+ API
endpoints, integration test suite with demo seed data.

Key bug fixes: "Provider not found" for local providers (LMStudio/Ollama), conversation 404 on model switch, Docker→host AI server connectivity.