[codex] complete TS effect-native closeout#985
Closed
kriegcloud wants to merge 163 commits into
Closed
Conversation
git-subtree-dir: ai-context/trustgraph-client git-subtree-split: 908f18cf814470ec3b72cc336bb945fb792ffdec
…t/trustgraph-client'
git-subtree-dir: ai-context/workbench-ui git-subtree-split: 32e36a5c2131e429a7081cfaf67dabad3193cda3
git-subtree-dir: ai-context/context-graph-demo git-subtree-split: 338a8ffadb1439013071ae922e55ed2421f17025
…t/context-graph-demo'
git-subtree-dir: ai-context/trustgraph-templates git-subtree-split: 338a8ffadb1439013071ae922e55ed2421f17025
…t/trustgraph-templates'
git-subtree-dir: ai-context/trustgraph-templates git-subtree-split: 42a5fd1b678f32be378062e30451e2052ccb95dd
…t/trustgraph-templates'
Minor fixes from linter: readonly modifiers, unused parameter prefixes, type narrowing in graph-rag BFS traversal and edge scoring. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix three critical bugs preventing the NATS message pipeline from working: - FlowProcessor now subscribes to config-push topic (was missing entirely), using DeliverPolicy.All to replay config on service restart - NATS streams use wildcard subjects (tg.flow.>) instead of per-topic narrow filters that caused 503 errors on publish - Subscriber dispatch loop has exponential backoff on errors to prevent tight error loops Add service runner scripts (gateway, config, LLM) and a 7-test integration suite that verifies config CRUD, WebSocket round-trip, and full LLM text-completion through the NATS pipeline. Fix Docker Compose infra: pin Tempo to v2.6.1, remove deprecated Loki config fields, add user:0 for volume permissions, remap conflicting ports (FalkorDB 6380, OTLP 4327/4328). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add missing topics (librarian, knowledge, collection-management, flow), pipeline message types (TextDocument, Chunk, Triples, EntityContexts), service message types (Librarian, Knowledge, Collection, Flow CRUD), and update AgentResponse for streaming chunk format. Add RequestResponseSpec enabling flow-scoped request/response calls (needed by knowledge extraction and agent services). Add requestor registry to Flow class with proper lifecycle management. Add end_of_dialog to gateway's isComplete() check for agent streaming. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document Pipeline (Team A): - LibrarianService: document storage with filesystem backend, metadata persistence, child document hierarchy, collection management - ChunkingService: recursive character text splitter with configurable chunk size/overlap, FlowProcessor pattern - KnowledgeExtractService: combined relationship + definition extraction using prompt service and LLM, emits RDF triples and entity contexts - KnowledgeCoreService: knowledge core CRUD with streaming export and flow-based loading ReAct Agent (Team B): - StreamingReActParser: state machine for parsing LLM output into Thought/Action/ActionInput/FinalAnswer sections - Three MVP tools: KnowledgeQuery (GraphRAG), DocumentQuery (DocRAG), TriplesQuery with RequestResponse clients - AgentService FlowProcessor with ReAct loop, tool execution, and streaming chunk responses (thought/observation/answer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-stage Containerfile for all Node.js services (single image, different CMD per docker-compose service). ESM entrypoints for gateway, config, text-completion, prompt, embeddings, agent, and librarian. Workbench gets a separate Containerfile (nginx:alpine) with SPA routing and API/WebSocket proxy to gateway. Docker Compose updated with 6 app services (gateway, config-service, text-completion, prompt, embeddings, workbench) using shared trustgraph-ts:local image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Flow Management Service: - FlowManagerService (AsyncProcessor) handling list/get/start/stop flows and list/get blueprints via kebab-case wire format - Default blueprint with all service topic mappings - Pushes flow config to config service on start/stop Config Seeding: - seed-config.ts script pushes prompt templates (extract-relationships, extract-definitions, document-prompt, kg-prompt) and default flow definition via gateway REST API Integration Tests: - Librarian CRUD: add-document, list-documents, get-content, delete - Agent query: verifies routing through gateway to agent service - Skip flags: SKIP_LIBRARIAN=1, SKIP_AGENT=1 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix Containerfiles: - Move tsconfig.json to workspace config layer for early availability - Add missing workspace package.json entries for pnpm lockfile resolution Docker Compose: - Move Grafana from port 3000 to 3030 (avoid conflicts) - Add agent, librarian, and flow-manager app services - Add librarian-data volume for document persistence Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix critical bug where all FlowProcessor services shared the same spec
names ("request"/"response"), causing them to steal each other's NATS
topics. Now each service uses unique spec names matching the flow config
topic keys (e.g., "text-completion-request", "prompt-request",
"agent-request").
Fix librarian NATS consumer timeout (500ms → 2000ms, below NATS minimum).
Update seed-config and test-pipeline with correct flow topic mappings.
Add prompt template runner script.
Smoke test results: 11/11 passing (config CRUD, WebSocket, LLM,
librarian CRUD). Agent routing verified via manual curl test.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Marks FlowProcessor and EmbeddingsService constructors as protected since these classes should only be instantiated via subclasses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add end-to-end document processing pipeline: - PDF decoder service (pdfjs-dist) extracts text per page from librarian docs - Ollama native LLM service for local model inference - FalkorDB triples store FlowProcessor consumer - Qdrant graph embeddings store FlowProcessor consumer - Fix spec name collisions in chunker/extractor (input→chunk-input, etc.) - Gateway /load endpoint to trigger document processing - Align flow manager blueprint and seed config with full pipeline topics - Add runner scripts and test coverage for document load Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cripts Wire up the query and retrieval side of the pipeline so the agent can answer questions from stored knowledge: - Triples query service (FalkorDB) — all SPO pattern queries via NATS - Graph embeddings query service (Qdrant) — entity vector similarity - Document embeddings query service (Qdrant) — chunk vector similarity - Graph RAG service — full concept→entity→traverse→score→synthesize pipeline - Document RAG service — embed→find chunks→synthesize pipeline - Runner scripts for chunker, extractor, embeddings (missing from Phase 5) - Add DocumentEmbeddingsRequest/Response schema types - Add RAG prompt templates (extract-concepts, edge-scoring, synthesize) - Add graph/doc embeddings query topics to seed config + flow manager - Add all pipeline/query/retrieval services to docker-compose - 8 new runner scripts, 8 new pnpm script aliases Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs found during end-to-end testing: 1. FlowProcessor never restarted flows when config changed — it only started them once. Stale NATS JetStream data from previous sessions caused services to bind to wrong topics. Fix: stop and restart flows on every config push that includes flow definitions. 2. Gateway publishToTopic sent messages without an id property. Pipeline FlowProcessor handlers check properties.id and silently return if missing. Fix: auto-generate a message id when publishing to topics. Both fixes validated: 13/13 integration tests passing, PDF decoder correctly receives and processes document messages through the pipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add full pipeline test that generates a real PDF, processes it through the entire pipeline, and verifies knowledge lands in FalkorDB: - Create test PDF generator using pdf-lib (2-page doc about Acme Corp) - Add testFullPipeline() to integration tests with store verification - Fix FalkorDB client connect() — createClient returns unconnected client in both TriplesStore and TriplesQuery classes Results: PDF decoded (2 pages) → chunked (2 chunks) → extracted (4 relationships) → 16 triples stored in FalkorDB including: alice-johnson → is-a-senior-engineer → acme-corporation cloudsync → uses-aws-for-hosting → amazon-web-services provenance: pages → prov:wasDerivedFrom → source document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…bench pages Phase 9 — four parallel workstreams: - Stream A: 14 Docker entrypoints for containerized deployment - Stream B: Pipeline hardening — robust JSON parsing, LLM retry logic, consumer negative-ack, FalkorDB test import fix - Stream C: Azure OpenAI, OpenAI-compatible, and Mistral LLM providers - Stream D: Workbench Prompts, Token Cost, Knowledge Cores pages + Settings feature switches Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unit tests: Consumer class (7), recursive-splitter (10), parseJsonResponse (11) — 28 total. Docker: add 5 commented LLM provider services, dev compose override, .env.example. Workbench: chat persistence, error boundary, disconnect banner, prompts error handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ll services Client SDK: add .catch() to graphRagStreaming/documentRagStreaming (silent timeout), null-guard JSON.parse in getPrompts/getSystemPrompt/getPrompt. Backend: implement "getvalues" config operation for token costs, null-check createTerm() in FalkorDB triples query, add knowledge-cores service entrypoint and Docker entry, return proper HTTP 400/404 for gateway error responses. Workbench: cancel button + elapsed timer for chat, clear agent spinner on error, flow dialog inline validation, responsive header wrapping, knowledge cores loading timeout, sidebar/page naming consistency, theme toggle indicator. Infrastructure: enable Grafana Explore for viewers, add gateway Prometheus scrape target, fix RAG pipeline dashboard layout (6 panels visible), filter Service Health to configured targets only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… improvements Three QA iterations to convergence (zero issues remaining): Workbench UI: - Connection badge: amber "Connected (no auth)" for unauthenticated state - Theme persistence: restore script in index.html + localStorage sync - Settings About section: add bottom padding so content isn't clipped - Clear messages: cancel in-flight requests when clearing chat - Feature switch labels: proper casing + acronym handling (MCP, LLM) - Token Cost badge: hidden during loading state - ARIA: role="switch", aria-checked on toggles, aria-labels on buttons - ConfigApi: null-safe chaining for getPrompts/getSystemPrompt Grafana dashboards: - Auto-refresh 30s on all 3 dashboards - Panel heights reduced to fit viewport without scrolling - Anonymous role upgraded to Editor for Explore access Infrastructure: - Nginx: DNS resolver with variable-based upstream (prevents crash loop) - Workbench port set to 3002 in .env Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ng, code quality - Fix light mode: theme-aware graph node labels, remove prose-invert for theme-safe markdown, add brand/semantic color overrides for light backgrounds - Add 404 catch-all route redirecting unknown paths to /chat - FalkorDB: add .catch() to connectPromise, add ensureConnected() to all store methods (createLiteral, relateNode, relateLiteral, deleteCollection) - Accessibility: dialog role/aria-modal, toast aria-live, dismiss/zoom/search button aria-labels, close panel aria-label - Lazy-load ForceGraph2D (splits 189KB into separate chunk, main bundle -26%) - Cap conversation localStorage at 200 messages to prevent quota overflow - Fix pnpm test: add --passWithNoTests to cli/mcp packages - Add upload error notification instead of silent catch - Remove unused class-variance-authority dep and dead tabs.tsx component - Add @types/node to flow package devDependencies - Remove stale FIXME comment in messages.ts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…od, delete MCP SDK server, remove ManagedRuntime facades Finishes the remaining EFFECT_NATIVE_REWRITE_PLAN stages in one verified slice: - fastify, @fastify/websocket, commander, zod removed from all package manifests - legacy @modelcontextprotocol/sdk stdio server deleted; effect/unstable/ai McpServer is canonical - no ManagedRuntime or Effect.runPromise program facades remain in production source - gateway server/rpc-contract and client rpc/socket moved onto Effect v4 native http/rpc/socket layers Gates (force-run, no cache): check:tsgo, build, test (96 tests / 11 tasks) all green. Native-class inventory: zero blocking production classes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…class inventory - biome.json (2.4.16, linter-only) wired as "lint" in all six packages - scripts/check-effect-laws.ts: Effect-native law enforcement encoding the adapted beep-effect effect-first/schema-first laws (no native JSON/switch/ sort/fetch/timers, no process.env, no throw new, no Effect.run* outside boundaries, no Schema-suffixed constants, no node:fs/path, AST-based pure-data interface detection per law 38/39) - ratcheting baseline allowlist (95 entries / 290 findings) that must shrink to documented exemptions only; stale counts fail the gate - root lint chains turbo lint + law check + native-class inventory - fix all 163 initial Biome findings: import-type style, templates, two `any`s, ten non-null assertions (librarian getService gate, A.matchRight in atoms, ensureNode returning nodes, main.tsx mount guard) Gates: lint, check:tsgo, build, test (force, 11 tasks) all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor License AgreementThank you for your contribution! Before we can accept it, the following contributor(s) must sign our CLA: Please read the appropriate agreement:
Once you have read the appropriate agreement, post the following as a comment on this PR (copy and paste exactly): The bot will record your signature and update this PR automatically. |
Author
|
Closing: this branch was opened from the wrong fork setup. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root Cause / Notes
The final live smoke exposed a cold-start race: request messages could be published before hand-written global service consumers existed, then skipped by NATS consumers using "new messages only" delivery. The Phase 7 fix sets the relevant request consumers to earliest pending delivery while keeping response consumers unchanged.
A fresh Ollama volume also needed
mxbai-embed-large:latestbefore demo embedding seeding; that model was pulled during validation.GitHub push protection caught historical local-only environment and Playwright capture files while publishing. The PR branch history was scrubbed to remove those tracked artifacts and keep
deploy/.env.exampleas the public environment template.Validation
bun run check:tsgobun scripts/check-effect-laws.ts && bun scripts/inventory-native-classes.tsbunx --bun turbo build lint test --forcebun run --cwd packages/mcp testbun run workbench:qa- 51 passeddocker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --buildfromts/deployafter freshdown -vbun run seedbun run seed:flowsdocker compose -f docker-compose.yml -f docker-compose.dev.yml exec -T ollama ollama pull mxbai-embed-large:latestbun run seed:demoSKIP_LLM=1 bun run test:pipeline- 11 passed, 0 failedbun run check:tsgobun scripts/check-effect-laws.ts && bun scripts/inventory-native-classes.tsgit diff --check origin/master...HEAD -- tsDependency / Boundary Checks
@modelcontextprotocol/sdkremains only in the allowed flow MCP-tool client boundary plus package/lock metadata.