feat: phase 1 foundation — chunking, config, attribution, logging, HNSW migration - #2
Conversation
- Add .agent/ to .gitignore (dev tooling, not part of committed source) - Add cache dirs (.mypy_cache, .ruff_cache, .qodo) and /pr_comments.json - README documents .agent/ in structure map and What Lives Here section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…SW migration - Fix SentenceSplitter: wire chunk_size=512, overlap=64 via transformations=[] - Add .mdx to ingestion file extensions (northflank, railway, render install docs) - Raise similarity_top_k from default 2 → 5 - Add SimilarityPostprocessor with cutoff=0.65 to filter low-confidence hits - Add source attribution: prints [score] file_name for each retrieved node - Move all hardcoded values (models, dims, collection, batch size, top-k) to .env - Replace print() with structured logging; level controlled by LOG_LEVEL env var - Add .env.example template (no secrets) - Add migrations/001_hnsw_index.sql (HNSW cosine index, run once in Supabase) - Add RAG_IMPROVEMENT_REPORT.md and SPRINT_PLAN.md project docs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a substantial OpenClaw documentation corpus and repo onboarding materials intended to serve as an interchangeable knowledge base for the RAG app.
Changes:
- Introduces a large
docs/corpus (concepts, CLI, channels, automation, reference) plus DNS/CNAME. - Adds i18n glossary assets and documentation for the docs-i18n workflow.
- Rewrites root
README.mdand addsAGENTS.mdto define repo structure and operational guidelines.
Reviewed changes
Copilot reviewed 103 out of 425 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/concepts/markdown-formatting.md | New concepts doc describing Markdown IR + chunking/render pipeline. |
| docs/concepts/features.md | New features overview page. |
| docs/concepts/context.md | New concepts doc explaining “context” and inspection commands. |
| docs/concepts/compaction.md | New compaction behavior/config documentation. |
| docs/concepts/architecture.md | New gateway architecture overview doc. |
| docs/concepts/agent.md | New agent runtime/workspace/bootstrap documentation. |
| docs/concepts/agent-workspace.md | New detailed agent workspace and backup guidance. |
| docs/concepts/agent-loop.md | New agent loop lifecycle and streaming doc. |
| docs/cli/webhooks.md | New CLI reference for openclaw webhooks. |
| docs/cli/voicecall.md | New CLI reference for openclaw voicecall. |
| docs/cli/update.md | New CLI reference for update workflow and options. |
| docs/cli/uninstall.md | New CLI reference for uninstall workflow. |
| docs/cli/tui.md | New CLI reference for the terminal UI. |
| docs/cli/system.md | New CLI reference for system events/heartbeat/presence. |
| docs/cli/status.md | New CLI reference for status diagnostics. |
| docs/cli/skills.md | New CLI reference for skills inspection. |
| docs/cli/setup.md | New CLI reference for setup initialization. |
| docs/cli/sessions.md | New CLI reference for session listing/cleanup including JSON shapes. |
| docs/cli/security.md | New CLI reference for security audit/fix checks. |
| docs/cli/secrets.md | New CLI reference for SecretRef lifecycle (reload/audit/configure/apply). |
| docs/cli/sandbox.md | New CLI reference for sandbox commands and usage patterns. |
| docs/cli/reset.md | New CLI reference for reset workflow. |
| docs/cli/qr.md | New CLI reference for iOS pairing QR/setup code generation. |
| docs/cli/plugins.md | New CLI reference for plugin management. |
| docs/cli/pairing.md | New CLI reference for pairing flows. |
| docs/cli/onboard.md | New CLI reference for interactive/non-interactive onboarding. |
| docs/cli/nodes.md | New CLI reference for node management and invocation. |
| docs/cli/node.md | New CLI reference for headless node host command surface. |
| docs/cli/models.md | New CLI reference for model status/list/set/scan and auth probes. |
| docs/cli/memory.md | New CLI reference for memory plugin status/index/search. |
| docs/cli/logs.md | New CLI reference for tailing gateway logs via RPC. |
| docs/cli/hooks.md | New CLI reference for installing/enabling/updating hooks. |
| docs/cli/health.md | New CLI reference for gateway health checks. |
| docs/cli/gateway.md | New CLI reference for gateway run/query/service/discovery commands. |
| docs/cli/doctor.md | New CLI reference for doctor checks and repair behaviors. |
| docs/cli/docs.md | New CLI reference for docs search. |
| docs/cli/dns.md | New CLI reference for wide-area discovery DNS helpers. |
| docs/cli/directory.md | New CLI reference for directory lookups and ID formats. |
| docs/cli/devices.md | New CLI reference for device pairing/token management. |
| docs/cli/dashboard.md | New CLI reference for opening the Control UI. |
| docs/cli/daemon.md | New CLI reference for legacy daemon alias. |
| docs/cli/cron.md | New CLI reference for cron job behavior and edits. |
| docs/cli/configure.md | New CLI reference for interactive configuration wizard. |
| docs/cli/config.md | New CLI reference for config get/set/unset/validate. |
| docs/cli/completion.md | New CLI reference for shell completion generation/install. |
| docs/cli/clawbot.md | New CLI reference for legacy clawbot alias namespace. |
| docs/cli/channels.md | New CLI reference for channels list/status/add/remove/capabilities. |
| docs/cli/browser.md | New CLI reference for browser control, profiles, and relay workflows. |
| docs/cli/backup.md | New CLI reference for backup create/verify behavior and caveats. |
| docs/cli/approvals.md | New CLI reference for exec approvals management. |
| docs/cli/agents.md | New CLI reference for agent lifecycle, bindings, and identity. |
| docs/cli/agent.md | New CLI reference for one-shot agent turns via Gateway. |
| docs/cli/acp.md | New CLI reference for ACP bridge usage and IDE integrations. |
| docs/ci.md | New overview of CI scoping/jobs and local equivalents. |
| docs/channels/zalouser.md | New channel doc for Zalo personal account plugin. |
| docs/channels/zalo.md | New channel doc for Zalo Bot API plugin. |
| docs/channels/troubleshooting.md | New per-channel troubleshooting ladder and failure signatures. |
| docs/channels/tlon.md | New channel doc for Tlon/Urbit plugin including auth and policies. |
| docs/channels/synology-chat.md | New channel doc for Synology Chat plugin setup and security notes. |
| docs/channels/pairing.md | New pairing concept doc (DM pairing + node pairing). |
| docs/channels/nostr.md | New channel doc for Nostr DM plugin setup and policies. |
| docs/channels/nextcloud-talk.md | New channel doc for Nextcloud Talk plugin setup and controls. |
| docs/channels/location.md | New doc describing normalized location parsing and context fields. |
| docs/channels/line.md | New channel doc for LINE plugin configuration and webhook requirements. |
| docs/channels/irc.md | New channel doc for IRC plugin setup and access/mention gating. |
| docs/channels/index.md | New channel index listing supported channel surfaces. |
| docs/channels/group-messages.md | New WhatsApp group behavior and config documentation. |
| docs/channels/channel-routing.md | New routing/session-key documentation and precedence rules. |
| docs/brave-search.md | New Brave Search API setup doc for web_search. |
| docs/automation/webhook.md | New webhook ingress docs including auth and session-key policy. |
| docs/automation/troubleshooting.md | New automation troubleshooting ladder for cron/heartbeat. |
| docs/automation/poll.md | New polls doc covering CLI/RPC/tool differences by channel. |
| docs/automation/auth-monitoring.md | New auth expiry monitoring guidance via models status --check. |
| docs/auth-credential-semantics.md | New credential resolution semantics reference (no frontmatter). |
| docs/CNAME | Adds docs site CNAME. |
| docs/.i18n/glossary.zh-CN.json | Adds zh-CN glossary term mappings. |
| docs/.i18n/glossary.ja-JP.json | Adds ja-JP glossary term mappings. |
| docs/.i18n/README.md | Documents i18n glossary/TM assets and expected formats. |
| README.md | Rewrites repo README for DOC-KB-RAG structure and usage. |
| AGENTS.md | Adds agent guidelines for working within this repo. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 752e92a163
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the RAG system's reliability, retrieval quality, and maintainability. It introduces foundational improvements by optimizing document processing, refining retrieval mechanisms, and centralizing system configurations. The changes aim to make the RAG agent more robust and efficient for handling interchangeable documentation sets. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new RAG (Retrieval-Augmented Generation) system, including comprehensive guidelines for AI agents, a detailed improvement report outlining critical to low-priority issues with a prioritized plan, and a significantly updated README.md to reflect the new project structure and usage. It also adds new documentation for various CLI commands (acp, agent, agents, approvals, backup, browser, channels, clawbot, completion, config, configure, cron, daemon, dashboard, devices, directory, dns, docs, gateway, health, hooks, logs, memory, message, models, node, nodes, onboard, pairing, plugins, qr, reset, sandbox, secrets, security, sessions, setup, skills, status, system, tui, uninstall, update, voicecall, webhooks), agent concepts (agent loop, workspace, compaction, context, features, markdown formatting, models, multi-agent, oauth, queue, sandbox, secrets, session, session pruning, streaming, system prompt, timezone, token use, tools, usage tracking), channel specifics (BlueBubbles, broadcast groups, channel routing, Discord, Feishu, Google Chat, group messages, groups, iMessage, index, IRC, LINE, location, Matrix, Mattermost, Microsoft Teams, Nextcloud Talk, Nostr, pairing, Signal, Slack, Synology Chat, Telegram, Tlon, troubleshooting, Twitch, WhatsApp, Zalo, Zalo Personal), and various assets like SVG icons and glossary files for internationalization. Additionally, it updates the .gitignore to exclude new cache and development files, and adds a SPRINT_PLAN.md for future RAG system enhancements.
- docs/cli/sandbox.md: convert read_when from scalar string to YAML list - docs/cli/acp.md: fix article — 'a OpenClaw' → 'an OpenClaw' - docs/.i18n/README.md: wrap glossary JSON example in array (text already said "array of entries") - docs/auth-credential-semantics.md: add missing frontmatter (summary, read_when) - rag-agent/.env.example: align API key placeholder with runtime sentinel (YOUR_API_KEY_HERE) - rag-agent/migrations/001_hnsw_index.sql: add note + parameterized DO block for non-default COLLECTION_NAME Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@copilot-pull-request-reviewer @chatgpt-codex-connector[bot] @gemini-code-assist[bot] /review I have addressed all feedback. Summary of fixes:
|
|
Codex Review: Didn't find any major issues. You're on a roll. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive documentation for the DOC-KB-RAG system, including agent guidelines, a detailed RAG improvement report outlining critical to low-priority issues with fixes and research findings, and a significantly updated README.md for better clarity and structure. It also adds extensive documentation for automation features like cron jobs, heartbeat scheduling, webhook integration (including Gmail Pub/Sub), and various chat channels (Discord, Feishu, Google Chat, iMessage, IRC, LINE, Matrix, Mattermost, MS Teams, Nextcloud Talk, Nostr, Signal, Slack, Telegram, Tlon, Twitch, WhatsApp, Zalo, Zalo Personal), covering their setup, access control, capabilities, and troubleshooting. New CLI command documentation is provided for acp, agent, agents, approvals, backup, browser, channels, clawbot, completion, config, configure, cron, daemon, dashboard, devices, directory, dns, docs, doctor, gateway, health, hooks, logs, memory, message, models, node, nodes, onboard, pairing, plugins, qr, reset, sandbox, secrets, security, sessions, setup, skills, status, system, tui, uninstall, update, and webhooks. Additionally, core concepts like agent loop, agent workspace, architecture, context, compaction, markdown formatting, multi-agent routing, OAuth, queueing, sandbox, secrets management, session management, session pruning, streaming, system prompt, timezone handling, token use, and usage tracking are thoroughly documented. The .gitignore file is updated to exclude new cache and agent-related directories, and various SVG assets for logos and showcase examples are added.
Summary
chunk_size=512, chunk_overlap=64viatransformations=[]— was imported but never used.mdxsupport:install/northflank.mdx,install/railway.mdx,install/render.mdxare now ingestedsimilarity_top_kfrom default 2 → 5, so the retriever considers more candidate chunksSimilarityPostprocessorwith cutoff0.65— prevents LLM being called when no chunk is relevant[score] file_namefor each retrieved node.env; add.env.exampletemplateprint()withlogging; level controlled viaLOG_LEVELenv varmigrations/001_hnsw_index.sql— run once in Supabase to fix the full-table-scan warningFiles changed
rag-agent/ingest.pyrag-agent/query.pyrag-agent/.env.examplerag-agent/migrations/001_hnsw_index.sqlRAG_IMPROVEMENT_REPORT.mdSPRINT_PLAN.mdTest plan
python rag-agent/ingest.py— confirm chunk count reflects 512-token splits and .mdx files appearpython rag-agent/query.py "how do I configure multi agents on openclaw"— confirm answer + sources printedLOG_LEVEL=DEBUG— confirm verbose output🤖 Generated with Claude Code