Skip to content

Commit 798ae16

Browse files
jpr5jpr5
authored
Add agentic retrieval: session state, vector grep, virtual files, related command, telemetry (#9)
## Summary Evolves mcp-docs bash tools from static in-memory filesystem exploration into an integrated agentic retrieval system that matches and exceeds Mintlify's ChromaFS approach. All features are opt-in via the `bash:` config subsection — existing configs work unchanged. - **Session state**: Persistent CWD tracking across commands. `cd /docs && ls` works across separate tool calls. Path validation rejects nonexistent directories. - **Vector-backed grep**: Configurable 3-pass search (semantic embeddings + ILIKE text + dedup) with graceful degradation when search infrastructure is unavailable. Falls back to in-memory grep for bash-only configs. - **Virtual files**: Auto-generated `INDEX.md` (file listing) and `SEARCH_TIPS.md` (usage guidance) injected into the virtual filesystem at startup. - **Cross-paradigm hints**: `related /path/to/file` command finds semantically similar files across all sources. Grep misses suggest companion search tools. - **File metadata**: `buildFileMetadata` and `formatLsLong` for `ls -l` style output with sizes and line counts. - **Workspace tracker**: Per-session writable `/workspace/` directory with 1MB size cap. - **Telemetry hooks**: Tracks file access, grep misses, and commands with buffer overflow protection (10K cap). - **Webhook refresh**: Atomic bash instance swap when sources are reindexed via GitHub webhooks. - **154 tests** covering all features, edge cases, and error paths. Full design: [Proposal on Notion](https://www.notion.so/33a3aa38185281f2b5ebdf52a1a35108) ## Test plan - [x] All 154 tests pass (`npx vitest run`) - [x] TypeScript compiles clean (`npx tsc --noEmit`) - [x] 2-round CR loop with 7 agents — 7 bugs found and fixed, round 2 clean - [ ] Deploy to Railway staging and verify bash tools work with new config options - [ ] Verify `related` command returns meaningful results against CopilotKit docs - [ ] Verify vector grep returns results for common queries (useAction, streaming, etc.) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 6f71b7c + c4e49fb commit 798ae16

42 files changed

Lines changed: 2771 additions & 57 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/deploy-pages.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
name: Deploy to GitHub Pages
2+
on:
3+
push:
4+
branches: [main, master]
5+
paths: ['docs/**']
6+
workflow_dispatch:
7+
permissions:
8+
contents: read
9+
pages: write
10+
id-token: write
11+
concurrency:
12+
group: pages
13+
cancel-in-progress: true
14+
jobs:
15+
deploy:
16+
runs-on: ubuntu-latest
17+
environment:
18+
name: github-pages
19+
url: ${{ steps.deploy.outputs.page_url }}
20+
steps:
21+
- uses: actions/checkout@v4
22+
- uses: actions/configure-pages@v4
23+
- uses: actions/upload-pages-artifact@v3
24+
with:
25+
path: docs
26+
- id: deploy
27+
uses: actions/deploy-pages@v4

‎.github/workflows/publish-docker.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: Publish Docker
2+
on:
3+
push:
4+
tags: ['v*']
5+
jobs:
6+
docker:
7+
runs-on: ubuntu-latest
8+
permissions:
9+
contents: read
10+
packages: write
11+
steps:
12+
- uses: actions/checkout@v4
13+
- uses: docker/setup-qemu-action@v3
14+
- uses: docker/setup-buildx-action@v3
15+
- uses: docker/login-action@v3
16+
with:
17+
registry: ghcr.io
18+
username: ${{ github.actor }}
19+
password: ${{ secrets.GITHUB_TOKEN }}
20+
- uses: docker/build-push-action@v5
21+
with:
22+
push: true
23+
tags: |
24+
ghcr.io/copilotkit/pathfinder:latest
25+
ghcr.io/copilotkit/pathfinder:${{ github.ref_name }}
26+
platforms: linux/amd64,linux/arm64
27+
cache-from: type=gha
28+
cache-to: type=gha,mode=max

‎.github/workflows/publish-release.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Publish Release
2+
on:
3+
push:
4+
branches: [main, master]
5+
jobs:
6+
release:
7+
runs-on: ubuntu-latest
8+
permissions:
9+
contents: write
10+
id-token: write
11+
steps:
12+
- uses: actions/checkout@v4
13+
with:
14+
fetch-depth: 0
15+
- uses: actions/setup-node@v4
16+
with:
17+
node-version: 22
18+
registry-url: https://registry.npmjs.org
19+
- name: Get version
20+
id: version
21+
run: echo "version=$(node -p 'require("./package.json").version')" >> $GITHUB_OUTPUT
22+
- name: Check if tag exists
23+
id: check_tag
24+
run: |
25+
if git rev-parse "v${{ steps.version.outputs.version }}" >/dev/null 2>&1; then
26+
echo "exists=true" >> $GITHUB_OUTPUT
27+
else
28+
echo "exists=false" >> $GITHUB_OUTPUT
29+
fi
30+
- name: Check if npm version published
31+
id: check_npm
32+
if: steps.check_tag.outputs.exists == 'false'
33+
run: |
34+
set +e
35+
OUTPUT=$(npm view "@copilotkit/pathfinder@${{ steps.version.outputs.version }}" version 2>&1)
36+
EXIT_CODE=$?
37+
set -e
38+
if [ $EXIT_CODE -eq 0 ]; then
39+
echo "published=true" >> $GITHUB_OUTPUT
40+
elif echo "$OUTPUT" | grep -qi "404\|not found\|is not in this registry"; then
41+
echo "published=false" >> $GITHUB_OUTPUT
42+
else
43+
echo "::error::npm view failed unexpectedly: $OUTPUT"
44+
exit 1
45+
fi
46+
- name: Install latest npm (OIDC requires >= 11.5.1)
47+
if: steps.check_tag.outputs.exists == 'false' && steps.check_npm.outputs.published == 'false'
48+
run: npm install -g npm@latest
49+
- name: Install and build
50+
if: steps.check_tag.outputs.exists == 'false' && steps.check_npm.outputs.published == 'false'
51+
run: npm ci && npm run build
52+
- name: Publish to npm (OIDC trusted publishing)
53+
id: npm_publish
54+
if: steps.check_tag.outputs.exists == 'false' && steps.check_npm.outputs.published == 'false'
55+
run: npm publish --access public
56+
- name: Create tag and release
57+
if: steps.check_tag.outputs.exists == 'false' && (steps.check_npm.outputs.published == 'true' || steps.npm_publish.outcome == 'success')
58+
env:
59+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
60+
run: |
61+
git tag "v${{ steps.version.outputs.version }}"
62+
git push origin "v${{ steps.version.outputs.version }}"
63+
gh release create "v${{ steps.version.outputs.version }}" --generate-notes

‎.github/workflows/unreleased-check.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: Unreleased Changes Check
2+
on:
3+
push:
4+
branches: [main, master]
5+
jobs:
6+
check:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- uses: actions/checkout@v4
10+
with:
11+
fetch-depth: 0
12+
- name: Check for unreleased changes
13+
run: |
14+
LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "")
15+
if [ -z "$LATEST_TAG" ]; then
16+
echo "No tags found, skipping"
17+
exit 0
18+
fi
19+
COMMITS=$(git rev-list "$LATEST_TAG"..HEAD -- src/ | wc -l | tr -d ' ')
20+
if [ "$COMMITS" -gt "0" ]; then
21+
echo "::warning::$COMMITS source commits since $LATEST_TAG — consider releasing"
22+
fi

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,5 @@ dist/
44
*.local
55
pgdata/
66
tmp/
7+
.superpowers/
8+
.claude/

‎CHANGELOG.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Changelog
2+
3+
## 1.1.0
4+
5+
### Minor Changes
6+
7+
- Rename project from mcp-docs to Pathfinder
8+
- Add agentic retrieval: session state, vector grep, virtual files, related command, telemetry
9+
- Add configurable bash tool options (grep_strategy, workspace, virtual_files)
10+
- Add Pathfinder landing page and documentation site
11+
- Add Mintlify migration tutorial
12+
- Add GitHub Actions for Pages deployment, releases, and Docker publishing
13+
- Add versioning infrastructure with CHANGELOG
14+
15+
## 1.0.0
16+
17+
### Initial Release
18+
19+
- Semantic search over documentation and code via pgvector + OpenAI embeddings
20+
- Bash tool filesystem exploration via just-bash virtual filesystem
21+
- Feedback collection tools with YAML-defined schemas
22+
- Config-driven via pathfinder.yaml
23+
- Webhook-triggered reindexing from GitHub push events
24+
- Nightly auto-reindex on configurable schedule
25+
- Docker deployment support
26+
- MIT License

‎Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,5 @@ WORKDIR /app
2222
COPY package.json package-lock.json ./
2323
RUN npm ci --omit=dev
2424
COPY --from=build /app/dist/ ./dist/
25-
COPY mcp-docs.yaml ./mcp-docs.yaml
25+
COPY pathfinder.yaml ./pathfinder.yaml
2626
CMD ["node", "dist/index.js"]

‎README.md‎

Lines changed: 48 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
1-
# mcp-docs
1+
# Pathfinder
22

33
A self-hosted MCP server that provides semantic search over your documentation and code. Configure it with a YAML file, deploy with Docker, and give your AI coding agents instant access to your project's knowledge.
44

55
## How It Works
66

7-
mcp-docs indexes your GitHub repositories — documentation (Markdown/MDX) and source code — into a PostgreSQL vector database using OpenAI embeddings. It exposes configurable search tools via the [Model Context Protocol (MCP)](https://modelcontextprotocol.io), so AI agents like Claude Code can search your docs and code semantically.
7+
Pathfinder indexes your GitHub repositories — documentation (Markdown/MDX) and source code — into a PostgreSQL vector database using OpenAI embeddings. It exposes configurable search tools via the [Model Context Protocol (MCP)](https://modelcontextprotocol.io), so AI agents like Claude Code can search your docs and code semantically.
88

99
## Quick Start
1010

1111
1. **Clone and configure:**
1212
```bash
13-
git clone https://github.com/CopilotKit/mcp-docs.git
14-
cd mcp-docs
15-
cp mcp-docs.example.yaml mcp-docs.yaml # edit for your project
16-
cp .env.example .env # add your OPENAI_API_KEY
13+
git clone https://github.com/CopilotKit/pathfinder.git
14+
cd pathfinder
15+
cp pathfinder.example.yaml pathfinder.yaml # edit for your project
16+
cp .env.example .env # add your OPENAI_API_KEY
1717
```
1818

1919
2. **Start the server:**
@@ -37,7 +37,7 @@ mcp-docs indexes your GitHub repositories — documentation (Markdown/MDX) and s
3737

3838
## Configuration
3939

40-
All configuration lives in `mcp-docs.yaml`. See [mcp-docs.example.yaml](mcp-docs.example.yaml) for a minimal starting point.
40+
All configuration lives in `pathfinder.yaml`. See [pathfinder.example.yaml](pathfinder.example.yaml) for a minimal starting point.
4141

4242
### Sources
4343

@@ -113,6 +113,44 @@ tools:
113113
114114
Each field in `schema` supports `type` (`string`, `number`, or `enum`), an optional `description` (shown to the agent), `required` (defaults to false), and `values` (required for `enum` fields). The validated input is written as JSONB to the `collected_data` table along with the tool name and a timestamp.
115115

116+
### Bash Tool Options
117+
118+
Bash tools expose source files as a read-only virtual filesystem that agents can explore with standard commands (`find`, `grep`, `cat`, `ls`, `head`). Several options control behavior:
119+
120+
```yaml
121+
tools:
122+
- name: explore-docs
123+
type: bash
124+
description: "Explore documentation files"
125+
sources: [docs]
126+
bash:
127+
session_state: true # Persistent CWD across commands (default: false)
128+
grep_strategy: hybrid # memory | vector | hybrid — enables qmd semantic search (default: memory, no qmd)
129+
virtual_files: true # Auto-generate INDEX.md, SEARCH_TIPS.md (default: false)
130+
```
131+
132+
- **session_state**: When enabled, `cd` persists across commands within a session. Agents can run `cd /docs` in one tool call and then `ls` or `cat file.md` in the next without repeating the path.
133+
- **grep_strategy**: Controls whether the `qmd` semantic search command is available. `memory` uses pure in-memory regex only (no `qmd`). `vector` or `hybrid` enable the `qmd` command, which performs semantic search via embeddings plus text `ILIKE`. The `vector` and `hybrid` modes require an `embedding` config block.
134+
- **virtual_files**: Auto-generates `/INDEX.md` (file listing with descriptions) and `/SEARCH_TIPS.md` (usage guidance) at the root of the virtual filesystem.
135+
136+
Agents can also run the `related` command inside bash tools to find semantically similar files across all mounted sources:
137+
138+
```bash
139+
related /docs/concepts/coagents.mdx
140+
```
141+
142+
This returns a ranked list of files from any source that are semantically related to the given file, useful for discovering cross-references between documentation and code.
143+
144+
When `grep_strategy` is set to `vector` or `hybrid`, agents can use the `qmd` command for semantic search:
145+
146+
```bash
147+
qmd "how do I configure authentication"
148+
```
149+
150+
This performs a 2-pass search (semantic embeddings + text ILIKE) with dedup and filtering, and returns file:line:content results. Standard `grep` is never intercepted — it always works with standard flags as agents expect.
151+
152+
**Note:** The virtual filesystem is read-only and shared across all MCP sessions for a given tool. Content refreshes on webhook or server restart.
153+
116154
### Built-in Chunker Types
117155

118156
| Type | Best For | Splits On |
@@ -145,7 +183,7 @@ The simplest way to run in production:
145183

146184
1. **Configure:**
147185
```bash
148-
cp mcp-docs.example.yaml mcp-docs.yaml # edit for your project
186+
cp pathfinder.example.yaml pathfinder.yaml # edit for your project
149187
```
150188

151189
2. **Set environment variables** in `.env`:
@@ -173,7 +211,7 @@ The server automatically indexes on first boot and runs a nightly reindex at the
173211

174212
For real-time re-indexing on push:
175213

176-
1. Add webhook config to `mcp-docs.yaml`:
214+
1. Add webhook config to `pathfinder.yaml`:
177215
```yaml
178216
webhook:
179217
repo_sources:
@@ -238,7 +276,7 @@ npx tsx scripts/test-path-filter.ts
238276
| `DATABASE_URL` | Yes | PostgreSQL connection string |
239277
| `GITHUB_WEBHOOK_SECRET` | No | HMAC secret for webhook verification |
240278
| `GITHUB_TOKEN` | No | GitHub token for private repos |
241-
| `MCP_DOCS_CONFIG` | No | Path to config file (default: `./mcp-docs.yaml`) |
279+
| `PATHFINDER_CONFIG` | No | Path to config file (default: `./pathfinder.yaml`) |
242280
| `PORT` | No | Server port (default: `3001`) |
243281

244282
## License

‎docker-compose.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,14 @@ services:
2626
OPENAI_API_KEY: ${OPENAI_API_KEY}
2727
GITHUB_TOKEN: ${GITHUB_TOKEN:-}
2828
GITHUB_WEBHOOK_SECRET: ${GITHUB_WEBHOOK_SECRET:-local-dev-secret}
29-
MCP_DOCS_CONFIG: ${MCP_DOCS_CONFIG:-./mcp-docs.yaml}
29+
PATHFINDER_CONFIG: ${PATHFINDER_CONFIG:-./pathfinder.yaml}
3030
PORT: 3001
3131
NODE_ENV: development
3232
LOG_LEVEL: debug
3333
volumes:
3434
- ./src:/app/src
3535
- ./scripts:/app/scripts
36-
- ./mcp-docs.yaml:/app/mcp-docs.yaml
36+
- ./pathfinder.yaml:/app/pathfinder.yaml
3737
depends_on:
3838
db:
3939
condition: service_healthy

‎docs/.nojekyll‎

Whitespace-only changes.

0 commit comments

Comments
 (0)