Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,18 @@ ANTHROPIC_API_KEY=sk-ant-...
# GITLAB_TOKEN=glpat-...
# GITLAB_API_URL=https://gitlab.example.com/api/v4

# Notion (knowledge-miner.json: syncntn)
# Notion — only if you add a Notion MCP server block back to
# knowledge-miner.json (none ships by default; see TRIUMVIRATE-SETUP.md Step 6)
# NOTION_STORAGE_URL=http://localhost:8000
# NOTION_WORKSPACE_ID=...

# Scribe — audio/video transcription (knowledge-miner.json: scribe)
# GEMINI_API_KEY=... # REQUIRED if you keep the scribe server (recipe uses bare ${GEMINI_API_KEY}); powers transcription
# NOTION_API_KEY=... # optional (recipe uses ${NOTION_API_KEY:-}); only scribe--scribe_notion_page needs it
# SCRIBE_GLOSSARY_URL=... # optional (recipe uses ${SCRIBE_GLOSSARY_URL:-}); unset = transcribe without a glossary
# Web UI credentials (triumvirate.json: webui). Defaults to admin:admin —
# CHANGE THESE for anything reachable beyond your own machine.
# WEBUI_USERNAME=admin
# WEBUI_PASSWORD=admin

# Scribe — audio/video transcription. Only if you add a scribe block back to
# knowledge-miner.json (none ships by default; see TRIUMVIRATE-SETUP.md Step 6)
# GEMINI_API_KEY=... # required by the scribe block (bare ${GEMINI_API_KEY}); powers transcription
# NOTION_API_KEY=... # optional (block uses ${NOTION_API_KEY:-}); only scribe--scribe_notion_page needs it
# SCRIBE_GLOSSARY_URL=... # optional (block uses ${SCRIBE_GLOSSARY_URL:-}); unset = transcribe without a glossary
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,33 @@

## Unreleased

### Changed

- **The public triumvirate recipes boot from a fresh clone.**
`knowledge-miner.json` no longer ships a `syncntn` (Notion) block pointing at
an org-internal adapter that isn't publicly available — with `NOTION_*` env
vars unset the block failed recipe load, and with them set it died at spawn
on the dangling `../syncntn` path. The `scribe` block is dropped for the
same reason: it hard-required `GEMINI_API_KEY` and a `../scribe-mcp`
sibling checkout, neither mentioned anywhere in the setup guides — a
guide-following fresh install always got a crashed miner. Notion and
Scribe are now add-a-block opt-ins, documented in SETUP.md and
TRIUMVIRATE-SETUP.md (the miner prompt's tool-name contracts are
unchanged). `triumvirate.json` declares
webui Basic-Auth defaulting to `admin`/`admin` (override via
`WEBUI_USERNAME` / `WEBUI_PASSWORD` in `.env`) instead of bare
`"webui": true`, which the non-loopback bind guard refuses to start.

### Fixed

- **`mcpServers.<id>.source` accepts cook's npm registry form.**
`validateRecipe` demanded `source.url`, but connectome-cook's source grammar
also has `{ "npm": "pkg@version" }` — which the shipped knowledge-miner
recipe uses for its gitlab server, so that recipe failed to load
(`mcpServers.gitlab.source.url must be a non-empty string`). Exactly one of
`url` / `npm` is now required; the field remains build-tooling metadata,
ignored at runtime.

### Added

- **Standing autobiographical production target.** Recipes may set `agent.strategy.productionBudgetTokens` to keep the summary forest deep enough for a later live context-budget descent without a fold storm. This is a context-token target passed through to Context Manager, not a provider-spend ceiling; omission preserves Context Manager defaults.
Expand Down
16 changes: 11 additions & 5 deletions recipes/SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,17 +69,17 @@ Works with both gitlab.com and self-hosted GitLab instances.

No separate installation needed — the recipe uses `npx` to run `@zereight/mcp-gitlab` on demand.

### Notion (optional, via an MCP server)
### Notion (optional, via an MCP server — not included by default)

If you want the agent to read your Notion workspace, point the recipe at any MCP server that exposes Notion search and page-read tools. The recipe's template entry is named `syncntn` after the particular Notion MCP adapter it was developed against, but the key is just a label — any Notion MCP server works, as long as its exposed tool names match what the system prompt references (`syncntn--search_pages`, `syncntn--get_page_markdown`, and friends). If your server uses different tool names, either rename the MCP key and update the prompt, or skip Notion entirely by removing the block.
The recipe ships **without** a Notion server: the adapter its prompt was developed against (`syncntn`) is not publicly available. If you want the agent to read your Notion workspace, add an `mcpServers` entry pointing at any MCP server that exposes Notion search and page-read tools. The entry name `syncntn` is just a label — any Notion MCP server works, as long as its exposed tool names match what the system prompt references (`syncntn--search_pages`, `syncntn--get_page_markdown`, and friends). If your server uses different tool names, either name the MCP key `syncntn` and update the prompt, or accept that the agent will discover the tools under whatever names they export.

Typical setup:

1. Install and start your Notion MCP server somewhere the recipe can launch it.
2. Note any configuration it needs (API credentials, workspace ID, storage URL).
3. Fill those values into the recipe's `syncntn` env block in Step 3 below.
3. Add a `syncntn` block with those values to the recipe in Step 3 below.

Don't have a Notion MCP server? Remove the `syncntn` entry from the recipe — the agent will adapt and work with whatever sources remain.
Don't have a Notion MCP server? Skip this — the agent adapts and works with whatever sources remain.

### DuckDuckGo web search (optional, but enabled by default)

Expand All @@ -98,6 +98,10 @@ cd ../connectome-host

The recipe expects the entry-point script at `../duckduckgo-mcp-server/.venv/bin/duckduckgo-mcp-server`. No API key needed. Don't want public-web access? Remove the `ddg` block from the recipe.

### Scribe — audio/video transcription (optional, not included by default)

The miner's prompt also knows how to drive [`dariakroshka/scribe-mcp`](https://github.com/dariakroshka/scribe-mcp) for transcribing recordings. It needs a Gemini API key (media is uploaded to Google's Gemini API) and a sibling checkout, so the shipped recipe omits it. To enable: clone scribe-mcp as a sibling of `connectome-host/`, run `bun install` in it, add a `scribe` block under `mcpServers` (see the [Triumvirate guide's Scribe section](./TRIUMVIRATE-SETUP.md#scribe--audiovideo-transcription-optional-off-by-default) for the exact JSON), and set `GEMINI_API_KEY` in `.env`.

## Step 3: Configure the recipe

Copy the template recipe and fill in your credentials:
Expand All @@ -120,8 +124,10 @@ Edit `my-recipe.json` and replace the placeholder values in `mcpServers`:
"ZULIP_RC_PATH": "./.zuliprc" // path to your .zuliprc
}
},
// Optional — NOT in the shipped recipe. Add only if you set up a
// Notion MCP server (see Step 2 above):
"syncntn": {
"command": "../syncntn/services/mcp/start_mcp_local.sh",
"command": "../your-notion-mcp/start.sh",
"env": {
"STORAGE_URL": "http://localhost:8000",
"WORKSPACE_ID": "YOUR_WORKSPACE_ID" // <-- replace this
Expand Down
82 changes: 68 additions & 14 deletions recipes/TRIUMVIRATE-SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,23 +138,26 @@ ANTHROPIC_API_KEY=sk-ant-...
ZULIP_CHANNEL=your-channel-name
```

Optional — **only** if you want the miner to extract from those sources (otherwise remove the relevant `mcpServers` block from `recipes/knowledge-miner.json` and skip these):
Optional — **only** if you want the miner to extract from GitLab (otherwise remove the `gitlab` block from `recipes/knowledge-miner.json` and skip these):

```ini
# GitLab (knowledge-miner.json: gitlab)
GITLAB_TOKEN=glpat-...
GITLAB_API_URL=https://gitlab.example.com/api/v4
```

# Notion (knowledge-miner.json: syncntn)
NOTION_STORAGE_URL=http://localhost:8000
NOTION_WORKSPACE_ID=...
Optional — the conductor's web UI is protected by Basic-Auth that defaults to `admin` / `admin`. Fine for a laptop; **change it** the moment the machine is reachable by anyone else:

```ini
WEBUI_USERNAME=...
WEBUI_PASSWORD=...
```

Bun auto-loads `.env`, so nothing else to wire. If a recipe references a `${VAR}` you haven't set, the child's startup will fail with a clear message telling you which variable is missing and which recipe referenced it.

## Step 6: Decide which data sources you want

The miner child uses `recipes/knowledge-miner.json`, which comes pre-wired to talk to **Zulip, Notion, GitLab, and DuckDuckGo** (public web). The recipe itself references credentials via `${VAR}` placeholders — you don't edit the recipe to fill in secrets; you set the env vars in Step 5 and the framework substitutes at load time.
The miner child uses `recipes/knowledge-miner.json`, which comes pre-wired to talk to **Zulip, GitLab, and DuckDuckGo** (public web); **Notion** and **Scribe** (audio/video transcription) connections can be added — see their subsections below. The recipe itself references credentials via `${VAR}` placeholders — you don't edit the recipe to fill in secrets; you set the env vars in Step 5 and the framework substitutes at load time.

You decide which sources are active by whether you **set the matching env vars** and whether you **keep the matching mcpServers block in the recipe**.

Expand All @@ -175,17 +178,29 @@ No separate install — the recipe runs `npx @zereight/mcp-gitlab` on demand.

To disable: remove the `gitlab` block from `recipes/knowledge-miner.json`. If you leave it in but don't set the env vars, the child will fail to start with a message like `Recipe "recipes/knowledge-miner.json" references environment variable ${GITLAB_TOKEN} which is not set.` — that's the system telling you to either fill in the env var or delete the block.

### Notion (optional)
### Notion (optional, off by default)

The recipe ships **without** a Notion server — the adapter it was developed against (`syncntn`) is not publicly available, so a default block would only produce a startup failure. The miner's system prompt still describes the `syncntn--*` tools; the agent simply won't have them until you wire a server in.

To enable: install a Notion MCP server (the recipe's template is named `syncntn`; any MCP server with matching tool names works — see [SETUP.md → Notion](./SETUP.md#notion-optional-via-an-mcp-server) for selection caveats) and set:
To enable: install a Notion MCP server (any server whose tool names match what the prompt references — see [SETUP.md → Notion](./SETUP.md#notion-optional-via-an-mcp-server) for selection caveats), then add a block to `recipes/knowledge-miner.json` under `mcpServers`:

```jsonc
"syncntn": {
"command": "../your-notion-mcp/start.sh", // however your server is launched
"env": {
"STORAGE_SERVICE_URL": "${NOTION_STORAGE_URL}",
"WORKSPACE_ID": "${NOTION_WORKSPACE_ID}"
}
}
```

and set the referenced vars in `.env`:

```ini
NOTION_STORAGE_URL=http://localhost:8000
NOTION_WORKSPACE_ID=...
```

To disable: remove the `syncntn` block from `recipes/knowledge-miner.json`. Same behavior as above — unset env + kept block = startup failure with a clear message.

### DuckDuckGo web search (optional, enabled by default)

The miner is wired to [`nickclyde/duckduckgo-mcp-server`](https://github.com/nickclyde/duckduckgo-mcp-server) as `ddg`. No API key — DuckDuckGo's public HTML search, scraped at request time.
Expand All @@ -207,14 +222,51 @@ Web hits get tagged `[WEB: <url>]` in mined reports — internal `[SRC]` always

To disable: remove the `ddg` block from `recipes/knowledge-miner.json`. The agent will skip the public web.

### Scribe — audio/video transcription (optional, off by default)

The miner's prompt knows how to use [`dariakroshka/scribe-mcp`](https://github.com/dariakroshka/scribe-mcp) to transcribe recordings (via Google's Gemini API — media leaves your machine). The recipe ships without the block: it requires a Gemini API key and a sibling checkout, neither of which a demo should demand.

To enable: install the server as a sibling of `connectome-host/`:

```bash
cd ..
git clone https://github.com/dariakroshka/scribe-mcp.git
cd scribe-mcp
bun install
cd ../connectome-host
```

then add this block to `recipes/knowledge-miner.json` under `mcpServers`:

```jsonc
"scribe": {
"command": "bun",
"args": ["../scribe-mcp/src/index.ts"],
"env": {
"GEMINI_API_KEY": "${GEMINI_API_KEY}",
"NOTION_API_KEY": "${NOTION_API_KEY:-}", // only scribe--scribe_notion_page needs it
"SCRIBE_GLOSSARY_PATH": "./input/glossary.txt",
"SCRIBE_GLOSSARY_URL": "${SCRIBE_GLOSSARY_URL:-}" // optional domain glossary
},
"source": {
"url": "https://github.com/dariakroshka/scribe-mcp.git",
"install": { "runtime": "bun", "run": "bun install --frozen-lockfile" },
"inContainer": { "path": "/scribe-mcp" }
}
}
```

and set `GEMINI_API_KEY=...` in `.env`.

### Summary table

| Source | Keep the block in recipe? | Env vars needed |
| Source | Block in recipe? | Env vars needed |
|---|---|---|
| Zulip | Yes | (configured via `.zuliprc`, no `${VAR}`) |
| GitLab | Yes if using, remove otherwise | `GITLAB_TOKEN`, `GITLAB_API_URL` |
| Notion | Yes if using, remove otherwise | `NOTION_STORAGE_URL`, `NOTION_WORKSPACE_ID` |
| DuckDuckGo | Yes if you want public web search, remove otherwise | none (no API key) |
| Zulip | Yes (default) | (configured via `.zuliprc`, no `${VAR}`) |
| GitLab | Yes (default) — remove if not using | `GITLAB_TOKEN`, `GITLAB_API_URL` |
| Notion | **No** — add a `syncntn` block if using | `NOTION_STORAGE_URL`, `NOTION_WORKSPACE_ID` |
| DuckDuckGo | Yes (default) — remove if not using | none (no API key) |
| Scribe | **No** — add a `scribe` block if using | `GEMINI_API_KEY` (+ optional `NOTION_API_KEY`, `SCRIBE_GLOSSARY_URL`) |

### Tweaks you can still make to the recipe files

Expand All @@ -237,6 +289,8 @@ What you'll see:
3. Press **Tab** a couple of times to cycle through view modes. One of them is the **process fleet** view — it lists the three children and their status. All three should reach **ready** (green). If any show **crashed** (red), jump to Troubleshooting.
4. Ask the conductor `are all three ready?` — it'll run `fleet--list` and confirm. This also serves as a quick "am I set up correctly" smoke test.

The conductor also serves a **web UI** on port 7340 (all interfaces, Basic-Auth). Credentials default to `admin` / `admin` unless you set `WEBUI_USERNAME` / `WEBUI_PASSWORD` in `.env` — see Step 5. Open `http://localhost:7340` to watch the fleet from a browser.

### The four view modes

Press **Tab** to cycle between views. Press **Ctrl+F** to jump straight to the process fleet view from anywhere.
Expand Down
30 changes: 0 additions & 30 deletions recipes/knowledge-miner.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,6 @@
}
}
},
"syncntn": {
"command": "../syncntn/services/mcp/start_mcp_local.sh",
"env": {
"STORAGE_SERVICE_URL": "${NOTION_STORAGE_URL}",
"WORKSPACE_ID": "${NOTION_WORKSPACE_ID}"
}
},
"gitlab": {
"command": "npx",
"args": [
Expand Down Expand Up @@ -66,29 +59,6 @@
"path": "/duckduckgo-mcp-server"
}
}
},
"scribe": {
"command": "bun",
"args": [
"../scribe-mcp/src/index.ts"
],
"env": {
"GEMINI_API_KEY": "${GEMINI_API_KEY}",
"NOTION_API_KEY": "${NOTION_API_KEY:-}",
"SCRIBE_GLOSSARY_PATH": "./input/glossary.txt",
"SCRIBE_GLOSSARY_URL": "${SCRIBE_GLOSSARY_URL:-}"
},
"source": {
"url": "https://github.com/dariakroshka/scribe-mcp.git",
"ref": "828ae8ba6ea83ce439ce6ba020ed0223cf096f82",
"install": {
"runtime": "bun",
"run": "bun install --frozen-lockfile"
},
"inContainer": {
"path": "/scribe-mcp"
}
}
}
},
"modules": {
Expand Down
7 changes: 6 additions & 1 deletion recipes/triumvirate.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,12 @@
"retrieval": false,
"wake": true,
"workspace": false,
"webui": true,
"webui": {
"basicAuth": {
"username": "${WEBUI_USERNAME:-admin}",
"password": "${WEBUI_PASSWORD:-admin}"
}
},
"fleet": {
"children": [
{
Expand Down
30 changes: 23 additions & 7 deletions src/recipe.ts
Original file line number Diff line number Diff line change
Expand Up @@ -271,13 +271,21 @@ export interface RecipeMcpServer {

/**
* How to obtain and install an MCP server at deploy time. Consumed by
* build tooling like connectome-cook. All fields optional-at-the-schema-
* layer except `url`; tools may require more depending on the install
* pattern they're generating.
* build tooling like connectome-cook. Exactly one of `url` (git form) or
* `npm` (registry form) must be set; tools may require more depending on
* the install pattern they're generating.
*/
export interface RecipeMcpServerSource {
/** Git URL to clone from. */
url: string;
/** Git URL to clone from. Mutually exclusive with `npm`. */
url?: string;
/**
* npm registry package spec (`pkg@version` / `@scope/pkg@version`) that
* build tooling bakes via a global install instead of a git clone —
* matches connectome-cook's `source.npm` grammar. The git-form fields
* (`ref`, `install`, `inContainer`, ...) don't apply. Mutually
* exclusive with `url`.
*/
npm?: string;
/**
* Git ref: branch, tag, or commit SHA. Default: "main".
* If the value starts with "refs/" (e.g. "refs/pull/3/head"), it's
Expand Down Expand Up @@ -1142,8 +1150,16 @@ export function validateRecipe(raw: unknown): Recipe {
throw new Error(`mcpServers.${id}.source must be an object`);
}
const src = server.source as Record<string, unknown>;
if (typeof src.url !== 'string' || !src.url) {
throw new Error(`mcpServers.${id}.source.url must be a non-empty string`);
const hasSrcUrl = typeof src.url === 'string' && src.url;
const hasSrcNpm = typeof src.npm === 'string' && src.npm;
if (hasSrcUrl && hasSrcNpm) {
throw new Error(`mcpServers.${id}.source must not set both "url" and "npm"`);
}
if (!hasSrcUrl && !hasSrcNpm) {
throw new Error(
`mcpServers.${id}.source must have a non-empty "url" (git clone) ` +
`or "npm" (registry package spec) string`,
);
}
if (src.ref !== undefined && typeof src.ref !== 'string') {
throw new Error(`mcpServers.${id}.source.ref must be a string`);
Expand Down
54 changes: 54 additions & 0 deletions test/recipe-mcp-source.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/**
* Tests for mcpServers.<id>.source validation: both of connectome-cook's
* source grammars (git `url` form and registry `npm` form) must pass — the
* shipped knowledge-miner recipe uses `source.npm` for its gitlab server,
* and a validator that only accepts `url` makes that recipe unloadable.
*/
import { describe, test, expect } from 'bun:test';
import { validateRecipe } from '../src/recipe.js';

function recipeWithSource(source: unknown) {
return {
name: 'source-test',
agent: { systemPrompt: 'sys' },
mcpServers: {
srv: {
command: 'npx',
args: ['-y', 'some-pkg'],
...(source !== undefined ? { source } : {}),
},
},
};
}

describe('mcpServers source validation', () => {
test('accepts the git url form', () => {
expect(() => validateRecipe(recipeWithSource({
url: 'https://github.com/x/y.git',
install: 'npm',
}))).not.toThrow();
});

test('accepts the npm registry form', () => {
expect(() => validateRecipe(recipeWithSource({
npm: '@zereight/mcp-gitlab@2.1.25',
}))).not.toThrow();
});

test('rejects a source with neither url nor npm', () => {
expect(() => validateRecipe(recipeWithSource({ install: 'npm' })))
.toThrow(/source must have a non-empty "url" \(git clone\) or "npm"/);
});

test('rejects a source with both url and npm', () => {
expect(() => validateRecipe(recipeWithSource({
url: 'https://github.com/x/y.git',
npm: 'y@1.0.0',
}))).toThrow(/must not set both "url" and "npm"/);
});

test('rejects empty-string url and npm', () => {
expect(() => validateRecipe(recipeWithSource({ url: '' }))).toThrow(/source/);
expect(() => validateRecipe(recipeWithSource({ npm: '' }))).toThrow(/source/);
});
});
Loading