Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
node_modules
mcp-server/node_modules
dist
mcp-server/dist
src-tauri/target
src-tauri/gen
.git
*.pptx
assets
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,7 @@ pnpm-debug.log*
# Brainstorm assets (not tracked in source control)
.superpowers/
.claude/

# headless web server local data
llm-wiki-data/
wikis/
54 changes: 54 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# LLM Wiki — self-hosted web server image
#
# docker build -t llm-wiki .
# docker run -p 8080:8080 -e LLM_WIKI_WEB_PASSWORD=change-me \
# -v llm-wiki-data:/data -v $HOME/wikis:/wikis llm-wiki
#
# Data (settings, agent sessions, uploads) lives in /data; put wiki projects
# under a mounted folder such as /wikis and create/open them from the UI.

# ── frontend ────────────────────────────────────────────────────────────────
FROM node:20-bookworm-slim AS frontend
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --no-audit --no-fund
COPY tsconfig*.json vite.config.ts index.html components.json ./
COPY src ./src
COPY mcp-server/package.json mcp-server/package-lock.json ./mcp-server/
RUN npm --prefix mcp-server ci --no-audit --no-fund
COPY mcp-server ./mcp-server
RUN npm run mcp:build && npx vite build

# ── backend ─────────────────────────────────────────────────────────────────
FROM rust:1.88-bookworm AS backend
RUN apt-get update && apt-get install -y --no-install-recommends \
protobuf-compiler pkg-config libssl-dev clang \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app/src-tauri
COPY src-tauri/Cargo.toml src-tauri/Cargo.lock ./
COPY src-tauri/build.rs src-tauri/windows-app-manifest.xml ./
COPY src-tauri/src ./src
COPY src-tauri/tauri.conf.json ./
RUN cargo build --release --no-default-features --features server --bin llm-wiki-server

# ── runtime ─────────────────────────────────────────────────────────────────
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=backend /app/src-tauri/target/release/llm-wiki-server /usr/local/bin/llm-wiki-server
COPY --from=frontend /app/dist ./dist
COPY --from=frontend /app/mcp-server/dist ./mcp-server/dist
COPY src-tauri/pdfium ./pdfium

ENV LLM_WIKI_DATA_DIR=/data \
LLM_WIKI_STATIC_DIR=/app/dist \
LLM_WIKI_RESOURCE_DIR=/app \
LLM_WIKI_WEB_HOST=0.0.0.0 \
LLM_WIKI_WEB_PORT=8080
# Set LLM_WIKI_BIND_HOST=0.0.0.0 to also expose the MCP/API (:19828) and
# clip server (:19827) outside the container; they stay loopback by default.
VOLUME ["/data"]
EXPOSE 8080 19827 19828
ENTRYPOINT ["llm-wiki-server"]
61 changes: 61 additions & 0 deletions FORK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Fork 維護說明(Knowledge Loom 版 llm_wiki)

這個 repo 是 [nashsu/llm_wiki](https://github.com/nashsu/llm_wiki) 的 fork,多了 **Web / Server Mode** 與 **Connectors**(見 `plans/web-server-mode.md`、`plans/connectors.md`)。

## 分支與遠端

| 名稱 | 用途 |
|------|------|
| remote `upstream` | 原始專案 `nashsu/llm_wiki`,只拉不推 |
| remote `origin` | 你的 fork:https://github.com/johnsonafool/llm_wiki |
| branch `main` | 永遠等於 `upstream/main`,不放自己的改動 |
| branch `web-mode` | 我們的版本:`main` + web/server mode + connectors |

## 已送出的上游 PR

| PR | 分支 | 狀態 |
|----|------|------|
| [nashsu/llm_wiki#729](https://github.com/nashsu/llm_wiki/pull/729) RFC: Web/Server mode + Connectors | `web-mode` | Draft,等維護者回覆方向;已提議拆成三個 PR |
| [nashsu/llm_wiki#730](https://github.com/nashsu/llm_wiki/pull/730) fix(api): honor the Chat task preset for local API / MCP chat | `fix/api-chat-preset`(從 `main` 分出,worktree 在 `D:\llm_wiki-fix`) | 開放中,1 檔 +179/−1,api_server 測試 37/37 |

上游若合併了 `fix/api-chat-preset`,同步後 `web-mode` 內的同一段程式碼會以上游版本為準。

## 設定自己的遠端(一次性)

在 GitHub 建一個空 repo(建議 private,名稱如 `llm_wiki`),然後:

```bash
git remote add origin git@github.com:<your-account>/llm_wiki.git
git push -u origin main
git push -u origin web-mode
```

## 同步上游更新

```powershell
.\scripts\sync-upstream.ps1 # Windows
```
```bash
./scripts/sync-upstream.sh # macOS / Linux / Git Bash
```

腳本做的事:`git fetch upstream` → `main` fast-forward 到 `upstream/main` → 把 `main` **merge** 進 `web-mode`(用 merge 不用 rebase,因為 `web-mode` 會推到遠端)→ 有衝突時停下來讓你處理。合併後請跑:

```bash
npm run build # 前端 typecheck + build
npm run web:check # cargo check(server feature)
cargo check --manifest-path src-tauri/Cargo.toml # 桌面 feature
```

## 我們在上游之外改了什麼(除了 web mode / connectors)

| 日期 | 改動 | 位置 |
|------|------|------|
| 2026-09-06 | RPC 路徑護欄:web 模式下 `read_file` / `write_file` / `list_directory` / `copy_*` / `delete_file` 等原生檔案指令只允許碰已登錄專案資料夾與資料目錄(與 `/web/file` 同一份白名單),本 session 內 `open_project` / `create_project` / `import_project_archive` 過的資料夾立即生效 | `src-tauri/src/web/rpc.rs`(`PATH_GUARDED_COMMANDS`)、`src-tauri/src/web/mod.rs`(`path_is_allowed_lenient`、`remember_opened_root`) |
| 2026-09-06 | 本機 API / MCP 的 chat 尊重 Settings → Models 的「Chat 使用的 preset」(`taskModelRouting.chatPresetId` + `providerConfigs`),之前只讀全域 `llmConfig`,若全域是 Claude Code CLI 就退化成只列搜尋結果 | `src-tauri/src/api_server.rs`(`chat_preset_llm_config`、`enabled_project_llm_config`) |

上游若之後自己修了同一件事,合併時以上游為準、刪掉我們的版本即可。

## 衝突熱區

上游改動最常和我們撞到的檔案:`src-tauri/src/lib.rs`(指令註冊改走 `app_commands.rs`)、`src-tauri/Cargo.toml`(features)、`src/main.tsx` / `src/App.tsx`(WebGate)、`src/lib/tauri-fetch.ts`(改走 `src/lib/backend/`)。上游新增 Tauri 指令時,記得同步加到 `src-tauri/src/web/rpc.rs` 的對應表,否則 web 版會缺該功能。
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,22 @@ npm run tauri dev # Development
npm run tauri build # Production build
```

### Web / Server Mode (self-hosted)

The same frontend can run in a browser against a headless Rust server — no Tauri, no desktop session required. Single-user / small-team oriented: one shared password, cookie sessions, everything else identical to the desktop app (ingest, chat, graph, MCP API on `:19828`).

```bash
npm install && npm run build # build the frontend into dist/
npm run web:serve # llm-wiki-server on http://127.0.0.1:8080
# or: docker compose up -d # set LLM_WIKI_WEB_PASSWORD in docker-compose.yml
```

See [plans/web-server-mode.md](plans/web-server-mode.md) for flags, the security model, and the architecture of the host abstraction (`src/lib/backend/`, `src-tauri/src/web/`).

### Connectors (external sources)

**Settings → Connectors** pulls documents from other systems into `raw/sources/@<name>/` on a schedule, where they are ingested like any other source. Ships with a local-folder connector and Google Drive (bring-your-own OAuth client, delta sync, Docs/Sheets/Slides exported as Office files). New kinds implement one Rust trait plus a descriptor — see [plans/connectors.md](plans/connectors.md).

### Chrome Extension

1. Open `chrome://extensions`
Expand Down
16 changes: 16 additions & 0 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,22 @@ npm run tauri dev # 开发模式
npm run tauri build # 生产构建
```

### Web / Server 模式(自架)

同一套前端可以在瀏覽器裡對著一個不依賴 Tauri 的 headless Rust 伺服器執行。定位是單人/小團隊自架:一組共用密碼、cookie session,其餘功能(匯入、Chat、圖譜、`:19828` 的 MCP API)與桌面版相同。

```bash
npm install && npm run build # 產出 dist/
npm run web:serve # llm-wiki-server 於 http://127.0.0.1:8080
# 或:docker compose up -d # 記得改 docker-compose.yml 裡的 LLM_WIKI_WEB_PASSWORD
```

細節見 [plans/web-server-mode.md](plans/web-server-mode.md)。

### 外部來源連接器

**設定 → 外部來源** 可以定時把其他系統的文件拉進 `raw/sources/@<名稱>/`,之後照一般原始資料的流程 ingest。內建本地資料夾與 Google Drive(自帶 OAuth client、delta 同步、Docs/Sheets/Slides 匯出成 Office 檔)。新增來源只要實作一個 Rust trait 加一個 descriptor,見 [plans/connectors.md](plans/connectors.md)。

### Chrome 扩展

1. 打开 `chrome://extensions`
Expand Down
19 changes: 19 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
services:
llm-wiki:
build: .
image: llm-wiki
ports:
- "8080:8080"
environment:
# Change this. Without it the server prints a random password at start.
LLM_WIKI_WEB_PASSWORD: change-me
# Optional: HTTP proxy for outbound LLM calls
# HTTPS_PROXY: http://proxy.local:3128
volumes:
- llm-wiki-data:/data
# Wiki projects live on the host; create/open them at /wikis/<name> in the UI.
- ./wikis:/wikis
restart: unless-stopped

volumes:
llm-wiki-data:
4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@
"test:mocks": "vitest run --exclude='**/*.real-llm.test.ts' --exclude='**/mcp-server/**'",
"test:llm": "vitest run real-llm --no-file-parallelism --reporter=verbose",
"mcp:build": "npm --prefix mcp-server run build",
"web:build": "npm run build && cargo build --release --manifest-path src-tauri/Cargo.toml --no-default-features --features server --bin llm-wiki-server",
"web:check": "cargo check --manifest-path src-tauri/Cargo.toml --no-default-features --features server --lib --bins",
"web:test": "cargo test --manifest-path src-tauri/Cargo.toml --no-default-features --features server --lib",
"web:serve": "cargo run --manifest-path src-tauri/Cargo.toml --no-default-features --features server --bin llm-wiki-server -- --static-dir dist",
"mcp:test": "npm --prefix mcp-server test",
"tauri": "tauri"
},
Expand Down
115 changes: 115 additions & 0 deletions plans/connectors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Connectors

> **中文摘要**:Connector 只負責「遠端長什麼樣」(`list_all`)和「拿一份下來」(`fetch`),不碰 wiki、ingest 或 UI。註冊表(`descriptors()` / `build()`)就是工廠:每種 connector 宣告自己的設定欄位與認證方式,設定頁據此自動產生表單。共用的 SyncEngine 負責 diff、寫入 `raw/sources/@<名稱>/`、刪除與狀態檔;之後交給既有的 Source Watch 走 ingest。目前有 `local-folder` 與 `google-drive`(OAuth PKCE、自帶 client、支援 delta)。

Connectors pull documents from external systems into a project's
`raw/sources/` folder. From there the existing pipeline takes over: the
source-folder watcher (or the startup rescan when the project is opened
next) queues new/changed files for ingest and runs the cascade cleanup for
removed ones. Nothing downstream knows a connector exists.

```
┌──────────────── connectors (Rust) ────────────────┐
remote system ─────►│ Connector::list_all(cursor) ─► Listing │
│ Connector::fetch(item) ─► bytes + file name │
│ ▼ │
│ sync::run_sync — diff vs .llm-wiki/connectors/<id>.state.json
│ add/update → raw/sources/@<name>/<remote path> │
│ delete → remove file (+ prune empty dirs) │
└───────────────────────┬───────────────────────────┘
source-folder watcher / startup rescan → ingest queue
```

## Layers

| Layer | Where | Responsibility |
|-------|-------|----------------|
| `Connector` trait | `src-tauri/src/connectors/mod.rs` | `test()`, `list_all(cursor) -> Listing`, `fetch(item) -> Fetched`. No project paths, no wiki. |
| Registry / factory | `mod.rs` — `descriptors()`, `build(ctx, instance, secrets)` | One `ConnectorDescriptor` per kind (label, config fields, auth kind, delta support) and one `match` arm that instantiates it. |
| Sync engine | `sync.rs` | Version diff, path sanitising, size/extension filters, atomic writes, deletions, per-instance state, progress events (`connectors://sync`). |
| Storage | `store.rs` | `<project>/.llm-wiki/connectors.json` (instances), `<project>/.llm-wiki/connectors/<id>.state.json` (cursor + item versions), `<app data>/connectors-secrets.json` (tokens/secrets, 0600 on Unix, never inside the project). |
| OAuth | `oauth.rs` | Authorization-code + PKCE, pending-state table, token exchange/refresh, callback HTML. |
| Scheduler | `scheduler.rs` | Every minute, run every enabled instance whose `interval_minutes` has elapsed — across all known projects, in the backend, so it works headless. |
| Commands | `commands.rs` | `connector_descriptors/list/save/delete/test/sync/reset/oauth_start/oauth_disconnect` — registered for Tauri IPC (`lib.rs`) and the web RPC bridge (`web/rpc.rs`). |
| UI | `src/components/settings/sections/connectors-section.tsx` | Generic form from descriptors; cards with status, Sync now, Test, Connect/Disconnect. `src/commands/connectors.ts` holds the typed wrappers. |

### Listing semantics

`Listing { items, is_delta, next_cursor }`:

* `is_delta = false` — a full snapshot. Anything the engine had that is not
in the snapshot is treated as deleted. Connectors without change feeds
(local folder, WebDAV, S3…) always do this.
* `is_delta = true` — only what changed since the cursor the engine passed
in; removals come back with `deleted = true`. `next_cursor` is stored and
passed to the next run. Google Drive uses `changes.list`.

Equal `version` strings (checksum, etag, `mtime-size`…) mean "unchanged":
no download. Items land at `raw/sources/@<instance name>/<path…>` with
every component sanitised (no separators, no `..`, no control characters).
Directories in the remote path become the `folderContext` hint the ingest
step already uses for classification.

## Kinds

### `local-folder`

Mirror a directory on the machine running the backend. Full walk every run,
hidden entries skipped, `version = mtime-size`. Useful on its own (a Syncthing
/ Dropbox / OneDrive-client folder) and as the reference implementation.

### `google-drive`

Drive API v3 with the `drive.readonly` scope.

* **Auth — bring your own OAuth client.** Create an OAuth client in Google
Cloud Console (APIs & Services → Credentials), enable the Drive API, and
paste client id / secret into the connector. Register the redirect URI the
form shows: `http://127.0.0.1:19828/api/v1/oauth/callback` for the desktop
app ("Desktop app" client type) or `https://<host>/web/connectors/oauth/callback`
for the web server ("Web application" type). Press **Connect**; tokens are
stored in `connectors-secrets.json` and refreshed automatically.
* **Scope:** a folder id (or full folder URL) or blank for all of My Drive.
Shared drives are included through `supportsAllDrives`.
* **Delta:** the first run walks the tree (`files.list`) after taking a
`changes.getStartPageToken`; later runs use `changes.list` and only touch
changed files. An expired cursor (Drive keeps them about a week) triggers
a full resync automatically.
* **Google-native files:** Docs / Sheets / Slides are exported as
`.docx` / `.xlsx` / `.pptx` (Drive caps exports at 10 MB); other native
types (Forms, Drawings…) are skipped.

## Adding a connector

1. Create `src-tauri/src/connectors/<kind>.rs` implementing `Connector`
and a `descriptor()` listing its config fields (`text`, `number`,
`boolean`, `secret`, `path`). Secret fields are stored in the secrets
file and never echoed back to the UI.
2. Add it to `descriptors()` and `build()` in `mod.rs`.
3. If it needs OAuth, add a `ProviderDef` in `oauth.rs` (auth/token URLs)
and set `AuthKind::OAuth2 { provider, scopes }` in the descriptor. The
connector reads a valid token with `oauth::access_token(ctx, id, provider)`.
4. Prefer a delta feed when the API offers one (`is_delta = true` +
`next_cursor`); otherwise return snapshots and let the engine diff.
5. Tests: unit-test the mapping helpers; the engine's own tests use a fake
connector, so no network is needed.

Natural next kinds: OneDrive / SharePoint (Graph `delta`), Dropbox
(`list_folder/continue` cursors), Notion (`last_edited_time` filter),
WebDAV / Nextcloud (snapshot), S3 (snapshot with ETag versions).

## Operational notes

* Syncs run in the backend; the scheduler ticks every minute and skips
instances that are already running. "Sync now" returns immediately and
reports through `connectors://sync` events (`started → listed →
progress… → finished | failed`).
* Deleting an instance can also purge its mirrored files; either way the
removal of `raw/sources/@<name>/…` files flows through the same
external-delete cleanup as any other source removal.
* Renaming an instance moves its folder: the engine purges the old mirror
and resyncs into the new name.
* The legacy **Scheduled Import** section is untouched; a `local-folder`
connector covers the same use case with a persistent state file and
backend scheduling, so it can replace it once you are happy with it.
Loading