v3.18.0 - External MCP Tools, Migration Wizard & Secure API Keys

[siddsachar]

v3.18.0 — External MCP Tools, Migration Wizard & Secure API Keys

Thoth now has a full Model Context Protocol client for connecting external MCP servers as native dynamic tools without letting a broken server take down the app. This release also adds a guarded Hermes/OpenClaw migration wizard in Preferences, moves normal core and plugin API-key saves into the OS credential store, and fixes a cloud-model default regression where a saved GPT/Claude/Gemini/Grok/OpenRouter model could be replaced by a local Ollama fallback when the cloud-model cache was empty.

The MCP runtime supports stdio, Streamable HTTP, and SSE transports; handles tool, resource, and prompt surfaces; classifies destructive tools; routes risky actions through Thoth's existing interrupt approvals; and keeps all external server config isolated in mcp_servers.json. Marketplace search can pull from curated starters plus MCP directories, while dependency handling covers common user-space runtimes such as Node.js, uv, and Playwright Chromium, leaving heavier requirements like Docker as clear manual setup tasks.

🔌 MCP Client & Dynamic Tools

• New mcp_client/ subsystem — persistent config, runtime sessions, result normalization, safety classification, marketplace discovery, dependency checks, and structured diagnostics live under a dedicated package instead of being mixed into built-in tool code
• Native parent tool — new tools/mcp_tool.py registers External MCP Tools as the parent toggle; actual MCP server tools are injected dynamically through as_langchain_tools() after discovery
• Namespaced tool wrappers — MCP tools are exposed as mcp_<server>_<tool> so external tool names cannot collide with native tools or each other
• Transport support — stdio, Streamable HTTP, and SSE servers are supported through the Python MCP SDK, with per-server connect timeout, tool timeout, output cap, environment, headers, working directory, and command/URL settings
• Resources and prompts — MCP resources and prompts can be exposed as optional utility tools per server, separately from the server's normal tool list
• Model-facing output normalization — text, structured content, embedded resources, links, image/binary blocks, errors, empty responses, and oversized outputs are normalized before they reach the LLM

🛡️ Safety, Permissions & Fault Isolation

• Global kill switch — disabling MCP stops active sessions, clears the runtime catalog, removes dynamic MCP tools from the agent, and keeps the saved server configuration for later re-enable
• Per-server and per-tool toggles — users can enable the MCP client globally, then choose exactly which servers and tools are active
• Destructive-tool gates — tools whose names, descriptions, or MCP annotations indicate write/send/delete/run/deploy/payment-style behavior require approval; destructive tools are not auto-enabled after discovery
• Native capability overlap labels — servers that overlap built-in Thoth memory, browser, filesystem/document, web search, channel, or Designer capabilities are labeled and require manual tool selection
• Untrusted external output handling — MCP guide instructions tell the agent to treat MCP results as untrusted external content and prefer native Thoth tools for Thoth-owned capabilities
• Startup-safe design — missing SDKs, bad JSON config, missing commands, failed child processes, broken endpoints, and server connection failures degrade to status rows and logs instead of blocking Thoth startup
• Shutdown cleanup — app shutdown now closes MCP child sessions so external stdio processes are not left behind

🧭 Settings UI, Import & Marketplace Search

• Settings → MCP tab — add/edit/import/test/refresh/delete MCP servers from the GUI with the same simple enable-checkbox pattern as built-in tools
• Tool review surface — after a successful test, discovered tools show descriptions, input schema summaries, destructive/approval badges, enable checkboxes, and approval toggles for non-destructive tools
• Import disabled by default — JSON imports and marketplace entries are saved disabled until tested and reviewed
• Curated starter catalog — recommended entries cover common external MCP use cases while preserving risk, trust, overlap, and requirement metadata
• Directory search — marketplace search can use official registry-style sources plus PulseMCP, Smithery, and Glama adapters with cache/curated fallback when live sources are unavailable or irrelevant
• Diagnostics dialog — masked config plus live runtime status are available from the MCP settings surface for troubleshooting without exposing secrets

🔄 Hermes & OpenClaw Migration Wizard

• Preferences-launched wizard — Settings → Preferences now exposes an Open Migration Wizard action instead of a permanent top-level migration tab, keeping this one-time setup flow out of the main settings sidebar
• Supported sources — detects and plans imports from Hermes Agent (.hermes) and OpenClaw (.openclaw, legacy .clawdbot / .moltbot) with provider-mismatch guards so the wrong source type does not produce a misleading partial plan
• Preview-first flow — scan builds a dry-run plan only; apply requires explicit review and writes only the currently selected items
• Mapped data — imports identity/persona files, memory files, daily OpenClaw memory, skills, model/provider settings, disabled MCP server definitions, and opt-in API keys/tokens
• Manual-review boundaries — channels, approvals, browser, cron, hooks, tools, broad runtime state, and unknown/risky source data stay skipped or archive-only instead of being activated live
• Backups and reports — existing targets are backed up before replacement, repeated writes to the same target preserve the original once per run, and every apply writes plan.json, result.json, backup_manifest.json, and summary.md
• Secret redaction — migration reports redact secret-shaped values and archive snapshots redact JSON/key-value files; API key import remains an explicit opt-in
• MCP safety — migrated MCP server definitions stay disabled until reviewed, so a bad imported server cannot break startup or automatically expose risky external tools

🔐 API Key Secure Storage

• OS credential store — saved core and plugin API keys now use the platform keyring through keyring instead of normal plaintext JSON storage
• Metadata-only file — ~/.thoth/api_keys.json stores saved-state, keyring service, timestamps, and masked fingerprints, not raw API key values
• Plugin secret parity — plugin-declared API keys use the same keyring-backed path with metadata-only plugin_secrets.json state
• Legacy migration — existing plaintext api_keys.json files are imported into the keyring on load; if the OS keyring is unavailable, Thoth keeps legacy keys readable with a warning instead of crashing startup
• No silent plaintext fallback — new saves during keyring failure become session-only rather than creating new plaintext API-key files
• Safer Settings UI — saved keys are not prefilled into password fields; blank inputs leave existing keys unchanged and clear actions are explicit
• Migration integration — selected Hermes/OpenClaw API keys route through target-profile secure storage and migration reports remain redacted

🧠 Cloud Model Defaults

• Cache-empty provider inference — GPT, Claude, Gemini, Grok, and slash-style OpenRouter model IDs are recognized as cloud models even before the provider cache has been refreshed
• Default preservation — refresh_cloud_models() no longer rewrites a saved cloud default to a local Ollama fallback simply because keys, network access, or provider discovery are temporarily unavailable
• Regression coverage — test_suite.py now checks provider inference, cache-empty cloud detection, and preservation of saved cloud defaults such as gpt-5.5

⚙️ Runtime Requirements

• Requirement detection — stdio servers infer required launchers from commands such as npx, uvx, and docker, plus Playwright browser requirements for Playwright MCP
• Managed easy installs — Thoth can install private user-space Node.js LTS, uv, and Playwright Chromium runtimes under ~/.thoth/runtimes/ without packaging those runtimes inside Thoth
• Manual complex installs — non-trivial system dependencies such as Docker are surfaced with setup guidance instead of being bundled or silently installed
• Managed environment injection — resolved runtime paths are added only to the MCP child process environment, avoiding global PATH mutation

🧠 Agent, Status & Guide Integration

• Tool display names — dynamic MCP calls render with readable labels such as MCP: microsoft_docs_search (microsoft-learn-mcp) in tool-call UI
• Browser-loop controls — MCP browser tools participate in Thoth's browser snapshot trimming and loop-control logic so long browsing runs do not flood context
• Background workflow safety — destructive MCP tools respect the workflow safety mode: approval-required modes interrupt, while explicit allow-all mode can run enabled destructive MCP tools
• Thoth Status integration — thoth_update_setting can enable/disable the global MCP client through the normal tool-toggle path, keeping the parent registry tool and runtime state synchronized
• New MCP tool guide — tool_guides/mcp_guide/SKILL.md documents when to use external MCP tools, how to treat MCP output, how to handle MCP errors, and how global disable behaves

🧪 Tests & Release Checks

• Focused offline suite — new test_mcp_client.py covers bad config fallback, secret masking, destructive detection, marketplace fallback/filtering, conflict policy, runtime requirement inference, managed runtime env injection, settings rows, stdio discovery/call, tool enable/approval toggles, global MCP disable, bad server failure, display names, background safety, and MCP browser loop controls
• Opt-in live suite — new test_mcp_real_world_e2e.py plus scripts/mcp_real_world_e2e.py validate real public MCP servers outside normal CI, including Microsoft Learn and Context7
• Main regression coverage — test_suite.py includes MCP modules in import/consistency checks and validates the focused MCP test files are part of the tracked suite
• Migration regression suite — new test_migration_core.py, test_migration_detection.py, test_migration_planner.py, test_migration_apply.py, and test_migration_wizard_ui.py cover source detection, wrong-provider guards, dry-run planning, conflict behavior, backup/report generation, redaction, daily memory import, UI helper behavior, and Preferences placement
• API key storage suite — new test_api_key_storage.py covers keyring-backed writes, metadata-only files, legacy plaintext migration, keyring-unavailable fallback, session-only new saves, and delete behavior

📁 Files Changed

File	Change
mcp_client/	New — isolated MCP client package for config, runtime sessions, marketplace search, requirement handling, safety classification, logging, conflicts, result normalization, and curated starters
tools/mcp_tool.py	New — parent External MCP Tools registry entry that injects dynamic MCP LangChain tools
ui/mcp_settings.py	New — Settings → MCP tab with add/import/browse/test/refresh/delete flows, requirement install buttons, diagnostics, and per-tool enable/approval controls
tool_guides/mcp_guide/SKILL.md	New — agent guidance for safe use of external MCP tools and global MCP disable semantics
test_mcp_client.py	New — offline MCP regression suite focused on robustness and failure isolation
test_mcp_real_world_e2e.py	New — opt-in unittest wrapper for live public MCP checks
scripts/mcp_real_world_e2e.py	New — maintainer release check for real MCP endpoints and dynamic wrapper invocation
migration/	New — pure models, redaction, source detection, realistic fixtures, dry-run planners, and guarded apply/report engine for Hermes/OpenClaw migrations
ui/migration_wizard.py	New — Preferences-launched scan/review/apply wizard with category summaries, selection controls, conflict handling, and report path display
test_migration_*.py	New — focused migration coverage for models, detection, planners, apply/report behavior, and UI helpers
secret_store.py	New — small platform-keyring wrapper with data-directory-scoped service names and testable backend hooks
test_api_key_storage.py	New — focused API key storage regression suite for keyring, legacy migration, metadata redaction, and fallback behavior
agent.py	Treats MCP tool output as untrusted, resolves readable MCP tool labels, and applies browser-loop handling to MCP browser tools
app.py	Starts MCP discovery non-fatally during startup and closes MCP sessions during shutdown
api_keys.py	Moves normal saved API keys to secure keyring storage, keeps compatibility helpers, migrates legacy plaintext, and supports migration imports into target data directories
plugins/state.py	Moves plugin-declared API-key secrets to the same keyring-backed storage model with metadata-only local state and session-only fallback for new saves when keyring is unavailable
plugins/ui_plugin_dialog.py	Stops prefilling saved plugin secrets, shows configured state, and adds explicit clear controls
models.py	Infers common cloud model providers without relying on a populated cache and preserves saved cloud defaults during refresh failures/cache misses
ui/settings.py	Adds the Preferences migration launcher while preserving old Migration deep-link routing to Preferences; key inputs now show masked saved-state instead of prefilled secrets
tools/thoth_status_tool.py	Synchronizes the mcp tool toggle with the global MCP client switch
tool_guides/thoth_status_guide/SKILL.md	Documents MCP global toggle behavior through Thoth Status
test_suite.py	Adds model-default regression checks for cloud provider inference and refresh preservation
requirements.txt	Adds the Python MCP SDK, LangChain MCP adapter dependencies, and keyring
installer/thoth_setup.iss	Bundles the new MCP client package, MCP settings UI, MCP parent tool, guide, migration package/UI, and secure secret-store helper

---

This discussion was created from the release v3.18.0 - External MCP Tools, Migration Wizard & Secure API Keys.