Integrate seekdb as ClawMaster's hybrid local data store

[webup]

Summary

Integrate seekdb as ClawMaster's default hybrid local data store for app-owned indexed and searchable data, while keeping openclaw.json and OpenClaw-owned runtime files as the source of truth.

This RFC updates the earlier proposal with one additional product constraint:

• Do not introduce any Python-based package, runtime dependency, installer step, or bundled skill into ClawMaster's user-facing path.

The revised direction supports:

• web/backend mode directly
• desktop on Linux and macOS Apple Silicon where embedded bindings are supported
• Windows primarily via WSL2-hosted backend + embedded SeekDB
• graceful fallback or server-mode support on unsupported embedded targets such as native Windows and macOS Intel

Problem

ClawMaster currently has no unified local datastore for app-owned state and search indexes.

Current state is split across:

• openclaw.json and profile-specific OpenClaw files
• .clawmaster/settings.json
• frontend localStorage
• derived/indexable content that is searched ad hoc or not persisted as a proper index

This creates several limitations:

• no consistent local-first search/index layer for docs, logs, memory mirrors, or future knowledge features
• no shared data abstraction for web and desktop runtime modes
• no clear place to put derived structured state that should not live in openclaw.json
• weak foundation for future features such as semantic search, hybrid retrieval, background indexing, and contextual diagnostics

Product Constraints

The SeekDB rollout must follow these constraints from day one:

• no Python-based package is added as an app dependency
• no Python runtime is required for setup, onboarding, or feature use
• no Python-based ClawHub skill is preinstalled or bundled
• no feature depends on users manually installing Python to use ClawMaster

Upstream projects may use Python internally for their own source builds, but ClawMaster must consume published artifacts through JavaScript/TypeScript-only product flows.

Why SeekDB

seekdb-js is still a strong fit because it provides:

• embedded local mode via a filesystem path
• server mode for future remote/shared deployments
• full-text, vector, sparse, and hybrid retrieval primitives
• a JavaScript/TypeScript API that fits our backend stack

Relevant current upstream constraints:

• seekdb requires Node >=20
• embedded mode depends on @seekdb/js-bindings
• embedded native binding support is currently Linux x64/arm64 and macOS arm64
• Windows native embedded bindings are not available yet
• macOS Intel native embedded bindings are not available yet

This means SeekDB should be adopted behind a backend-only adapter and not exposed directly to the frontend.

Goals

• establish a single ClawMaster-owned hybrid local datastore
• keep OpenClaw runtime/config as the source of truth for OpenClaw-owned data
• support profile-scoped storage isolation
• support keyword search first and hybrid/vector search where available
• make indexing asynchronous so app launch remains fast
• provide graceful fallback on unsupported platforms
• make Windows support practical by leveraging WSL2 when OpenClaw is already run there
• keep all ClawMaster-owned integration paths JavaScript/TypeScript-based from the user's perspective

Non-goals

• moving openclaw.json itself into SeekDB
• replacing OpenClaw memory/runtime internals
• exposing low-level DB administration as a first-class primary workflow initially
• blocking app launch on model download or initial vectorization
• requiring SeekDB for every feature from day one
• bundling or preinstalling third-party Python-based SeekDB skills

Proposed Architecture

1. Backend storage abstraction

Introduce a backend-only storage interface under a shared backend package, for example packages/storage/ or packages/backend/src/storage/.

Responsibilities:

• init()
• health()
• getCapabilities()
• upsertDocuments()
• deleteBySource()
• search()
• hybridSearch()
• stats()
• rebuild()
• reset()
• export()
• import()

Initial implementations:

• SeekdbEmbeddedStore
• SeekdbServerStore
• FallbackFileStore

Frontend and modules should talk only to ClawMaster APIs, never to SeekDB directly.

2. Shared Node storage worker

Do not wire SeekDB logic into the web frontend or into Rust.

Instead, standardize on a Node-based storage worker that can be used by both runtime modes:

• Express mode imports and runs the storage layer directly
• desktop mode invokes the same Node worker through the host runtime
• on Windows with WSL2 mode enabled, desktop/backend calls route through wsl.exe -d <distro> node ...

This keeps the storage implementation consistent across web and desktop, and avoids splitting the database stack between Rust and Node.

3. Profile-scoped storage root

SeekDB data must be isolated by active OpenClaw profile.

Proposed layout:

• ~/.clawmaster/data/default/seekdb
• ~/.clawmaster/data/dev/seekdb
• ~/.clawmaster/data/named/<profile>/seekdb

This keeps reset, rebuild, export, and import safe and avoids cross-profile contamination.

4. Data ownership model

Keep a clear split.

OpenClaw-owned source of truth:

• openclaw.json
• OpenClaw logs
• OpenClaw memory files/state
• OpenClaw-managed skills/plugins/channels/models config

ClawMaster-owned data in SeekDB:

• searchable docs index
• searchable logs index
• mirrored/search-optimized memory index
• curated catalog cache / derived metadata
• future onboarding/task state after localStorage phase-out
• future diagnostics snapshots and searchable event timeline

5. Initial collections

Tentative collections:

• docs_index
• logs_index
• memory_index
• catalog_cache
• task_state
• diagnostic_events

Common document shape:

• id
• profileKey
• module
• sourceType
• sourcePath
• title
• content
• tags
• metadata
• updatedAt

6. Search behavior

Release in layers:

• first: exact and full-text search
• second: vector search when embeddings are ready
• third: hybrid search for modules that benefit from semantic and keyword matching

Initial app launch and page rendering must not wait for vector index readiness.

7. Embedding strategy

Do not make first-run model download a hard requirement for using the app.

Initial behavior:

• keyword and full-text index is available first
• vector index is optional and can warm in background
• UI shows indexing state clearly
• if vector capabilities are unavailable, features degrade to keyword search instead of erroring

Platform Strategy

Linux

• use embedded SeekDB as the default engine
• package and test it as a first-class path

macOS Apple Silicon

• use embedded SeekDB as the default engine
• package and test it as a first-class path

macOS Intel

• do not assume embedded support
• support SeekDB through server mode or FallbackFileStore
• keep the overall feature set working without blocking the user on unsupported native bindings

Windows

• do not depend on native embedded support
• support SeekDB primarily through WSL2-hosted backend/runtime
• keep SeekDB files inside the Linux filesystem, not /mnt/c/...
• if WSL2 is not enabled, fall back cleanly to FallbackFileStore or a user-configured SeekDB server

Web/backend mode

This is the first-class initial target.

• Express backend hosts the storage adapter
• embedded mode runs directly where supported
• fallback or server mode is used where embedded mode is unavailable

Desktop mode

Do not expose SeekDB directly to the browser/frontend.

Desktop should call the same backend-facing storage layer used by web mode.

UI Recommendation

Do not create a standalone database page initially.

Instead, add a new subsection under Settings, for example:

• Settings > Local Data
• or Settings > Data Store

Initial UI responsibilities:

• engine status
• active profile scope
• storage path
• database size
• collection count
• index readiness / background task state
• last rebuild time
• actions: rebuild, reset, export, import, open folder

A standalone page should only be considered later if we add:

• collection explorer
• indexing jobs/history
• repair/compaction workflows
• query inspector / relevance debugging
• admin-grade DB observability

Skills and Ecosystem Policy

ClawMaster should not preinstall a third-party SeekDB skill as part of the initial SeekDB rollout.

Reasons:

• current SeekDB-related ClawHub skills are not part of the trusted product surface
• at least some candidate skills rely on Python or other extra runtime assumptions
• preinstalling an unvetted skill would blur the trust boundary between ClawMaster core features and external community automation

If we add a SeekDB helper skill later, it should meet all of these requirements:

• JavaScript/TypeScript-only user-facing runtime path
• first-party maintained or explicitly security-reviewed
• optional, not required for core Local Data features
• installable through the existing Skills flow rather than silently bundled into core runtime

Rollout Plan

Phase 0: RFC + compatibility validation

• validate current supported platforms in CI/dev
• confirm backend API shape
• confirm WSL2 workflow for Windows
• document the no-Python dependency rule for the feature
• define storage root and version metadata

Phase 1: storage foundation

• add storage adapter abstraction
• add FallbackFileStore
• add engine capability detection
• add profile-scoped path resolver
• add Settings > Local Data section
• add health/status endpoints

Phase 2: SeekDB integration

• add SeekdbEmbeddedStore
• add SeekdbServerStore
• initialize DB/version metadata
• implement collection creation/bootstrap
• implement rebuild/reset/export/import primitives
• implement background indexing orchestration

Phase 3: first feature adoption

Migrate the first two consumers:

• Docs module search/index
• contextual Logs search/index

These are the best first candidates because they are app-owned search surfaces and do not redefine OpenClaw source-of-truth boundaries.

Phase 4: advanced adoption

• mirrored memory search index for faster retrieval in Memory module
• curated catalog caching
• diagnostics/event index
• eventual migration of onboarding/task state out of localStorage

Phase 5: ecosystem hardening

• supported desktop packaging for Linux/macOS where possible
• WSL2 guidance + health checks on Windows
• fallback/unsupported messaging for native Windows and macOS Intel
• evaluate a pure JS/TS first-party SeekDB helper only if it adds clear user value

API / Backend Work Items

• add GET /api/storage/status
• add POST /api/storage/rebuild
• add POST /api/storage/reset
• add POST /api/storage/export
• add POST /api/storage/import
• add module-specific search APIs backed by the storage adapter
• add per-profile capability reporting

Testing Plan

Unit tests

• profile-aware path resolution
• storage adapter contract tests
• fallback engine behavior on unsupported targets
• schema/version bootstrap logic
• export/import/reset/rebuild logic
• runtime engine selection for native, WSL2, server, and fallback paths

Integration tests

• backend status route with and without SeekDB availability
• docs indexing/search flow
• logs indexing/search flow
• unsupported platform degradation path
• WSL2-targeted environment detection logic where practical

UI tests

• Settings Local Data section renders correct engine state
• rebuild/reset/export/import flows
• docs/logs search works with indexed data
• unsupported platform messaging is actionable

CI strategy

• run embedded SeekDB tests only on supported runners
• run fallback adapter tests on all runners
• run WSL2 smoke coverage on Windows CI
• avoid silently failing packaging due to missing native bindings
• do not add Python as an application prerequisite for any user-facing feature

Risks

• native binding support is incomplete across ClawMaster's desktop matrix
• first-run embedding/model warm-up may be slow if not deferred
• packaging complexity increases for desktop builds
• Windows support depends on WSL2 or fallback until upstream native support improves
• storage migration/versioning needs discipline from the first commit

Open Questions

• should we standardize on a single Node storage worker across both web and desktop from the first PR?
• should vector search be opt-in initially, with full-text enabled by default?
• do we want to expose remote/server-mode SeekDB or OceanBase as an advanced option in a later phase?
• when, if ever, should a first-party pure JS/TS SeekDB helper skill be offered in the Skills module?

Recommendation

• keep the current MVP work separate from SeekDB implementation PRs
• start with Settings > Local Data, not a standalone DB page
• support Windows first through WSL2 when OpenClaw is also run there
• keep the first real feature adoption limited to Docs + Logs search
• do not preinstall any third-party SeekDB skill in the initial rollout
• do not add any Python-based package or skill to the ClawMaster product path

Acceptance Criteria for Initial Delivery

• backend storage adapter exists and is covered by tests
• active profile resolves to an isolated storage root
• app works without SeekDB on unsupported targets
• settings page exposes local data engine status and basic management actions
• docs and logs can index/search through the new adapter
• Linux and macOS Apple Silicon support embedded SeekDB
• Windows users running OpenClaw in WSL2 can use SeekDB through a WSL2-hosted backend
• native Windows and macOS Intel degrade cleanly to fallback or server mode
• no feature requires SeekDB synchronously at app boot
• no user-facing ClawMaster flow requires Python or a Python-based skill

References

• seekdb-js README: https://github.com/oceanbase/seekdb-js/tree/main/packages/seekdb
• seekdb package metadata: https://www.npmjs.com/package/seekdb
• seekdb native bindings README: https://github.com/oceanbase/seekdb-js/tree/main/packages/bindings
• existing ClawMaster WSL2 runtime support: feat: unify diagnostics escalation paths #21

[webup]

Implementation update from the PowerMem / seekdb workstream:

Keep this RFC scoped to ClawMaster Local Data / app-owned indexed data. PowerMem memory backend selection is tracked separately under #31 / #32.

Current platform support wording should be updated to this matrix:

• Linux x64 / arm64: embedded seekdb remains the preferred Local Data target when the Node/runtime/native binding checks pass.
• macOS arm64: do not claim embedded seekdb as production-ready by default yet. Local verification currently hits a native vsag::VsagException; upstream tracking is Embedded seekdb on macOS aborts with uncaught vsag::VsagException oceanbase/seekdb-js#37. Treat this as capability-gated until fixed and verified.
• macOS x64: no embedded seekdb assumption; use fallback/server path.
• Windows native: no embedded seekdb assumption. Local Data should use the ClawMaster fallback path unless/until seekdb Windows embedded support is released and verified.
• Windows + WSL2 runtime: keep this as an explicit WSL-managed path only when runtime/filesystem ownership is clear; do not silently mix host Windows and WSL stores.

Important distinction: SQLite is the intentional PowerMem memory fallback on Windows/macOS where seekdb embedded is not production-ready. It is not the Local Data backend described by this RFC unless we explicitly choose to add a SQLite Local Data adapter later.

Recommended RFC adjustment: describe embedded seekdb as default only on verified/stable targets, and document macOS arm64 / Windows as capability-gated with fallback paths until upstream native support is production-ready.

[webup]

PR #36 finishes the PowerMem runtime and OpenClaw bridge work tracked by #31 and #32, but I am leaving #12 open intentionally.

Reason: #12 is broader than the managed-memory bridge rollout. It covers ClawMaster-wide SeekDB adoption as the default local data layer for app-owned indexed/searchable data, including Local Data / storage abstraction, platform strategy, and non-memory consumers.

What #36 establishes:

• PowerMem is now the managed memory runtime.
• The OpenClaw bridge is wired for web/backend and desktop flows.
• Bridge/runtime path handling, WSL handling, desktop packaging, and bridge UI gating are in place.

What still belongs to #12:

• broader SeekDB-backed Local Data/storage rollout beyond memory
• remaining platform policy follow-through outside the memory bridge path
• additional cross-feature storage adoption and follow-up hardening

So after merging #36, #31 and #32 should close, while #12 remains the umbrella RFC/follow-up for the wider SeekDB storage program.