diff --git a/docs/en/development/memory-layer.md b/docs/en/development/memory-layer.md index 53d97a2ff..956a5ac0b 100644 --- a/docs/en/development/memory-layer.md +++ b/docs/en/development/memory-layer.md @@ -90,20 +90,14 @@ the same citation fields through list and exact-read operations. unavailable. Explicit `vector` and `hybrid` requests fail when the configured profile does not provide that capability. -## Enable SQLite Vec1 +## Enable SQLite vector search -SQLite vector search is enabled only when both a Vec1 0.7 or newer loadable extension and an embedding model are -supplied. PowerContext does not install or build the native extension; provide a compatible library for the target -operating system and architecture: +SQLite vector search is enabled when an embedding model is supplied. The `powercontext[builtin]` extra bundles +`sqlite-vec`, so no extension path or separate native-library installation is required: ```python -from pathlib import Path - config = BuiltinConfig( - database=SQLiteConfig( - url="sqlite+aiosqlite:///powercontext.db", - vec1_extension=Path("/opt/sqlite-extensions/vec1"), - ) + database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db") ) async with open_builtin_runtime( config, @@ -112,7 +106,7 @@ async with open_builtin_runtime( ... ``` -The SQLite profile composes FTS5 and Vec1 strategies. It reports `fts`, `vector`, and `hybrid` through Memory +The SQLite profile composes FTS5 and sqlite-vec strategies. It reports `fts`, `vector`, and `hybrid` through Memory capabilities. Stored projections and query vectors must use the same `EmbeddingProfile`, including model name, dimension, distance, and normalization. Changing that profile requires rebuilding projections before vector search resumes. @@ -145,7 +139,7 @@ async with open_builtin_runtime( The OceanBase profile uses the same index composition as SQLite. Its full-text strategy is always available. Supplying an embedding model adds a `VECTOR` projection and HNSW strategy, enabling `vector` and `hybrid` modes. SQLite FTS5 and -OceanBase FULLTEXT therefore serve the same Runtime and Server search calls; Vec1 and HNSW do the same for vector +OceanBase FULLTEXT therefore serve the same Runtime and Server search calls; sqlite-vec and HNSW do the same for vector search. ## Operational checks @@ -155,7 +149,7 @@ Before serving requests, verify: - the selected profile opens and initializes successfully; - each tenant or project maps to the intended scope ID; - scheduled extraction has a candidate pipeline; -- Vec1 configuration includes a matching embedding model; +- SQLite vector search has a matching embedding model; - OceanBase vector search has a matching embedding model; - capability responses match the indexes actually initialized; - database and scheduler resources close with the process lifecycle. diff --git a/docs/en/development/pydantic-ai-inference.md b/docs/en/development/pydantic-ai-inference.md index b21bf668f..85d8ac914 100644 --- a/docs/en/development/pydantic-ai-inference.md +++ b/docs/en/development/pydantic-ai-inference.md @@ -33,15 +33,14 @@ Vector search needs the embedding model and its complete deployment profile: export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_MODEL="provider:embedding-model" export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_PROFILE_ID="project-embedding-v1" export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_DIMENSION="1536" -export POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION="/opt/sqlite-extensions/vec1" ``` Provider credentials remain in the environment variables understood by the selected Pydantic AI provider. They are not fields on PowerContext models. The Server rejects a partial embedding profile. `embedding_model`, `embedding_profile_id`, and `embedding_dimension` -must be configured together. Vec1 also requires that embedding configuration because the index dimension and stored -vectors must agree. +must be configured together. SQLite vector search uses that embedding configuration because the index dimension and +stored vectors must agree. ## Compose generation directly @@ -99,9 +98,9 @@ embedding_model = PydanticAIEmbeddingModel( ) ``` -Pass this adapter to `open_builtin_contexts()` or `open_builtin_runtime()` with a `SQLiteConfig` that selects the Vec1 -extension. The adapter verifies output count, order, dimension, and finite numeric values, then applies the declared -unit normalization before vectors reach persistence. +Pass this adapter to `open_builtin_contexts()` or `open_builtin_runtime()` with a `SQLiteConfig`. The bundled +sqlite-vec index is enabled automatically. The adapter verifies output count, order, dimension, and finite numeric +values, then applies the declared unit normalization before vectors reach persistence. An `EmbeddingProfile` is a deployment contract, not descriptive metadata. Stored projections and query embeddings must use the same profile. When the model, dimension, or normalization changes, rebuild Memory projections from the diff --git a/docs/en/development/remote-access-implementation.md b/docs/en/development/remote-access-implementation.md index 542a50f4f..71257f8bf 100644 --- a/docs/en/development/remote-access-implementation.md +++ b/docs/en/development/remote-access-implementation.md @@ -56,7 +56,7 @@ export POWERCONTEXT_SERVER_DATABASE_KIND="oceanbase" export POWERCONTEXT_SERVER_DATABASE_URL="mysql+aoceanbase://user:password@host:2881/powercontext?charset=utf8mb4" ``` -Both database choices expose full-text search through the same Server API. With an embedding model, SQLite uses Vec1 +Both database choices expose full-text search through the same Server API. With an embedding model, SQLite uses sqlite-vec and OceanBase uses HNSW for `vector` and `hybrid` searches. Inference configuration is documented in [Configure Pydantic AI inference](pydantic-ai-inference.md). diff --git a/docs/en/docs/reference/configuration.md b/docs/en/docs/reference/configuration.md index 4a193e379..ebeadfaf5 100644 --- a/docs/en/docs/reference/configuration.md +++ b/docs/en/docs/reference/configuration.md @@ -178,24 +178,21 @@ Optional settings are `POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_NORMALIZATION` an Embedding normalization defaults to `unit`. -### SQLite Vec1 +### SQLite vector search -SQLite vector and hybrid search additionally require a -[SQLite Vec1](https://sqlite.org/vec1/doc/trunk/doc/vec1.md) 0.7 or newer loadable extension. PowerContext does not -download, build, or update this native library. Obtain it for the Server's operating system and architecture, then -set its path together with the complete embedding profile: +SQLite vector and hybrid search use [sqlite-vec](https://alexgarcia.xyz/sqlite-vec/), which is bundled with the +`powercontext[builtin]` dependency set. Configure the complete embedding profile; no extension path is needed: ```bash export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_MODEL=provider:embedding-model export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_PROFILE_ID=embedding-model-v1 export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_DIMENSION=1024 export POWERCONTEXT_SERVER_DATABASE_URL=sqlite+aiosqlite:////srv/powercontext/powercontext.db -export POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION=/opt/sqlite-extensions/vec1 powercontext server run ``` -The extension path must identify a library that the SQLite loader can open. PowerContext loads and probes the -extension when the Server opens the database; startup fails if the library is incompatible or older than 0.7. +PowerContext loads and probes the bundled extension when the Server opens the database. Startup fails if the package +does not contain a library compatible with the current platform or SQLite build. In another terminal, confirm that the initialized runtime reports vector and hybrid search: @@ -203,8 +200,7 @@ In another terminal, confirm that the initialized runtime reports vector and hyb powercontext capabilities ``` -If Vec1 is unavailable, leave `POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION` unset. SQLite full-text search remains -available without an embedding model or native extension. +SQLite full-text search remains available when no embedding model is configured. ## CLI Server connection diff --git a/docs/zh/development/memory-layer.md b/docs/zh/development/memory-layer.md index 64b2673c3..0bb11b2fb 100644 --- a/docs/zh/development/memory-layer.md +++ b/docs/zh/development/memory-layer.md @@ -88,19 +88,14 @@ operation 返回相同的 citation 字段。 `mode="auto"` 会选择当前可用的最强模式,并可在 query embedding 暂时不可用时回退到 FTS。显式请求 `vector` 或 `hybrid` 时,如果 profile 没有提供相应能力,操作会失败。 -## 启用 SQLite Vec1 +## 启用 SQLite 向量检索 -只有同时提供 0.7 或更高版本的 Vec1 loadable extension 和 embedding model,SQLite 才会启用向量检索。 -PowerContext 不负责安装或构建这个 native extension;请提供适用于目标操作系统和架构的 library: +提供 embedding model 后,SQLite 会启用向量检索。`powercontext[builtin]` 已捆绑 `sqlite-vec`,无需配置 extension +路径或单独安装 native library: ```python -from pathlib import Path - config = BuiltinConfig( - database=SQLiteConfig( - url="sqlite+aiosqlite:///powercontext.db", - vec1_extension=Path("/opt/sqlite-extensions/vec1"), - ) + database=SQLiteConfig(url="sqlite+aiosqlite:///powercontext.db") ) async with open_builtin_runtime( config, @@ -109,7 +104,7 @@ async with open_builtin_runtime( ... ``` -SQLite profile 会组合 FTS5 和 Vec1 strategy,并通过 Memory capabilities 报告 `fts`、`vector` 和 `hybrid`。持久化 +SQLite profile 会组合 FTS5 和 sqlite-vec strategy,并通过 Memory capabilities 报告 `fts`、`vector` 和 `hybrid`。持久化 projection 与 query vector 必须使用同一个 `EmbeddingProfile`,包括 model name、dimension、distance 和 normalization。更换 profile 后,应先重建 projection,再恢复 vector search。 @@ -141,7 +136,7 @@ async with open_builtin_runtime( OceanBase profile 与 SQLite 使用相同的 index 组合方式。全文 strategy 始终可用;提供 embedding model 后,会增加 `VECTOR` projection 和 HNSW strategy,并启用 `vector` 与 `hybrid` mode。SQLite FTS5 与 OceanBase FULLTEXT -服务于同一组 Runtime 和 Server search 调用,Vec1 与 HNSW 也通过同一接口提供向量检索。 +服务于同一组 Runtime 和 Server search 调用,sqlite-vec 与 HNSW 也通过同一接口提供向量检索。 ## 运行检查 @@ -150,7 +145,7 @@ OceanBase profile 与 SQLite 使用相同的 index 组合方式。全文 strateg - 所选 profile 能够成功打开并完成初始化; - 每个 tenant 或 project 映射到预期的 scope ID; - 定时 extraction 已经配置 candidate pipeline; -- Vec1 配置包含匹配的 embedding model; +- SQLite vector search 配置了匹配的 embedding model; - OceanBase vector search 配置了匹配的 embedding model; - capability response 与实际初始化的 index 一致; - database 和 scheduler 资源会随进程生命周期关闭。 diff --git a/docs/zh/development/pydantic-ai-inference.md b/docs/zh/development/pydantic-ai-inference.md index ce438799f..70b65463e 100644 --- a/docs/zh/development/pydantic-ai-inference.md +++ b/docs/zh/development/pydantic-ai-inference.md @@ -32,13 +32,12 @@ vector search 需要 embedding model 和完整的 deployment profile: export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_MODEL="provider:embedding-model" export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_PROFILE_ID="project-embedding-v1" export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_DIMENSION="1536" -export POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION="/opt/sqlite-extensions/vec1" ``` provider credential 仍使用所选 Pydantic AI provider 支持的环境变量,不属于 PowerContext model 字段。 Server 会拒绝不完整的 embedding profile。`embedding_model`、`embedding_profile_id` 和 -`embedding_dimension` 必须一起配置。Vec1 也依赖这组配置,因为 index dimension 必须与持久化向量一致。 +`embedding_dimension` 必须一起配置。SQLite vector search 使用这组配置,因为 index dimension 必须与持久化向量一致。 ## 直接组合 generation @@ -95,8 +94,8 @@ embedding_model = PydanticAIEmbeddingModel( ) ``` -将这个 adapter 传给 `open_builtin_contexts()` 或 `open_builtin_runtime()`,并通过 `SQLiteConfig` 选择 Vec1 -extension。向量进入持久化之前,adapter 会校验输出数量、顺序、dimension 和数值有效性,并执行 profile 声明的单位归一化。 +将这个 adapter 与 `SQLiteConfig` 一起传给 `open_builtin_contexts()` 或 `open_builtin_runtime()` 后,会自动启用捆绑的 +sqlite-vec index。向量进入持久化之前,adapter 会校验输出数量、顺序、dimension 和数值有效性,并执行 profile 声明的单位归一化。 `EmbeddingProfile` 是 deployment contract,不是描述性 metadata。持久化 projection 和 query embedding 必须使用 同一个 profile。model、dimension 或 normalization 发生变化后,应从权威 Memory revision 重建 projection。 diff --git a/docs/zh/development/remote-access-implementation.md b/docs/zh/development/remote-access-implementation.md index ed59a46b0..4ca94f3eb 100644 --- a/docs/zh/development/remote-access-implementation.md +++ b/docs/zh/development/remote-access-implementation.md @@ -54,7 +54,7 @@ export POWERCONTEXT_SERVER_DATABASE_KIND="oceanbase" export POWERCONTEXT_SERVER_DATABASE_URL="mysql+aoceanbase://user:password@host:2881/powercontext?charset=utf8mb4" ``` -两种 database 都通过同一组 Server API 提供全文检索。配置 embedding model 后,SQLite 使用 Vec1,OceanBase +两种 database 都通过同一组 Server API 提供全文检索。配置 embedding model 后,SQLite 使用 sqlite-vec,OceanBase 使用 HNSW 提供 `vector` 和 `hybrid` 检索。 inference 配置见[配置 Pydantic AI 推理](pydantic-ai-inference.md)。 diff --git a/docs/zh/docs/reference/configuration.md b/docs/zh/docs/reference/configuration.md index ef3ce3044..5a4e328c9 100644 --- a/docs/zh/docs/reference/configuration.md +++ b/docs/zh/docs/reference/configuration.md @@ -172,24 +172,21 @@ export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_DIMENSION=1024 Embedding normalization 默认为 `unit`。 -### SQLite Vec1 +### SQLite 向量检索 -SQLite vector 和 hybrid search 还需要 0.7 或更高版本的 -[SQLite Vec1](https://sqlite.org/vec1/doc/trunk/doc/vec1.md) loadable extension。PowerContext 不负责下载、构建或更新 -这个 native library。请先获取适用于 Server 操作系统和架构的构建产物,再同时配置 extension 路径和完整的 -embedding profile: +SQLite vector 和 hybrid search 使用 [sqlite-vec](https://alexgarcia.xyz/sqlite-vec/),它已包含在 +`powercontext[builtin]` 依赖中。只需配置完整的 embedding profile,无需配置 extension 路径: ```bash export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_MODEL=provider:embedding-model export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_PROFILE_ID=embedding-model-v1 export POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_DIMENSION=1024 export POWERCONTEXT_SERVER_DATABASE_URL=sqlite+aiosqlite:////srv/powercontext/powercontext.db -export POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION=/opt/sqlite-extensions/vec1 powercontext server run ``` -extension 路径必须指向 SQLite loader 可以打开的 library。Server 打开数据库时,PowerContext 会加载并探测该 -extension;如果 library 不兼容或版本低于 0.7,启动会失败。 +Server 打开数据库时,PowerContext 会加载并探测捆绑的 extension;如果当前 platform 或 SQLite build 与 package +中的 library 不兼容,启动会失败。 在另一个终端确认初始化后的 Runtime 已报告 vector 和 hybrid search: @@ -197,8 +194,7 @@ extension;如果 library 不兼容或版本低于 0.7,启动会失败。 powercontext capabilities ``` -如果没有可用的 Vec1,请不要设置 `POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION`。即使没有 embedding model 或 -native extension,SQLite full-text search 仍然可用。 +没有配置 embedding model 时,SQLite full-text search 仍然可用。 ## CLI Server 连接 diff --git a/e2e/bub/uv.lock b/e2e/bub/uv.lock index 3e803e6c5..60cc5df6d 100644 --- a/e2e/bub/uv.lock +++ b/e2e/bub/uv.lock @@ -1643,6 +1643,8 @@ requires-dist = [ { name = "rfc8785", specifier = ">=0.1.4,<1" }, { name = "sqlalchemy", extras = ["asyncio"], marker = "extra == 'builtin'", specifier = ">=2,<3" }, { name = "sqlalchemy", extras = ["asyncio"], marker = "extra == 'server'", specifier = ">=2,<3" }, + { name = "sqlite-vec", marker = "extra == 'builtin'", specifier = ">=0.1.9,<0.2" }, + { name = "sqlite-vec", marker = "extra == 'server'", specifier = ">=0.1.9,<0.2" }, { name = "typer", marker = "extra == 'cli'", specifier = ">=0.16,<1" }, { name = "uvicorn", marker = "extra == 'server'", specifier = ">=0.34,<1" }, ] diff --git a/pyproject.toml b/pyproject.toml index 8cf2989e9..7f898291d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -43,6 +43,7 @@ builtin = [ "pydantic-settings>=2.7,<3", "pyobvector>=0.2.28,<0.3", "sqlalchemy[asyncio]>=2,<3", + "sqlite-vec>=0.1.9,<0.2", ] client = [ "httpx[socks]>=0.28,<1", diff --git a/src/powercontext/builtin/persistence/sqlite/memory_index.py b/src/powercontext/builtin/persistence/sqlite/memory_index.py index 86d8347e0..36c22e162 100644 --- a/src/powercontext/builtin/persistence/sqlite/memory_index.py +++ b/src/powercontext/builtin/persistence/sqlite/memory_index.py @@ -12,15 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""SQLite Memory search indexes using FTS5 and Vec1.""" +"""SQLite Memory search indexes using FTS5 and sqlite-vec.""" from __future__ import annotations import json -import re import struct from collections.abc import Mapping -from pathlib import Path from typing import Any from sqlalchemy import ( @@ -36,6 +34,7 @@ select, text, ) +from sqlalchemy.exc import SQLAlchemyError from sqlalchemy.ext.asyncio import AsyncConnection from powercontext.artifacts import ArtifactRef @@ -105,11 +104,10 @@ ) -SQLITE_MEMORY_VEC1_TABLES = (SQLITE_MEMORY_VECTOR_ENTRIES_TABLE,) +SQLITE_MEMORY_VECTOR_TABLES = (SQLITE_MEMORY_VECTOR_ENTRIES_TABLE,) _FTS_TABLE_NAME = "pc_memory_entry_fts" -_MINIMUM_VEC1_VERSION = (0, 7) _CREATE_FTS_SQL = """ CREATE VIRTUAL TABLE IF NOT EXISTS pc_memory_entry_fts USING fts5( scope_id UNINDEXED, @@ -152,27 +150,28 @@ ) """ ) -_CREATE_VECTOR_SQL = "CREATE VIRTUAL TABLE IF NOT EXISTS pc_memory_entry_vec USING vec1(embedding)" _DELETE_VECTOR_SQL = text("DELETE FROM pc_memory_entry_vec WHERE rowid = :vector_id") _INSERT_VECTOR_SQL = text("INSERT INTO pc_memory_entry_vec (rowid, embedding) VALUES (:vector_id, :embedding)") _SELECT_VECTOR_SQL = text("SELECT embedding FROM pc_memory_entry_vec WHERE rowid = :vector_id") _VECTOR_SEARCH_SQL = text( """ - WITH candidates AS ( - SELECT rowid, embedding - FROM pc_memory_entry_vec(:query_vector, :parameters) + WITH nearest AS ( + SELECT rowid, distance + FROM pc_memory_entry_vec + WHERE embedding MATCH :query_vector + AND k = :neighbor_limit ) SELECT m.memory_artifact_id, m.head_revision, m.entry_id, m.entry_version_id, v.text, - vec1_l2_distance(:query_vector, c.embedding) AS distance - FROM candidates AS c - JOIN pc_memory_vector_entries AS m ON m.vector_id = c.rowid + nearest.distance + FROM nearest + JOIN pc_memory_vector_entries AS m ON m.vector_id = nearest.rowid JOIN pc_memory_entry_versions AS v ON v.scope_id = m.scope_id AND v.memory_artifact_id = m.memory_artifact_id AND v.entry_version_id = m.entry_version_id WHERE m.scope_id = :scope_id AND m.memory_artifact_id IN (SELECT value FROM json_each(:memory_artifact_ids)) - ORDER BY vec1_l2_distance(:query_vector, c.embedding), + ORDER BY nearest.distance, m.memory_artifact_id, m.entry_id, m.entry_version_id LIMIT :candidate_limit """ @@ -304,31 +303,29 @@ async def _insert_row( await connection.execute(_INSERT_FTS_SQL, values) -class SQLiteMemoryVec1Index: - """SQLite Vec1 strategy over rebuildable active-head embeddings.""" +class SQLiteMemoryVectorIndex: + """sqlite-vec strategy over rebuildable active-head embeddings.""" - tables: tuple[Table, ...] = SQLITE_MEMORY_VEC1_TABLES + tables: tuple[Table, ...] = SQLITE_MEMORY_VECTOR_TABLES - def __init__(self, extension: str | Path, profile: EmbeddingProfile) -> None: + def __init__(self, profile: EmbeddingProfile) -> None: if profile.dimension < 1 or profile.distance != "l2" or profile.normalization != "unit": raise CapabilityNotSupportedError( "vector", - "Vec1 requires a positive unit-normalized L2 embedding profile", + "sqlite-vec requires a positive unit-normalized L2 embedding profile", ) - self.extension = str(extension) self.profile = profile self.capabilities = MemoryCapabilities(vector=True, embedding_profile=profile, fts=False) async def initialize(self, connection: AsyncConnection, /) -> None: if connection.dialect.name != "sqlite": - raise CapabilityNotSupportedError("sqlite-vec1") + raise CapabilityNotSupportedError("sqlite-vec") try: - info = (await connection.exec_driver_sql("SELECT vec1_info()")).scalar_one_or_none() - except Exception as error: - raise CapabilityNotSupportedError("vector", "SQLite Vec1 probe failed") from error - _validate_vec1_info(info) - try: - await connection.exec_driver_sql(_CREATE_VECTOR_SQL) + await connection.exec_driver_sql("SELECT vec_version()") + await connection.exec_driver_sql( + "CREATE VIRTUAL TABLE IF NOT EXISTS pc_memory_entry_vec " + f"USING vec0(embedding float[{self.profile.dimension}])" + ) probe = _pack_vector((0.0,) * self.profile.dimension) await connection.execute( _INSERT_VECTOR_SQL, @@ -336,15 +333,15 @@ async def initialize(self, connection: AsyncConnection, /) -> None: ) row = ( await connection.exec_driver_sql( - "SELECT rowid FROM pc_memory_entry_vec(?, ?)", - (probe, json.dumps({"k": 1}, separators=(",", ":"))), + "SELECT rowid FROM pc_memory_entry_vec WHERE embedding MATCH ? AND k = 1", + (probe,), ) ).one_or_none() await connection.execute(_DELETE_VECTOR_SQL, {"vector_id": -1}) - except Exception as error: - raise CapabilityNotSupportedError("vector", "SQLite Vec1 probe failed") from error + except SQLAlchemyError as error: + raise CapabilityNotSupportedError("vector", "sqlite-vec probe failed") from error if row is None or int(row[0]) != -1: - raise CapabilityNotSupportedError("vector", "SQLite Vec1 probe returned an invalid row") + raise CapabilityNotSupportedError("vector", "sqlite-vec probe returned an invalid row") async def replace( self, @@ -417,7 +414,7 @@ async def search( _VECTOR_SEARCH_SQL, { "query_vector": query_vector, - "parameters": json.dumps({"k": total}, separators=(",", ":")), + "neighbor_limit": total, "scope_id": scope_id, "memory_artifact_ids": json.dumps( tuple(ref.artifact_id for ref in request.memories), @@ -534,10 +531,10 @@ def _pack_vector(vector: tuple[float, ...]) -> bytes: def _unpack_vector(value: object, dimension: int) -> tuple[float, ...]: if not isinstance(value, bytes | bytearray | memoryview): - raise CapabilityNotSupportedError("vector", "SQLite Vec1 returned an invalid vector") + raise CapabilityNotSupportedError("vector", "sqlite-vec returned an invalid vector") packed = bytes(value) if len(packed) != struct.calcsize(f"={dimension}f"): - raise CapabilityNotSupportedError("vector", "SQLite Vec1 returned the wrong vector dimension") + raise CapabilityNotSupportedError("vector", "sqlite-vec returned the wrong vector dimension") return tuple(struct.unpack(f"={dimension}f", packed)) @@ -550,9 +547,3 @@ def _embedding_hash(profile: EmbeddingProfile, entry_hash: str) -> str: normalization=profile.normalization, entry_content_hash=entry_hash, ) - - -def _validate_vec1_info(info: object) -> None: - match = re.search(r"\bversion\s+(\d+)\.(\d+)\b", "" if info is None else str(info)) - if match is None or tuple(int(part) for part in match.groups()) < _MINIMUM_VEC1_VERSION: - raise CapabilityNotSupportedError("vector", "SQLite Vec1 0.7 or newer is required") diff --git a/src/powercontext/builtin/persistence/sqlite/profile.py b/src/powercontext/builtin/persistence/sqlite/profile.py index c9cdc6be8..79c64e6ca 100644 --- a/src/powercontext/builtin/persistence/sqlite/profile.py +++ b/src/powercontext/builtin/persistence/sqlite/profile.py @@ -23,6 +23,7 @@ from typing import Literal from weakref import WeakKeyDictionary +import sqlite_vec from aiosqlite import Connection from pydantic import BaseModel, ConfigDict, Field, field_validator from sqlalchemy import Table, event @@ -50,7 +51,6 @@ class SQLiteConfig(BaseModel): journal_mode: Literal["WAL", "DELETE", "MEMORY"] = "WAL" foreign_keys: bool = True echo: bool = False - vec1_extension: Path | None = None @field_validator("url") @classmethod @@ -118,10 +118,8 @@ def _create_database_directory(value: str) -> None: def _configure_sqlite(engine: AsyncEngine, config: SQLiteConfig) -> None: @event.listens_for(engine.sync_engine, "connect") def set_pragmas(dbapi_connection: DBAPIConnection, _connection_record: object) -> None: - extension = config.vec1_extension - if extension is not None: - run_async = dbapi_connection.run_async - run_async(lambda connection: _load_extension(connection, extension)) + run_async = dbapi_connection.run_async + run_async(_load_sqlite_vec) cursor = dbapi_connection.cursor() try: cursor.execute(f"PRAGMA busy_timeout = {config.busy_timeout_ms}") @@ -130,10 +128,10 @@ def set_pragmas(dbapi_connection: DBAPIConnection, _connection_record: object) - cursor.close() -async def _load_extension(connection: Connection, extension: Path) -> None: +async def _load_sqlite_vec(connection: Connection) -> None: await connection.enable_load_extension(True) try: - await connection.load_extension(str(extension)) + await connection.load_extension(sqlite_vec.loadable_path()) finally: await connection.enable_load_extension(False) diff --git a/src/powercontext/builtin/runtime/composition.py b/src/powercontext/builtin/runtime/composition.py index 14f3b3637..acf75d7b4 100644 --- a/src/powercontext/builtin/runtime/composition.py +++ b/src/powercontext/builtin/runtime/composition.py @@ -53,7 +53,7 @@ ) from powercontext.builtin.persistence.oceanbase.profile import OceanBaseConfig, OceanBaseProfile from powercontext.builtin.persistence.sqlite.experience_index import SQLiteExperienceFTSIndex -from powercontext.builtin.persistence.sqlite.memory_index import SQLiteMemoryFTSIndex, SQLiteMemoryVec1Index +from powercontext.builtin.persistence.sqlite.memory_index import SQLiteMemoryFTSIndex, SQLiteMemoryVectorIndex from powercontext.builtin.persistence.sqlite.profile import SQLiteConfig, SQLiteProfile from powercontext.builtin.persistence.tables import BUILTIN_TABLES from powercontext.builtin.runtime.application import BuiltinRuntime @@ -317,10 +317,8 @@ async def open_builtin_contexts( if isinstance(database, SQLiteConfig): experience_index = SQLiteExperienceFTSIndex() indexes: list[MemoryIndex] = [SQLiteMemoryFTSIndex()] - if database.vec1_extension is not None: - if embedding_model is None: - raise ValueError("SQLite Vec1 requires an embedding model") # noqa: TRY003 - indexes.append(SQLiteMemoryVec1Index(database.vec1_extension, embedding_model.profile)) + if embedding_model is not None: + indexes.append(SQLiteMemoryVectorIndex(embedding_model.profile)) index = CompositeMemoryIndex(*indexes) async with SQLiteProfile.open( database, diff --git a/tests/builtin/persistence/test_sqlite_profile.py b/tests/builtin/persistence/test_sqlite_profile.py index ec5247767..5c80a07e8 100644 --- a/tests/builtin/persistence/test_sqlite_profile.py +++ b/tests/builtin/persistence/test_sqlite_profile.py @@ -51,6 +51,7 @@ async def scenario() -> None: async with SQLiteProfile.open(SQLiteConfig(), tables=SHARED_TABLES) as profile: async with profile.database.transaction() as connection: assert await connection.scalar(select(func.sqlite_version())) is not None + assert (await connection.exec_driver_sql("SELECT vec_version()")).scalar_one() != "" pragma = await connection.exec_driver_sql("PRAGMA foreign_keys") assert int(pragma.scalar() or 0) == 1 await connection.execute( diff --git a/tests/e2e/real_experience_skill/harness.py b/tests/e2e/real_experience_skill/harness.py index 5784b6c9b..009a751a4 100644 --- a/tests/e2e/real_experience_skill/harness.py +++ b/tests/e2e/real_experience_skill/harness.py @@ -2012,12 +2012,6 @@ def _validate_configured_settings(settings: ServerSettings) -> None: _fail("configured E2E requires POWERCONTEXT_SERVER_INFERENCE_GENERATION_MODEL") if inference.embedding_model is None: _fail("configured E2E requires POWERCONTEXT_SERVER_INFERENCE_EMBEDDING_MODEL") - if isinstance(settings.database, SQLiteConfig): - extension = settings.database.vec1_extension - if extension is None or not extension.is_file(): - _fail("configured SQLite E2E requires an installed Vec1 extension file") - if not hasattr(sqlite3.Connection, "enable_load_extension"): - _fail("configured SQLite E2E requires a Python build with SQLite extension loading enabled") def _new_configured_scopes() -> ConfiguredScopes: @@ -2288,7 +2282,7 @@ def _arguments(argv: Sequence[str] | None) -> argparse.Namespace: parser.add_argument( "--configured", action="store_true", - help="Use real generation, embedding, database, Vec1, and External Skill settings from the environment.", + help="Use real generation, embedding, database, and External Skill settings from the environment.", ) parser.add_argument( "--env-file", diff --git a/tests/e2e/test_memory_search_concurrency.py b/tests/e2e/test_memory_search_concurrency.py index ba8328f30..940d92c3d 100644 --- a/tests/e2e/test_memory_search_concurrency.py +++ b/tests/e2e/test_memory_search_concurrency.py @@ -238,10 +238,4 @@ def _database_config( pytest.skip("set POWERCONTEXT_TEST_OCEANBASE_URL to a dedicated OceanBase MySQL-mode test database") return OceanBaseConfig(url=SecretStr(url)) - extension = os.environ.get("POWERCONTEXT_VEC1_EXTENSION") - if mode in {"vector", "hybrid"} and (extension is None or not Path(extension).is_file()): - pytest.skip("set POWERCONTEXT_VEC1_EXTENSION to a Vec1 extension file") - return SQLiteConfig( - url=f"sqlite+aiosqlite:///{tmp_path / f'memory-search-{mode}.db'}", - vec1_extension=None if extension is None else Path(extension), - ) + return SQLiteConfig(url=f"sqlite+aiosqlite:///{tmp_path / f'memory-search-{mode}.db'}") diff --git a/tests/e2e/test_observability.py b/tests/e2e/test_observability.py index 6f4dad4c3..08a8dbb9e 100644 --- a/tests/e2e/test_observability.py +++ b/tests/e2e/test_observability.py @@ -593,8 +593,8 @@ def test_vector_search_exports_embedding_under_memory_search_without_recording_t lambda _model, **_kwargs: TestEmbeddingModel(dimensions=3), ) monkeypatch.setattr( - "powercontext.builtin.runtime.composition.SQLiteMemoryFTSIndex", - _VectorMemoryIndex, + "powercontext.builtin.runtime.composition.SQLiteMemoryVectorIndex", + lambda _profile: _VectorMemoryIndex(), ) exporter = InMemorySpanExporter() diff --git a/tests/e2e/test_runtime_server.py b/tests/e2e/test_runtime_server.py index f0a80121f..61a6de486 100644 --- a/tests/e2e/test_runtime_server.py +++ b/tests/e2e/test_runtime_server.py @@ -597,13 +597,7 @@ def test_server_databases_share_vector_and_hybrid_search_behavior( pytest.skip("set POWERCONTEXT_TEST_OCEANBASE_URL to a dedicated OceanBase MySQL-mode test database") database = OceanBaseConfig(url=SecretStr(OCEANBASE_URL)) else: - configured = os.environ.get("POWERCONTEXT_VEC1_EXTENSION") - if configured is None or not Path(configured).is_file(): - pytest.skip("set POWERCONTEXT_VEC1_EXTENSION to a Vec1 extension file") - database = SQLiteConfig( - url=f"sqlite+aiosqlite:///{tmp_path / 'vector-runtime.db'}", - vec1_extension=Path(configured), - ) + database = SQLiteConfig(url=f"sqlite+aiosqlite:///{tmp_path / 'vector-runtime.db'}") scope_id = f"vector-e2e-{uuid4()}" app = create_server_app( settings=ServerSettings( diff --git a/tests/e2e/test_sqlite_vec1.py b/tests/e2e/test_sqlite_vec.py similarity index 82% rename from tests/e2e/test_sqlite_vec1.py rename to tests/e2e/test_sqlite_vec.py index 9e5c9ef57..2e63da4c9 100644 --- a/tests/e2e/test_sqlite_vec1.py +++ b/tests/e2e/test_sqlite_vec.py @@ -15,10 +15,6 @@ from __future__ import annotations import asyncio -import os -from pathlib import Path - -import pytest from powercontext.builtin.artifacts.memory import ( EmbeddingProfile, @@ -53,23 +49,10 @@ def vector(text: str) -> tuple[float, float, float]: return EmbeddingResult(vectors=vectors) -def _vec1_extension() -> Path: - configured = os.environ.get("POWERCONTEXT_VEC1_EXTENSION") - if configured is None: - pytest.skip("POWERCONTEXT_VEC1_EXTENSION is not configured") - extension = Path(configured) - if not extension.is_file(): - pytest.skip("POWERCONTEXT_VEC1_EXTENSION does not point to a file") - return extension - - -def test_sqlite_vec1_supports_vector_and_hybrid_search(tmp_path) -> None: +def test_sqlite_vec_supports_vector_and_hybrid_search(tmp_path) -> None: async def scenario() -> None: model = _KeywordEmbeddingModel() - config = SQLiteConfig( - url=f"sqlite+aiosqlite:///{tmp_path / 'memory.db'}", - vec1_extension=_vec1_extension(), - ) + config = SQLiteConfig(url=f"sqlite+aiosqlite:///{tmp_path / 'memory.db'}") async with open_builtin_contexts( BuiltinConfig(database=config), embedding_model=model, diff --git a/tests/test_server.py b/tests/test_server.py index 4f49fba1a..57bbc7396 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -55,7 +55,7 @@ class _FailingEmbeddingModel: model="test:embedding", dimension=3, distance="l2", - normalization="none", + normalization="unit", ) def __init__(self, error: Exception) -> None: @@ -131,16 +131,6 @@ def test_settings_load_server_environment(monkeypatch) -> None: assert settings.external_skills.codex_roots[0].path.as_posix() == "/srv/project/.agents/skills" -def test_server_settings_vec1_preserves_file_database(tmp_path, monkeypatch) -> None: - data_dir = tmp_path / "powercontext-data" - monkeypatch.setenv("POWERCONTEXT_HOME", str(data_dir)) - monkeypatch.setenv("POWERCONTEXT_SERVER_DATABASE_VEC1_EXTENSION", str(tmp_path / "vec1")) - - settings = ServerSettings() - - assert settings.database.url == f"sqlite+aiosqlite:///{data_dir / 'powercontext.db'}" - - def test_server_settings_select_oceanbase(monkeypatch) -> None: url = "mysql+aoceanbase://root:test@127.0.0.1:2881/powercontext?charset=utf8mb4" monkeypatch.setenv("POWERCONTEXT_SERVER_DATABASE_KIND", "oceanbase") diff --git a/tests/test_server_embedding.py b/tests/test_server_embedding.py index f57c90319..ea08a7bfc 100644 --- a/tests/test_server_embedding.py +++ b/tests/test_server_embedding.py @@ -65,6 +65,6 @@ def test_embedding_settings_reject_partial_profiles(values: dict[str, object]) - InferenceConfig.model_validate(values) -def test_component_config_rejects_unknown_values() -> None: - with pytest.raises(ValidationError): - SQLiteConfig.model_validate({"legacy_path": "powercontext.db"}) +def test_sqlite_config_rejects_the_removed_vec1_extension_path() -> None: + with pytest.raises(ValidationError, match="vec1_extension"): + SQLiteConfig.model_validate({"vec1_extension": "/opt/sqlite-extensions/vec1"}) diff --git a/uv.lock b/uv.lock index 70ab422b4..7a7f9bf5b 100644 --- a/uv.lock +++ b/uv.lock @@ -1807,6 +1807,7 @@ builtin = [ { name = "pydantic-settings" }, { name = "pyobvector" }, { name = "sqlalchemy", extra = ["asyncio"] }, + { name = "sqlite-vec" }, ] cli = [ { name = "httpx", extra = ["socks"] }, @@ -1834,6 +1835,7 @@ server = [ { name = "pydantic-settings" }, { name = "pyobvector" }, { name = "sqlalchemy", extra = ["asyncio"] }, + { name = "sqlite-vec" }, { name = "uvicorn" }, ] tracing-otlp = [ @@ -1881,6 +1883,7 @@ requires-dist = [ { name = "pyobvector", marker = "extra == 'builtin'", specifier = ">=0.2.28,<0.3" }, { name = "rfc8785", specifier = ">=0.1.4,<1" }, { name = "sqlalchemy", extras = ["asyncio"], marker = "extra == 'builtin'", specifier = ">=2,<3" }, + { name = "sqlite-vec", marker = "extra == 'builtin'", specifier = ">=0.1.9,<0.2" }, { name = "typer", marker = "extra == 'cli'", specifier = ">=0.16,<1" }, { name = "uvicorn", marker = "extra == 'server'", specifier = ">=0.34,<1" }, ] @@ -2880,6 +2883,18 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/f2/2f/2076eca54f6a8ed1c86301bdb4bb2ae4181b0c1c4dbc041062ca997dc1b2/sqlglot-30.13.0-py3-none-any.whl", hash = "sha256:08f87ff7b052246d61b731628c8c2db0bc91f2c9e69f5ba68a1d160a9f5b49b1", size = 719120, upload-time = "2026-07-20T20:16:53.248Z" }, ] +[[package]] +name = "sqlite-vec" +version = "0.1.9" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/68/85/9fad0045d8e7c8df3e0fa5a56c630e8e15ad6e5ca2e6106fceb666aa6638/sqlite_vec-0.1.9-py3-none-macosx_10_6_x86_64.whl", hash = "sha256:1b62a7f0a060d9475575d4e599bbf94a13d85af896bc1ce86ee80d1b5b48e5fb", size = 131171, upload-time = "2026-03-31T08:02:31.717Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a4/3d/3677e0cd2f92e5ebc43cd29fbf565b75582bff1ccfa0b8327c7508e1084f/sqlite_vec-0.1.9-py3-none-macosx_11_0_arm64.whl", hash = "sha256:1d52e30513bae4cc9778ddbf6145610434081be4c3afe57cd877893bad9f6b6c", size = 165434, upload-time = "2026-03-31T08:02:32.712Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/00/d4/f2b936d3bdc38eadcbd2a87875815db36430fab0363182ba5d12cd8e0b51/sqlite_vec-0.1.9-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e921e592f24a5f9a18f590b6ddd530eb637e2d474e3b1972f9bbeb773aa3cb9", size = 160076, upload-time = "2026-03-31T08:02:33.796Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/6f/ad/6afd073b0f817b3e03f9e37ad626ae341805891f23c74b5292818f49ac63/sqlite_vec-0.1.9-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux1_x86_64.whl", hash = "sha256:1515727990b49e79bcaf75fdee2ffc7d461f8b66905013231251f1c8938e7786", size = 163388, upload-time = "2026-03-31T08:02:34.888Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/42/89/81b2907cda14e566b9bf215e2ad82fc9b349edf07d2010756ffdb902f328/sqlite_vec-0.1.9-py3-none-win_amd64.whl", hash = "sha256:4a28dc12fa4b53d7b1dced22da2488fade444e96b5d16fd2d698cd670675cf32", size = 292804, upload-time = "2026-03-31T08:02:36.035Z" }, +] + [[package]] name = "sse-starlette" version = "3.4.5"