test(core): benchmark multilingual embeddings - #1392
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7138d80c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a6b7609b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c703e56f1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3aa520ca1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
|
Fixed the Windows SQLite integration failure in afd4dbb. This was not Milvus-related: the Windows Python SQLite build omits the optional |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afd4dbb88a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55853e3846
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a17d05d3f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70b555350d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ba26e78f87
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 253183d499
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c9d11a4d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d05ab12d79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
Basic Memory's default FastEmbed model,
BAAI/bge-small-en-v1.5, is intentionally small andEnglish-focused. Issue #1372 asks us to evaluate multilingual alternatives from reproducible
evidence before changing any default or planning a Cloud-wide reindex.
This PR supplies the bounded Core evidence: a checked-in judged corpus, a production-path harness,
and baseline/candidate results across local and hosted storage adapters. BGE remains the default.
Advances #1372. Cloud capacity and development validation are tracked separately in
basic-memory-cloud#1898.What Changed
multilingual-retrieval-v2: 17 notes and 23 judged queries covering English, Chinese,Japanese, Korean, Arabic, Russian, Spanish, Thai, mixed-language text, cross-language retrieval,
chunk boundaries, and negative queries.
PostgreSQL/Milvus, with vector and hybrid retrieval modes.
latency, model load, indexing throughput, RSS, cache size, and vector storage.
explicitly retaining BGE as the default.
Implementation Details
by another ONNX model.
production cutoff. This separates embedding/ranking failures from threshold-filtered empties.
by an isolated Milvus Lite database. It is not modeled as a replacement SQL backend.
asymmetric
passage:/query:prefixes are part of its explicit contract.retrieval-query/retrieval-passage tasks and its license requires separate Cloud clearance.
dbstatto count only the vector manifest, its indexes,sqlite-vec shadow tables, and their schema-owned autoindexes; unrelated database and FTS pages
are excluded.
the returned matched chunk contains the later legal-hold passage.
resource.getrusageis unavailable.FastEmbed's Hugging Face-first source order when its cataloged URL-fallback cache also exists.
manifest.
dbstatand loadable-extensioncapabilities and skips only that assertion when the platform SQLite build lacks either one.
lower-is-better.
Initial evidence favors multilingual MiniLM for further Cloud testing: it preserved the English
baseline, improved vector MRR@10 from
0.7684to0.8684, halved the wrong-top rate, and acceptedno negative queries. The result was identical across sqlite-vec, pgvector, and Milvus. It is not
selected as a new default because neither long-note boundary query retrieved the relevant later
passage as the highest-scoring chunk, the
0.55cutoff hid three other correctly ranked positives,and its cache/RSS cost needs Cloud-equivalent validation.
Testing
Automated
UV_PYTHON=3.12 just fast-check: passed.UV_PYTHON=3.12 just typecheck: passed.UV_PYTHON=3.12 BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run --extra milvus pytest -p pytest_mock -q --no-cov --import-mode=importlib test-int/semantic/test_multilingual_benchmark_contract.py: 21 passed.UV_PYTHON=3.12 BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run --extra milvus pytest -p pytest_mock -q --no-cov --import-mode=importlib test-int/semantic/test_semantic_quality.py -k "sqlite-fastembed or postgres-fastembed": 2 passed, 3 deselected.Benchmark runs
just benchmark-multilingual-compare sqlite vector 0.55: BGE and MiniLM passed.just benchmark-multilingual-compare postgres vector 0.55: BGE and MiniLM passed.just benchmark-multilingual-compare milvus vector 0.55: BGE and MiniLM passed.just benchmark-multilingual-compare sqlite hybrid 0.55: BGE and MiniLM passed.just benchmark-multilingual-compare postgres hybrid 0.55: BGE and MiniLM passed.just benchmark-multilingual-compare milvus hybrid 0.55: BGE and MiniLM passed.git diff --check: passed.Generated
.benchmarks/*.jsonlartifacts are intentionally ignored; the reviewed results andmachine context are recorded in
docs/multilingual-embedding-benchmark.md.Risks / Follow-ups
production latency SLO or HNSW capacity result.
and record the exact provider/model behavior.
basic-memory-cloud#1898; this PR makes no production, fleet-reindex, or default-model change.