feat(search): add turbovec vector backend - #88
Conversation
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cec59f477
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Model: r.cfg.Search.Embeddings.Model, | ||
| InputVersion: store.EmbeddingInputVersion, | ||
| Dimensions: dimensions, | ||
| VectorBackend: r.cfg.Search.Embeddings.VectorBackend, |
There was a problem hiding this comment.
When this option is set to turbovec, hybrid search now routes semantic scoring through the Python bridge. In environments where Python or the turbovec package is not installed, or where the embedding dimensions are unsupported, SearchMessagesSemantic returns a backend error such as score embeddings: run turbovec bridge...; however hybridSemanticUnavailable only falls back for missing embeddings or query-embedding errors. As a result discrawl search --mode hybrid fails instead of returning the FTS results it already computed, which breaks the documented fallback behavior for this optional backend.
Useful? React with 👍 / 👎.
1 participant
Summary
[search.embeddings].vector_backend = "turbovec"semantic scoring via Crawlkit v0.12.0 while preserving exact cosine as the default.Supersedes #83 because the contributor branch has
maintainerCanModify=false, so the pseudo-version could not be updated in place.Proof
DISCRAWL_TEST_TURBOVEC_PYTHON=/tmp/.../bin/python go test ./internal/cli ./internal/store -run 'TestSearchSemanticCommandUsesTurboVecBackend|TestSearchMessagesSemanticTurboVecBackend' -count=1 -vGOWORK=off go test ./...GOWORK=off go test -race ./...go test -count=1 ./... -coverprofile=coverage.outplus coverage floor: 85.0%golangci-lint run ./...staticcheck ./...deadcode -test ./...gofumpt -l .go vet ./...go build ./...go mod verifygovulncheck ./...gosec -exclude=G101,G115,G202,G301,G304 ./...gitleaks dir . --no-banner --redactgitleaks git --no-banner --redactgoreleaser release --snapshot --clean --skip=publishautoreview --mode local: clean, no accepted/actionable findings