22
33Thank you for your interest in contributing to Selectools! We welcome contributions from the community.
44
5+ ** Current Version:** v0.8.0 (Embeddings & RAG Support)
6+ ** Test Status:** ✅ 463 tests passing (100%)
7+ ** Next Release:** v0.9.0 (Hybrid Search, Reranking)
8+
59## Getting Started
610
711### Development Setup
@@ -49,9 +53,18 @@ The hooks will automatically run on staged files when you commit. They include:
49535 . ** Set up API keys for testing**
5054
5155``` bash
56+ # LLM Providers
5257export OPENAI_API_KEY=" your-key-here"
5358export ANTHROPIC_API_KEY=" your-key-here" # Optional
5459export GEMINI_API_KEY=" your-key-here" # Optional
60+
61+ # Embedding Providers (for RAG features)
62+ export VOYAGE_API_KEY=" your-key-here" # Optional (Anthropic embeddings)
63+ export COHERE_API_KEY=" your-key-here" # Optional
64+
65+ # Vector Stores (for RAG features)
66+ export PINECONE_API_KEY=" your-key-here" # Optional
67+ export PINECONE_ENV=" your-env-here" # Optional
5568```
5669
5770## Available Scripts
@@ -61,7 +74,7 @@ Similar to `npm run` scripts, here are the common commands for this project:
6174### Testing
6275
6376``` bash
64- # Run all tests
77+ # Run all tests (463 tests)
6578pytest tests/ -v
6679
6780# Run tests quietly (summary only)
@@ -70,11 +83,20 @@ pytest tests/ -q
7083# Run specific test file
7184pytest tests/test_framework.py -v
7285
86+ # Run RAG-specific tests
87+ pytest tests/test_embedding_providers.py -v
88+ pytest tests/test_vector_stores_crud.py -v
89+ pytest tests/test_rag_workflow.py -v
90+
7391# Run tests matching a pattern
7492pytest tests/ -k " test_tool" -v
93+ pytest tests/ -k " rag" -v
7594
7695# Run with coverage (if installed)
7796pytest tests/ --cov=src/selectools --cov-report=html
97+
98+ # Skip end-to-end tests (require real API keys)
99+ pytest tests/ -m " not e2e" -v
78100```
79101
80102### Code Quality
@@ -111,6 +133,11 @@ python examples/customer_support_bot.py
111133
112134# Data analysis agent example
113135python examples/data_analysis_agent.py
136+
137+ # RAG (Retrieval-Augmented Generation) examples
138+ python examples/rag_basic_demo.py # Basic RAG with in-memory store
139+ python examples/rag_advanced_demo.py # Advanced RAG with PDF + SQLite
140+ python examples/semantic_search_demo.py # Multi-provider semantic search
114141```
115142
116143### Development Scripts
@@ -220,31 +247,66 @@ selectools/
220247│ ├── cli.py # CLI interface
221248│ ├── env.py # Environment variable loading
222249│ ├── exceptions.py # Custom exception classes
250+ │ ├── models.py # Model registry (120+ models)
223251│ ├── parser.py # TOOL_CALL parser
224252│ ├── pricing.py # LLM pricing data and cost calculation
225253│ ├── prompt.py # Prompt builder
226254│ ├── tools.py # Tool definitions and registry
227255│ ├── types.py # Core types (Message, Role, etc.)
228- │ ├── usage.py # Usage tracking (tokens, costs)
229- │ └── providers/ # LLM provider adapters
230- │ ├── base.py # Provider interface
231- │ ├── openai_provider.py # OpenAI implementation
232- │ ├── anthropic_provider.py # Anthropic implementation
233- │ ├── gemini_provider.py # Google Gemini implementation
234- │ └── stubs.py # Local/test providers
235- ├── tests/ # Test suite
256+ │ ├── usage.py # Usage tracking (tokens, costs, analytics)
257+ │ ├── providers/ # LLM provider adapters
258+ │ │ ├── base.py # Provider interface
259+ │ │ ├── openai_provider.py # OpenAI implementation
260+ │ │ ├── anthropic_provider.py # Anthropic implementation
261+ │ │ ├── gemini_provider.py # Google Gemini implementation
262+ │ │ ├── ollama_provider.py # Ollama local models
263+ │ │ └── stubs.py # Local/test providers
264+ │ ├── embeddings/ # Embedding providers (v0.8.0)
265+ │ │ ├── provider.py # EmbeddingProvider interface
266+ │ │ ├── openai.py # OpenAI embeddings
267+ │ │ ├── anthropic.py # Anthropic/Voyage embeddings
268+ │ │ ├── gemini.py # Google Gemini embeddings
269+ │ │ └── cohere.py # Cohere embeddings
270+ │ ├── rag/ # RAG components (v0.8.0)
271+ │ │ ├── __init__.py # RAGAgent high-level API
272+ │ │ ├── vector_store.py # VectorStore interface
273+ │ │ ├── chunking.py # Text splitting strategies
274+ │ │ ├── loaders.py # Document loaders
275+ │ │ ├── tools.py # RAGTool & SemanticSearchTool
276+ │ │ └── stores/ # Vector store implementations
277+ │ │ ├── memory.py # In-memory (NumPy)
278+ │ │ ├── sqlite.py # SQLite persistent storage
279+ │ │ ├── chroma.py # ChromaDB integration
280+ │ │ └── pinecone.py # Pinecone cloud storage
281+ │ └── toolbox/ # Pre-built tools (v0.5.1)
282+ │ ├── file.py # File operations
283+ │ ├── web.py # Web scraping & HTTP
284+ │ ├── data.py # Data processing
285+ │ ├── datetime_tools.py # Date/time utilities
286+ │ └── text.py # Text manipulation
287+ ├── tests/ # Test suite (463 tests)
236288│ ├── conftest.py # Pytest configuration
237289│ ├── test_framework.py # Core framework tests
238290│ ├── test_better_errors.py # Error handling tests
239291│ ├── test_cost_tracking.py # Cost tracking tests
240- │ └── test_e2e_providers.py # End-to-end provider tests
292+ │ ├── test_e2e_providers.py # End-to-end provider tests
293+ │ ├── test_models.py # Model registry tests
294+ │ ├── test_embedding_providers.py # Embedding provider tests
295+ │ ├── test_vector_stores_crud.py # Vector store CRUD tests
296+ │ ├── test_document_loaders.py # Document loader tests
297+ │ ├── test_text_chunking.py # Chunking strategy tests
298+ │ ├── test_rag_workflow.py # RAG integration tests
299+ │ └── test_sqlite_integration.py # SQLite persistence tests
241300├── examples/ # Usage examples
242301│ ├── search_weather.py # Basic tool usage
243302│ ├── async_agent_demo.py # Async operations
244303│ ├── conversation_memory_demo.py
245304│ ├── cost_tracking_demo.py
246305│ ├── customer_support_bot.py
247- │ └── data_analysis_agent.py
306+ │ ├── data_analysis_agent.py
307+ │ ├── rag_basic_demo.py # Basic RAG with in-memory store
308+ │ ├── rag_advanced_demo.py # Advanced RAG with PDF + SQLite
309+ │ └── semantic_search_demo.py # Multi-provider semantic search
248310└── scripts/ # Development scripts
249311 ├── README.md # Release documentation
250312 ├── release.py # Python release script
@@ -325,21 +387,31 @@ We especially welcome contributions in these areas:
325387- Add support for new LLM providers (Cohere, AI21, etc.)
326388- Improve existing provider implementations
327389- Add vision support to more providers
390+ - Add new embedding providers
328391
329392### 🛠️ ** New Tools**
330393- Pre-built tools for common use cases
331394- Integration with popular APIs and services
332395- Example tools demonstrating best practices
333396
397+ ### 🗄️ ** RAG & Vector Stores (New in v0.8.0!)**
398+ - Add new vector store integrations (Weaviate, Qdrant, Milvus)
399+ - Implement reranking support (Cohere, Jina)
400+ - Add hybrid search (vector + BM25 keyword)
401+ - Improve chunking strategies (contextual, agentic)
402+ - Create advanced RAG examples and tutorials
403+
334404### 📚 ** Documentation**
335405- Improve README examples
336- - Add tutorials and guides
406+ - Add tutorials and guides (especially for RAG features!)
337407- Fix typos and clarify confusing sections
408+ - Add comparison guides (vs LangChain, LlamaIndex)
338409
339410### 🧪 ** Testing**
340- - Increase test coverage
411+ - Increase test coverage (currently 463 tests passing!)
341412- Add integration tests
342413- Add performance benchmarks
414+ - Test RAG workflows with different vector stores
343415
344416### 🐛 ** Bug Fixes**
345417- Fix reported issues
@@ -448,6 +520,122 @@ def create_your_tool() -> Tool:
448520
4495213 . ** Update documentation**
450522
523+ ## Adding RAG Features (New in v0.8.0!)
524+
525+ ### Adding a New Vector Store
526+
527+ To add support for a new vector database:
528+
529+ 1 . ** Create a new vector store implementation**
530+
531+ ``` python
532+ # src/selectools/rag/stores/your_store.py
533+
534+ from typing import List, Optional
535+ from ..vector_store import VectorStore, Document, SearchResult
536+
537+ class YourVectorStore (VectorStore ):
538+ """ Your vector database implementation."""
539+
540+ def __init__ (self , embedder , ** config ):
541+ self .embedder = embedder
542+ # Initialize your vector DB client
543+ # self.client = your_db.Client(**config)
544+
545+ def add_documents (
546+ self ,
547+ documents : List[Document],
548+ embeddings : Optional[List[List[float ]]] = None
549+ ) -> List[str ]:
550+ """ Add documents to the store."""
551+ # Generate embeddings if not provided
552+ if embeddings is None :
553+ texts = [doc.text for doc in documents]
554+ embeddings = self .embedder.embed_texts(texts)
555+
556+ # Insert into your vector DB
557+ # ids = self.client.insert(...)
558+ return ids
559+
560+ def search (
561+ self ,
562+ query_embedding : List[float ],
563+ top_k : int = 5 ,
564+ filter : Optional[dict ] = None
565+ ) -> List[SearchResult]:
566+ """ Search for similar documents."""
567+ # results = self.client.search(query_embedding, top_k, filter)
568+ # return [SearchResult(document=..., score=...) for r in results]
569+ pass
570+
571+ def delete (self , ids : List[str ]) -> None :
572+ """ Delete documents by ID."""
573+ # self.client.delete(ids)
574+ pass
575+ ```
576+
577+ 2 . ** Register in the factory**
578+
579+ ``` python
580+ # src/selectools/rag/vector_store.py
581+
582+ @ staticmethod
583+ def create (store_type : str , embedder , ** kwargs ) -> " VectorStore" :
584+ if store_type == " your_store" :
585+ from .stores.your_store import YourVectorStore
586+ return YourVectorStore(embedder, ** kwargs)
587+ # ... existing stores
588+ ```
589+
590+ 3 . ** Add tests**
591+
592+ ``` python
593+ # tests/test_vector_stores_crud.py
594+
595+ class TestYourVectorStore :
596+ def test_add_and_search (self , mock_embedder ):
597+ store = VectorStore.create(" your_store" , embedder = mock_embedder)
598+ docs = [Document(text = " test" , metadata = {})]
599+ ids = store.add_documents(docs)
600+
601+ query_emb = mock_embedder.embed_query(" test" )
602+ results = store.search(query_emb, top_k = 1 )
603+
604+ assert len (results) == 1
605+ assert results[0 ].document.text == " test"
606+ ```
607+
608+ ### Adding a New Embedding Provider
609+
610+ 1 . ** Create the provider**
611+
612+ ``` python
613+ # src/selectools/embeddings/your_provider.py
614+
615+ from typing import List
616+ from .provider import EmbeddingProvider
617+
618+ class YourEmbeddingProvider (EmbeddingProvider ):
619+ def __init__ (self , model : str = " default-model" , api_key : str = None ):
620+ self .model = model
621+ self .api_key = api_key or os.getenv(" YOUR_API_KEY" )
622+ # self.client = your_sdk.Client(api_key=self.api_key)
623+
624+ def embed_texts (self , texts : List[str ]) -> List[List[float ]]:
625+ """ Embed multiple texts."""
626+ # response = self.client.embeddings.create(input=texts, model=self.model)
627+ # return [emb.embedding for emb in response.data]
628+ pass
629+
630+ def embed_query (self , query : str ) -> List[float ]:
631+ """ Embed a single query."""
632+ return self .embed_texts([query])[0 ]
633+ ```
634+
635+ 2 . ** Add model definitions to ` models.py ` **
636+
637+ 3 . ** Add tests and update documentation**
638+
451639## Code Review Process
452640
4536411 . A maintainer will review your PR
0 commit comments