Skip to content

Commit 17810dc

Browse files
committed
Update CONTRIBUTING.md and ROADMAP.md for v0.8.0 release and upcoming features
- Updated CONTRIBUTING.md to reflect the current version (v0.8.0) and next release (v0.9.0), including test status and new environment variable configurations for embedding and vector store providers. - Enhanced documentation with new RAG features, embedding providers, and testing instructions. - Revised ROADMAP.md to indicate the completion of v0.8.0 and outline planned features for v0.9.0, including hybrid search and advanced chunking strategies.
1 parent 2488ac5 commit 17810dc

2 files changed

Lines changed: 256 additions & 56 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 201 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
Thank you for your interest in contributing to Selectools! We welcome contributions from the community.
44

5+
**Current Version:** v0.8.0 (Embeddings & RAG Support)
6+
**Test Status:** ✅ 463 tests passing (100%)
7+
**Next Release:** v0.9.0 (Hybrid Search, Reranking)
8+
59
## Getting Started
610

711
### Development Setup
@@ -49,9 +53,18 @@ The hooks will automatically run on staged files when you commit. They include:
4953
5. **Set up API keys for testing**
5054

5155
```bash
56+
# LLM Providers
5257
export OPENAI_API_KEY="your-key-here"
5358
export ANTHROPIC_API_KEY="your-key-here" # Optional
5459
export GEMINI_API_KEY="your-key-here" # Optional
60+
61+
# Embedding Providers (for RAG features)
62+
export VOYAGE_API_KEY="your-key-here" # Optional (Anthropic embeddings)
63+
export COHERE_API_KEY="your-key-here" # Optional
64+
65+
# Vector Stores (for RAG features)
66+
export PINECONE_API_KEY="your-key-here" # Optional
67+
export PINECONE_ENV="your-env-here" # Optional
5568
```
5669

5770
## Available Scripts
@@ -61,7 +74,7 @@ Similar to `npm run` scripts, here are the common commands for this project:
6174
### Testing
6275

6376
```bash
64-
# Run all tests
77+
# Run all tests (463 tests)
6578
pytest tests/ -v
6679

6780
# Run tests quietly (summary only)
@@ -70,11 +83,20 @@ pytest tests/ -q
7083
# Run specific test file
7184
pytest tests/test_framework.py -v
7285

86+
# Run RAG-specific tests
87+
pytest tests/test_embedding_providers.py -v
88+
pytest tests/test_vector_stores_crud.py -v
89+
pytest tests/test_rag_workflow.py -v
90+
7391
# Run tests matching a pattern
7492
pytest tests/ -k "test_tool" -v
93+
pytest tests/ -k "rag" -v
7594

7695
# Run with coverage (if installed)
7796
pytest tests/ --cov=src/selectools --cov-report=html
97+
98+
# Skip end-to-end tests (require real API keys)
99+
pytest tests/ -m "not e2e" -v
78100
```
79101

80102
### Code Quality
@@ -111,6 +133,11 @@ python examples/customer_support_bot.py
111133

112134
# Data analysis agent example
113135
python examples/data_analysis_agent.py
136+
137+
# RAG (Retrieval-Augmented Generation) examples
138+
python examples/rag_basic_demo.py # Basic RAG with in-memory store
139+
python examples/rag_advanced_demo.py # Advanced RAG with PDF + SQLite
140+
python examples/semantic_search_demo.py # Multi-provider semantic search
114141
```
115142

116143
### Development Scripts
@@ -220,31 +247,66 @@ selectools/
220247
│ ├── cli.py # CLI interface
221248
│ ├── env.py # Environment variable loading
222249
│ ├── exceptions.py # Custom exception classes
250+
│ ├── models.py # Model registry (120+ models)
223251
│ ├── parser.py # TOOL_CALL parser
224252
│ ├── pricing.py # LLM pricing data and cost calculation
225253
│ ├── prompt.py # Prompt builder
226254
│ ├── tools.py # Tool definitions and registry
227255
│ ├── types.py # Core types (Message, Role, etc.)
228-
│ ├── usage.py # Usage tracking (tokens, costs)
229-
│ └── providers/ # LLM provider adapters
230-
│ ├── base.py # Provider interface
231-
│ ├── openai_provider.py # OpenAI implementation
232-
│ ├── anthropic_provider.py # Anthropic implementation
233-
│ ├── gemini_provider.py # Google Gemini implementation
234-
│ └── stubs.py # Local/test providers
235-
├── tests/ # Test suite
256+
│ ├── usage.py # Usage tracking (tokens, costs, analytics)
257+
│ ├── providers/ # LLM provider adapters
258+
│ │ ├── base.py # Provider interface
259+
│ │ ├── openai_provider.py # OpenAI implementation
260+
│ │ ├── anthropic_provider.py # Anthropic implementation
261+
│ │ ├── gemini_provider.py # Google Gemini implementation
262+
│ │ ├── ollama_provider.py # Ollama local models
263+
│ │ └── stubs.py # Local/test providers
264+
│ ├── embeddings/ # Embedding providers (v0.8.0)
265+
│ │ ├── provider.py # EmbeddingProvider interface
266+
│ │ ├── openai.py # OpenAI embeddings
267+
│ │ ├── anthropic.py # Anthropic/Voyage embeddings
268+
│ │ ├── gemini.py # Google Gemini embeddings
269+
│ │ └── cohere.py # Cohere embeddings
270+
│ ├── rag/ # RAG components (v0.8.0)
271+
│ │ ├── __init__.py # RAGAgent high-level API
272+
│ │ ├── vector_store.py # VectorStore interface
273+
│ │ ├── chunking.py # Text splitting strategies
274+
│ │ ├── loaders.py # Document loaders
275+
│ │ ├── tools.py # RAGTool & SemanticSearchTool
276+
│ │ └── stores/ # Vector store implementations
277+
│ │ ├── memory.py # In-memory (NumPy)
278+
│ │ ├── sqlite.py # SQLite persistent storage
279+
│ │ ├── chroma.py # ChromaDB integration
280+
│ │ └── pinecone.py # Pinecone cloud storage
281+
│ └── toolbox/ # Pre-built tools (v0.5.1)
282+
│ ├── file.py # File operations
283+
│ ├── web.py # Web scraping & HTTP
284+
│ ├── data.py # Data processing
285+
│ ├── datetime_tools.py # Date/time utilities
286+
│ └── text.py # Text manipulation
287+
├── tests/ # Test suite (463 tests)
236288
│ ├── conftest.py # Pytest configuration
237289
│ ├── test_framework.py # Core framework tests
238290
│ ├── test_better_errors.py # Error handling tests
239291
│ ├── test_cost_tracking.py # Cost tracking tests
240-
│ └── test_e2e_providers.py # End-to-end provider tests
292+
│ ├── test_e2e_providers.py # End-to-end provider tests
293+
│ ├── test_models.py # Model registry tests
294+
│ ├── test_embedding_providers.py # Embedding provider tests
295+
│ ├── test_vector_stores_crud.py # Vector store CRUD tests
296+
│ ├── test_document_loaders.py # Document loader tests
297+
│ ├── test_text_chunking.py # Chunking strategy tests
298+
│ ├── test_rag_workflow.py # RAG integration tests
299+
│ └── test_sqlite_integration.py # SQLite persistence tests
241300
├── examples/ # Usage examples
242301
│ ├── search_weather.py # Basic tool usage
243302
│ ├── async_agent_demo.py # Async operations
244303
│ ├── conversation_memory_demo.py
245304
│ ├── cost_tracking_demo.py
246305
│ ├── customer_support_bot.py
247-
│ └── data_analysis_agent.py
306+
│ ├── data_analysis_agent.py
307+
│ ├── rag_basic_demo.py # Basic RAG with in-memory store
308+
│ ├── rag_advanced_demo.py # Advanced RAG with PDF + SQLite
309+
│ └── semantic_search_demo.py # Multi-provider semantic search
248310
└── scripts/ # Development scripts
249311
├── README.md # Release documentation
250312
├── release.py # Python release script
@@ -325,21 +387,31 @@ We especially welcome contributions in these areas:
325387
- Add support for new LLM providers (Cohere, AI21, etc.)
326388
- Improve existing provider implementations
327389
- Add vision support to more providers
390+
- Add new embedding providers
328391

329392
### 🛠️ **New Tools**
330393
- Pre-built tools for common use cases
331394
- Integration with popular APIs and services
332395
- Example tools demonstrating best practices
333396

397+
### 🗄️ **RAG & Vector Stores (New in v0.8.0!)**
398+
- Add new vector store integrations (Weaviate, Qdrant, Milvus)
399+
- Implement reranking support (Cohere, Jina)
400+
- Add hybrid search (vector + BM25 keyword)
401+
- Improve chunking strategies (contextual, agentic)
402+
- Create advanced RAG examples and tutorials
403+
334404
### 📚 **Documentation**
335405
- Improve README examples
336-
- Add tutorials and guides
406+
- Add tutorials and guides (especially for RAG features!)
337407
- Fix typos and clarify confusing sections
408+
- Add comparison guides (vs LangChain, LlamaIndex)
338409

339410
### 🧪 **Testing**
340-
- Increase test coverage
411+
- Increase test coverage (currently 463 tests passing!)
341412
- Add integration tests
342413
- Add performance benchmarks
414+
- Test RAG workflows with different vector stores
343415

344416
### 🐛 **Bug Fixes**
345417
- Fix reported issues
@@ -448,6 +520,122 @@ def create_your_tool() -> Tool:
448520

449521
3. **Update documentation**
450522

523+
## Adding RAG Features (New in v0.8.0!)
524+
525+
### Adding a New Vector Store
526+
527+
To add support for a new vector database:
528+
529+
1. **Create a new vector store implementation**
530+
531+
```python
532+
# src/selectools/rag/stores/your_store.py
533+
534+
from typing import List, Optional
535+
from ..vector_store import VectorStore, Document, SearchResult
536+
537+
class YourVectorStore(VectorStore):
538+
"""Your vector database implementation."""
539+
540+
def __init__(self, embedder, **config):
541+
self.embedder = embedder
542+
# Initialize your vector DB client
543+
# self.client = your_db.Client(**config)
544+
545+
def add_documents(
546+
self,
547+
documents: List[Document],
548+
embeddings: Optional[List[List[float]]] = None
549+
) -> List[str]:
550+
"""Add documents to the store."""
551+
# Generate embeddings if not provided
552+
if embeddings is None:
553+
texts = [doc.text for doc in documents]
554+
embeddings = self.embedder.embed_texts(texts)
555+
556+
# Insert into your vector DB
557+
# ids = self.client.insert(...)
558+
return ids
559+
560+
def search(
561+
self,
562+
query_embedding: List[float],
563+
top_k: int = 5,
564+
filter: Optional[dict] = None
565+
) -> List[SearchResult]:
566+
"""Search for similar documents."""
567+
# results = self.client.search(query_embedding, top_k, filter)
568+
# return [SearchResult(document=..., score=...) for r in results]
569+
pass
570+
571+
def delete(self, ids: List[str]) -> None:
572+
"""Delete documents by ID."""
573+
# self.client.delete(ids)
574+
pass
575+
```
576+
577+
2. **Register in the factory**
578+
579+
```python
580+
# src/selectools/rag/vector_store.py
581+
582+
@staticmethod
583+
def create(store_type: str, embedder, **kwargs) -> "VectorStore":
584+
if store_type == "your_store":
585+
from .stores.your_store import YourVectorStore
586+
return YourVectorStore(embedder, **kwargs)
587+
# ... existing stores
588+
```
589+
590+
3. **Add tests**
591+
592+
```python
593+
# tests/test_vector_stores_crud.py
594+
595+
class TestYourVectorStore:
596+
def test_add_and_search(self, mock_embedder):
597+
store = VectorStore.create("your_store", embedder=mock_embedder)
598+
docs = [Document(text="test", metadata={})]
599+
ids = store.add_documents(docs)
600+
601+
query_emb = mock_embedder.embed_query("test")
602+
results = store.search(query_emb, top_k=1)
603+
604+
assert len(results) == 1
605+
assert results[0].document.text == "test"
606+
```
607+
608+
### Adding a New Embedding Provider
609+
610+
1. **Create the provider**
611+
612+
```python
613+
# src/selectools/embeddings/your_provider.py
614+
615+
from typing import List
616+
from .provider import EmbeddingProvider
617+
618+
class YourEmbeddingProvider(EmbeddingProvider):
619+
def __init__(self, model: str = "default-model", api_key: str = None):
620+
self.model = model
621+
self.api_key = api_key or os.getenv("YOUR_API_KEY")
622+
# self.client = your_sdk.Client(api_key=self.api_key)
623+
624+
def embed_texts(self, texts: List[str]) -> List[List[float]]:
625+
"""Embed multiple texts."""
626+
# response = self.client.embeddings.create(input=texts, model=self.model)
627+
# return [emb.embedding for emb in response.data]
628+
pass
629+
630+
def embed_query(self, query: str) -> List[float]:
631+
"""Embed a single query."""
632+
return self.embed_texts([query])[0]
633+
```
634+
635+
2. **Add model definitions to `models.py`**
636+
637+
3. **Add tests and update documentation**
638+
451639
## Code Review Process
452640

453641
1. A maintainer will review your PR

0 commit comments

Comments
 (0)