Skip to content

TEST_STRATEGY.md

Latest commit

 

History

History
243 lines (184 loc) · 6.58 KB

TEST_STRATEGY.md

File metadata and controls

243 lines (184 loc) · 6.58 KB

Unit Testing Strategy for TrustGraph Microservices

Overview

This document outlines the unit testing strategy for the TrustGraph microservices architecture. The approach focuses on testing business logic while mocking external infrastructure to ensure fast, reliable, and maintainable tests.

1. Test Framework: pytest + pytest-asyncio

  • pytest: Standard Python testing framework with excellent fixture support
  • pytest-asyncio: Essential for testing async processors
  • pytest-mock: Built-in mocking capabilities

2. Core Testing Patterns

Service Layer Testing

@pytest.mark.asyncio
async def test_text_completion_service():
    # Test the core business logic, not external APIs
    processor = TextCompletionProcessor(model="test-model")
    
    # Mock external dependencies
    with patch('processor.llm_client') as mock_client:
        mock_client.generate.return_value = "test response"
        
        result = await processor.process_message(test_message)
        assert result.content == "test response"

Message Processing Testing

@pytest.fixture
def mock_pulsar_consumer():
    return AsyncMock(spec=pulsar.Consumer)

@pytest.fixture  
def mock_pulsar_producer():
    return AsyncMock(spec=pulsar.Producer)

async def test_message_flow(mock_consumer, mock_producer):
    # Test message handling without actual Pulsar
    processor = FlowProcessor(consumer=mock_consumer, producer=mock_producer)
    # Test message processing logic

3. Mock Strategy

Mock External Services (Not Infrastructure)

  • ✅ Mock: LLM APIs, Vector DBs, Graph DBs
  • ❌ Don't Mock: Core business logic, data transformations
  • ✅ Mock: Pulsar clients (infrastructure)
  • ❌ Don't Mock: Message validation, processing logic

Dependency Injection Pattern

class TextCompletionProcessor:
    def __init__(self, llm_client=None, **kwargs):
        self.llm_client = llm_client or create_default_client()
        
# In tests
processor = TextCompletionProcessor(llm_client=mock_client)

4. Test Categories

Unit Tests (70%)

  • Individual service business logic
  • Message processing functions
  • Data transformation logic
  • Configuration parsing
  • Error handling

Integration Tests (20%)

  • Service-to-service communication patterns
  • Database operations with test containers
  • End-to-end message flows

Contract Tests (10%)

  • Pulsar message schemas
  • API response formats
  • Service interface contracts

5. Test Structure

tests/
├── unit/
│   ├── test_text_completion/
│   ├── test_embeddings/
│   ├── test_storage/
│   └── test_utils/
├── integration/
│   ├── test_flows/
│   └── test_databases/
├── fixtures/
│   ├── messages.py
│   ├── configs.py
│   └── mocks.py
└── conftest.py

6. Key Testing Tools

  • testcontainers: For database integration tests
  • responses: Mock HTTP APIs
  • freezegun: Time-based testing
  • factory-boy: Test data generation

7. Service-Specific Testing Approaches

Text Completion Services

  • Mock LLM provider APIs (OpenAI, Claude, Ollama)
  • Test prompt construction and response parsing
  • Verify rate limiting and error handling
  • Test token counting and metrics collection

Embeddings Services

  • Mock embedding providers (FastEmbed, Ollama)
  • Test vector dimension consistency
  • Verify batch processing logic
  • Test embedding storage operations

Storage Services

  • Use testcontainers for database integration tests
  • Mock database clients for unit tests
  • Test query construction and result parsing
  • Verify data persistence and retrieval logic

Query Services

  • Mock vector similarity search operations
  • Test graph traversal logic
  • Verify result ranking and filtering
  • Test query optimization

8. Best Practices

Test Isolation

  • Each test should be independent
  • Use fixtures for common setup
  • Clean up resources after tests
  • Avoid test order dependencies

Async Testing

  • Use @pytest.mark.asyncio for async tests
  • Mock async dependencies properly
  • Test concurrent operations
  • Handle timeout scenarios

Error Handling

  • Test both success and failure scenarios
  • Verify proper exception handling
  • Test retry mechanisms
  • Validate error response formats

Configuration Testing

  • Test different configuration scenarios
  • Verify parameter validation
  • Test environment variable handling
  • Test configuration defaults

9. Example Test Implementation

# tests/unit/test_text_completion/test_openai_processor.py
import pytest
from unittest.mock import AsyncMock, patch
from trustgraph.model.text_completion.openai import Processor

@pytest.fixture
def mock_openai_client():
    return AsyncMock()

@pytest.fixture
def processor(mock_openai_client):
    return Processor(client=mock_openai_client, model="gpt-4")

@pytest.mark.asyncio
async def test_process_message_success(processor, mock_openai_client):
    # Arrange
    mock_openai_client.chat.completions.create.return_value = AsyncMock(
        choices=[AsyncMock(message=AsyncMock(content="Test response"))]
    )
    
    message = {
        "id": "test-id",
        "prompt": "Test prompt",
        "temperature": 0.7
    }
    
    # Act
    result = await processor.process_message(message)
    
    # Assert
    assert result.content == "Test response"
    mock_openai_client.chat.completions.create.assert_called_once()

@pytest.mark.asyncio
async def test_process_message_rate_limit(processor, mock_openai_client):
    # Arrange
    mock_openai_client.chat.completions.create.side_effect = RateLimitError("Rate limited")
    
    message = {"id": "test-id", "prompt": "Test prompt"}
    
    # Act & Assert
    with pytest.raises(RateLimitError):
        await processor.process_message(message)

10. Running Tests

# Run all tests
pytest

# Run unit tests only
pytest tests/unit/

# Run with coverage
pytest --cov=trustgraph --cov-report=html

# Run async tests
pytest -v tests/unit/test_text_completion/

# Run specific test file
pytest tests/unit/test_text_completion/test_openai_processor.py

11. Continuous Integration

  • Run tests on every commit
  • Enforce minimum code coverage (80%+)
  • Run tests against multiple Python versions
  • Include integration tests in CI pipeline
  • Generate test reports and coverage metrics

Conclusion

This testing strategy ensures that TrustGraph microservices are thoroughly tested without relying on external infrastructure. By focusing on business logic and mocking external dependencies, we achieve fast, reliable tests that provide confidence in code quality while maintaining development velocity.