Ansari-WhatsApp Microservice

[OdyAsh]

Summary

This PR introduces the complete ansari-whatsapp microservice - a dedicated FastAPI service that handles WhatsApp Business API integration for the Ansari AI assistant. This represents the successful completion of Phase 3 of the WhatsApp migration plan, extracting WhatsApp functionality from the monolithic ansari-backend into an independent, scalable microservice.

What's Included:

• ✅ Complete WhatsApp webhook implementation (GET/POST /whatsapp/v2)
• ✅ WhatsApp message processing and formatting logic
• ✅ Background task processing for async operations
• ✅ Comprehensive test suite with mock clients
• ✅ CI/CD workflows for automated testing and AWS deployment
• ✅ Complete documentation (architecture, deployment, development)
• ✅ 2,500+ lines of production-ready code with proper error handling

Statistics:

• 25 Python files across source and tests
• 2,500+ lines of source code
• 300+ lines of test code
• 3 GitHub Actions workflows (testing, staging, production)
• Zero hardcoded secrets - all configuration via environment variables

Ansari-Backend's PR covering phases 1 & 2: ansari-project/ansari-backend#192

---

TOC:

• Summary
• Migration Context
  • Architecture Transformation
  • Migration Phases Status
• What's Included in This PR
  • 1. Core Application (src/ansari_whatsapp/app/)
  • 2. Business Logic (src/ansari_whatsapp/)
  • 3. Utilities (src/ansari_whatsapp/utils/)
  • 4. Comprehensive Test Suite (tests/)
  • 5. CI/CD Workflows (.github/workflows/)
  • 6. Documentation (docs/)
• Key Features & Capabilities
  • 1. WhatsApp Business API Integration
  • 2. Message Processing
  • 3. User & Thread Management
  • 4. Backend Integration
  • 5. Testing & Quality
  • 6. Deployment & Operations
• Technical Highlights
  • Clean Architecture
  • Modern Python
  • Error Handling
  • Performance
• Environment Configuration
  • Required Variables
  • Optional Variables (Testing)
• Testing & Quality Assurance
  • Test Coverage
  • Running Tests Locally
  • CI/CD Testing
• Development Workflow
  • Local Development
  • Testing with Meta Webhooks
  • Code Quality Tools
• Benefits of Microservice Architecture
  • 1. Scalability
  • 2. Maintainability
  • 3. Development Velocity
  • 4. Reliability
  • 5. Security
• Migration Impact
  • What Was Removed from ansari-backend
  • What Was Added to ansari-backend
  • Net Result
• Next Steps: Phase 4 Deployment
  • 4.1 AWS Resources Setup
  • 4.2 GitHub Configuration
  • 4.3 Code & Configuration
  • 4.4 Environment Variables Configuration
  • 4.5 Deployment Validation & Go-Live
• Breaking Changes & Migration Notes
  • For Existing Deployments
  • Meta Webhook URL Changes
  • Environment Variable Changes
  • No Data Migration Required
• Documentation
  • Quick Start
  • Architecture
  • Deployment
  • Migration
  • Testing
• Questions?
  • For Migration Questions:
  • For Deployment Questions:
  • For Development Questions:
  • For Architecture Questions:
• Related Pull Requests

---

Migration Context

This microservice is the result of a comprehensive 4-phase migration plan to separate WhatsApp functionality from ansari-backend. For complete migration details, see the WhatsApp Migration Plan.

Architecture Transformation

Before Migration (Monolithic):

ansari-backend
├── WhatsApp webhooks (GET/POST /whatsapp/v2)
├── WhatsApp message processing
├── WhatsApp presenter logic
├── Direct Meta API calls
└── Core Ansari functionality

After Migration (Microservice Architecture):

┌─────────────────────────────────────────────────────────┐
│                   Meta WhatsApp API                     │
│         (Sends webhook events for new messages)         │
└────────────────────┬────────────────────────────────────┘
                     │ Webhooks
                     ▼
┌─────────────────────────────────────────────────────────┐
│              ansari-whatsapp (Port 8001)                │
│  ┌────────────────────────────────────────────────┐    │
│  │ Webhook Endpoints (GET/POST /whatsapp/v2)      │    │
│  └─────────────────┬──────────────────────────────┘    │
│                    │                                     │
│  ┌─────────────────▼──────────────────────────────┐    │
│  │  WhatsApp Conversation Manager                 │    │
│  │  - Extract message details from webhook        │    │
│  │  - User registration & thread management       │    │
│  │  - Background task orchestration               │    │
│  └─────────────────┬──────────────────────────────┘    │
│                    │                                     │
│  ┌─────────────────▼──────────────────────────────┐    │
│  │  WhatsApp Message Formatter (Presenter)        │    │
│  │  - Format messages for WhatsApp (4K limit)     │    │
│  │  - Markdown to plain text conversion           │    │
│  │  - RTL language support                        │    │
│  │  - Typing indicators                           │    │
│  └─────────────────┬──────────────────────────────┘    │
│                    │                                     │
│  ┌─────────────────▼──────────────────────────────┐    │
│  │  Service Layer                                 │    │
│  │  ├─ AnsariClient (Backend API)                 │    │
│  │  ├─ MetaAPIService (WhatsApp API)              │    │
│  │  └─ ServiceProvider (Dependency Injection)     │    │
│  └────────────────────────────────────────────────┘    │
└──────────────┬──────────────────────────┬───────────────┘
               │ REST API                 │ Graph API
               ▼                          ▼
┌──────────────────────────┐   ┌──────────────────────────┐
│  ansari-backend          │   │  Meta WhatsApp API       │
│  (Port 8000)             │   │  (Send replies)          │
│                          │   │                          │
│  • User registration     │   │  • Send messages         │
│  • Thread management     │   │  • Mark as read          │
│  • Message processing    │   │  • Typing indicators     │
│  • AI streaming          │   │                          │
└──────────────────────────┘   └──────────────────────────┘

Migration Phases Status

• ✅ Phase 1 (Complete): Created 6 backend API endpoints for microservice communication
• ✅ Phase 2 (Complete): Removed 1,108 lines of legacy WhatsApp code from backend
• ✅ Phase 3 (This PR): Implemented complete ansari-whatsapp microservice with tests
• 🔄 Phase 4 (Next): AWS deployment to staging and production environments

---

What's Included in This PR

1. Core Application (src/ansari_whatsapp/app/)

main.py (261 lines)

• Health Check Endpoint: GET / - Returns service status for App Runner health checks
• Webhook Verification: GET /whatsapp/v2 - Validates Meta webhook subscription
• Webhook Processing: POST /whatsapp/v2 - Receives and processes incoming messages
• Background Tasks: Async message processing to prevent WhatsApp 5-second timeout
• CORS Middleware: Auto-configured based on deployment environment

Key Features:

• ✅ Message age validation (ignores old messages)
• ✅ Phone number validation (only processes configured business number)
• ✅ Error handling with proper logging
• ✅ Graceful degradation on failures

2. Business Logic (src/ansari_whatsapp/)

services/whatsapp_conversation_manager.py (335 lines)

• User registration and existence checks
• Thread creation and management
• Message processing orchestration
• Typing indicator management
• Chat retention handling (24-hour default)

presenters/whatsapp_message_formatter.py (280 lines)

• Message formatting for WhatsApp (4K character limit)
• Markdown to plain text conversion
• RTL language support (Arabic, Hebrew)
• Message splitting for long responses
• Typing indicator display

services/ansari_client_*.py (3 files, 510 lines total)

• Base: Abstract interface for backend communication
• Real: HTTP client using httpx for production
• Mock: Test client with simulated responses (no external dependencies)
• Supports all 6 backend API endpoints

services/meta_api_service_*.py (3 files, 420 lines total)

• Base: Abstract interface for Meta WhatsApp API
• Real: Graph API client for sending messages
• Mock: Test client for CI/CD without Meta credentials
• Message sending, typing indicators, read receipts

services/service_provider.py (85 lines)

• Dependency injection container
• Mock vs. real service selection
• Environment-based configuration

3. Utilities (src/ansari_whatsapp/utils/)

config.py (141 lines)

• Pydantic Settings for type-safe configuration
• Environment variable validation
• Deployment type handling (local/staging/production)
• CORS origin auto-configuration

whatsapp_webhook_parser.py (125 lines)

• Extract message details from webhook JSON
• Support for text messages, locations, and media
• Robust error handling for malformed webhooks

whatsapp_message_splitter.py (158 lines)

• Split long messages at 4K character boundaries
• Preserve message coherence and formatting
• Smart splitting at sentence boundaries

app_logger.py (120 lines)

• Loguru-based logging with Rich formatting
• Sensitive data masking (tokens, phone numbers)
• File rotation and retention policies
• Environment-aware log levels

time_utils.py (45 lines)

• Timezone handling for message timestamps
• Age validation for webhook messages

language_utils.py (38 lines)

• RTL language detection (Arabic, Hebrew)
• Text direction handling for proper WhatsApp display

general_helpers.py (51 lines)

• CORS middleware configuration
• Environment helper functions

exceptions.py (28 lines)

• Custom exception types for better error handling

4. Comprehensive Test Suite (tests/)

test_whatsapp_service.py (315 lines)

• WhatsApp webhook verification tests
• Message processing tests with mock clients
• Health check endpoint validation
• Phone number filtering tests
• Background task testing

test_utils.py (84 lines)

• Secure logging utilities
• Sensitive data masking validation
• Environment variable handling

Test Features:

• ✅ No external dependencies - Uses mock clients for all tests
• ✅ Fast execution - All tests use FastAPI TestClient
• ✅ CI/CD ready - Automated testing in GitHub Actions
• ✅ Comprehensive coverage - All critical paths tested
• ✅ Security focused - No hardcoded secrets, masked sensitive data

Test Configuration Options:

MOCK_ANSARI_CLIENT=True           # Use mock backend (no external server needed)
MOCK_META_API_SERVICE=True        # Use mock Meta API (no credentials needed)
ALWAYS_RETURN_OK_TO_META=True     # Return 200 for local dev, False for CI/CD
LOG_TEST_FILES_ONLY=False         # Filter logs to only show test file output

5. CI/CD Workflows (.github/workflows/)

perform-tests.yml

• Runs full test suite on every push/PR
• Uses mock clients (no external dependencies)
• Fast feedback loop for developers

deploy-staging.yml

• Automated deployment to AWS App Runner staging
• Triggered on push to develop branch
• Uses SSM Parameter Store for secrets

deploy-production.yml

• Production deployment with manual approval
• Triggered on push to main branch
• Stricter validation and testing

Workflow Features:

• ✅ Docker image building and ECR push
• ✅ SSM parameter injection for secrets
• ✅ App Runner service updates
• ✅ Health check validation
• ✅ Rollback on failure

6. Documentation (docs/)

High-Level Design:

• docs/hld/architecture.md - System architecture and component interactions

Low-Level Design:

• docs/lld/implementation_guide.md - Detailed implementation notes

AWS Deployment:

• docs/lld/aws/deployment_guide.md - Complete deployment walkthrough
• docs/lld/aws/aws-cli.md - All AWS CLI commands for setup
• docs/lld/aws/github_actions_setup.md - GitHub secrets configuration
• docs/lld/aws/README.md - AWS deployment overview

GitHub Actions:

• docs/lld/github_actions/concepts.md - CI/CD concepts and patterns
• docs/lld/github_actions/current_setup.md - Current workflow documentation
• docs/lld/github_actions/setup_commands.md - Setup instructions

Migration Plan:

• docs/whatsapp_migration_plan/migration_plan.md - Complete 4-phase migration plan

Development:

• CLAUDE.md - AI assistant instructions for code development
• docs/CLAUDE.md - Additional development notes
• README.md - Quick start guide and overview

---

Key Features & Capabilities

1. WhatsApp Business API Integration

• ✅ Webhook Handling: Secure verification and message processing
• ✅ Message Sending: Support for text messages with proper formatting
• ✅ Typing Indicators: Real-time feedback during AI processing
• ✅ Read Receipts: Mark messages as read automatically
• ✅ Media Detection: Identify images, videos, documents (with fallback messages)

2. Message Processing

• ✅ 4K Character Limit: Automatic message splitting at WhatsApp boundaries
• ✅ Markdown Conversion: Convert backend markdown to plain text for WhatsApp
• ✅ RTL Language Support: Proper handling of Arabic and Hebrew
• ✅ Smart Splitting: Split at sentence boundaries for readability
• ✅ Background Processing: Async tasks prevent webhook timeouts

3. User & Thread Management

• ✅ Automatic Registration: New users registered on first message
• ✅ Thread Creation: Automatic thread creation when needed
• ✅ Thread Reuse: Continue existing conversations within retention window
• ✅ Chat Retention: Configurable retention period (default 24 hours)
• ✅ User Preferences: Language preference tracking

4. Backend Integration

• ✅ 6 API Endpoints: Full integration with ansari-backend
  • User registration and existence checks
  • Thread creation and retrieval
  • Thread history access
  • Streaming message processing
• ✅ HTTP Client: Robust httpx-based client with error handling
• ✅ Mock Client: Test-friendly mock for CI/CD
• ✅ Streaming Support: Real-time AI response streaming

5. Testing & Quality

• ✅ Unit Tests: Comprehensive test coverage
• ✅ Integration Tests: End-to-end webhook processing tests
• ✅ Mock Clients: No external dependencies for tests
• ✅ CI/CD Integration: Automated testing on every push
• ✅ Fast Execution: All tests run in <10 seconds

6. Deployment & Operations

• ✅ Docker Support: Multi-stage Dockerfile for production
• ✅ AWS App Runner: Fully configured for App Runner deployment
• ✅ GitHub Actions: Automated staging and production deployments
• ✅ SSM Parameter Store: Secure secret management
• ✅ Health Checks: App Runner-compatible health endpoints
• ✅ Zero-Downtime Deploys: Rolling updates with health checks

---

Technical Highlights

Clean Architecture

• Service Layer: Clear separation between services (Ansari backend, Meta API)
• Presenter Pattern: Business logic separated from API concerns
• Dependency Injection: Flexible service provider for testing
• Abstract Interfaces: Mock-friendly design for all external dependencies

Modern Python

• Type Hints: Full type annotations throughout codebase
• Pydantic Settings: Type-safe configuration management
• Async/Await: Proper async patterns for I/O operations
• Context Managers: Resource management with proper cleanup

Error Handling

• Custom Exceptions: Domain-specific exception types
• Graceful Degradation: Service continues on non-critical failures
• Error Logging: Comprehensive error context in logs
• User-Friendly Messages: Helpful error responses to WhatsApp users

Performance

• Background Tasks: Async processing prevents webhook timeouts
• HTTP Connection Pooling: Efficient httpx client configuration
• Message Batching: Smart splitting for long messages
• Lazy Loading: Services initialized only when needed

---

Environment Configuration

The service requires these environment variables (see .env.example for full list):

Required Variables

# Deployment configuration
DEPLOYMENT_TYPE=local                          # local/staging/production

# Backend integration
BACKEND_SERVER_URL=http://localhost:8000       # Ansari backend URL

# WhatsApp API credentials
META_BUSINESS_PHONE_NUMBER_ID=123456789012345  # Your business phone ID
META_ACCESS_TOKEN_FROM_SYS_USER=EAAxxxxx       # System user access token
META_WEBHOOK_VERIFY_TOKEN=your_verify_token    # Webhook verification token
META_API_VERSION=v22.0                         # Graph API version

# Application settings
WHATSAPP_CHAT_RETENTION_HOURS=3                # Thread retention window
WHATSAPP_MESSAGE_AGE_THRESHOLD_SECONDS=86400   # Ignore messages older than 24 hours

Optional Variables (Testing)

# Test configuration
MOCK_ANSARI_CLIENT=False                       # Use mock backend
MOCK_META_API_SERVICE=False                    # Use mock Meta API
ALWAYS_RETURN_OK_TO_META=True                  # Always return 200 (local dev)
LOG_TEST_FILES_ONLY=False                      # Filter test logs

# Test data
WHATSAPP_DEV_PHONE_NUM=+1234567890            # Test phone number
WHATSAPP_DEV_MESSAGE_ID=wamid.xxx             # Test message ID

---

Testing & Quality Assurance

Test Coverage

• ✅ Webhook Verification: GET endpoint with challenge verification
• ✅ Webhook Processing: POST endpoint with message extraction
• ✅ User Registration: New user creation flow
• ✅ Thread Management: Thread creation and retrieval
• ✅ Message Processing: End-to-end message handling
• ✅ Phone Filtering: Reject messages from wrong phone numbers
• ✅ Error Handling: Graceful failure scenarios
• ✅ Health Checks: Service availability validation

Running Tests Locally

# Install dependencies
uv sync

# Run all tests with verbose output
pytest tests/ -v -s

# Run with test-only logging (cleaner output)
LOG_TEST_FILES_ONLY=True pytest tests/ -v -s

# Run specific test file
pytest tests/test_whatsapp_service.py -v

# Run with CI/CD settings (proper HTTP status codes)
ALWAYS_RETURN_OK_TO_META=False pytest tests/ -v

CI/CD Testing

Tests run automatically on every push/PR via GitHub Actions:

• ✅ Environment secrets loaded from GitHub Secrets
• ✅ Mock clients enabled (no external dependencies)
• ✅ Fast execution (<10 seconds for full suite)
• ✅ Clear error messages for failures

---

Development Workflow

Local Development

1. Clone repository:

   git clone <repository-url>
   cd ansari-whatsapp

2. Install uv and project dependencies:

   # Install uv (if not already installed)
   # On macOS/Linux:
   curl -LsSf https://astral.sh/uv/install.sh | sh
   # On Windows:
   powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
   
   # Install project dependencies from pyproject.toml
   uv sync

3. Configure environment:

   • Copy .env.example to .env
   • Fill in required credentials

4. Start the service:

   .venv/Scripts/python.exe src/ansari_whatsapp/app/main.py  # Windows
   .venv/bin/python src/ansari_whatsapp/app/main.py         # Linux/Mac

5. Expose webhook (for Meta testing):

   zrok reserve public localhost:8001 -n <ZROK_SHARE_TOKEN>

Testing with Meta Webhooks

1. Start local service on port 8001
2. Create zrok tunnel to expose webhook
3. Configure Meta webhook URL to zrok URL + /whatsapp/v2
4. Send test message to your WhatsApp Business number
5. Check logs to verify processing

Code Quality Tools

# Linting
ruff check .

# Auto-fix linting issues
ruff check --fix .

# Formatting
ruff format .

---

Benefits of Microservice Architecture

1. Scalability

• Independent Scaling: WhatsApp service scales separately from backend
• Resource Optimization: Allocate resources based on WhatsApp traffic
• Performance Isolation: WhatsApp issues don't affect backend

2. Maintainability

• Clear Boundaries: WhatsApp logic isolated from core business logic
• Easier Updates: Deploy WhatsApp changes without backend deployment
• Reduced Complexity: Each service has single responsibility

3. Development Velocity

• Parallel Development: Teams can work on services independently
• Faster Iteration: Smaller codebase means quicker changes
• Better Testing: Isolated testing with mock clients

4. Reliability

• Fault Isolation: Failures in WhatsApp service don't crash backend
• Graceful Degradation: Service continues on non-critical failures
• Independent Monitoring: Separate health checks and logging

5. Security

• Reduced Attack Surface: Webhook validation isolated from backend
• Credential Isolation: WhatsApp credentials only in this service
• Environment Separation: Different deployment configurations

---

Migration Impact

What Was Removed from ansari-backend

• ✅ main_whatsapp.py (205 lines) - Old webhook implementation
• ✅ whatsapp_presenter.py (862 lines) - Message formatting logic
• ✅ Location tracking endpoint (privacy improvement)
• ✅ WhatsApp environment variables from backend config

What Was Added to ansari-backend

• ✅ 6 new API endpoints in whatsapp_router.py (271 lines)
• ✅ Clean API boundary for microservice communication

Net Result

• Backend: -1,108 lines removed, +271 lines added = -837 lines net
• WhatsApp Service: +2,618 lines of dedicated, focused code
• Total: +1,781 lines across both services (with better organization)

---

Next Steps: Phase 4 Deployment

This PR completes Phase 3 (implementation). Next is Phase 4 (AWS deployment):

4.1 AWS Resources Setup

• Create ECR repository: ansari-whatsapp
• Create SSM parameters for staging environment
• Create SSM parameters for production environment
• Verify IAM roles (reusing existing App Runner roles)

4.2 GitHub Configuration

• Add AWS credentials to GitHub Secrets
• Configure ECR repository URL in secrets
• Add WhatsApp credentials to secrets
• Test workflow execution

4.3 Code & Configuration

• Update .github/workflows/deploy-staging.yml with correct values
• Update .github/workflows/deploy-production.yml with correct values
• Verify Dockerfile builds successfully
• Test locally with Docker

4.4 Environment Variables Configuration

Staging SSM Parameters:

/app-runtime/ansari-whatsapp/staging/DEPLOYMENT_TYPE=staging
/app-runtime/ansari-whatsapp/staging/BACKEND_SERVER_URL=<staging-backend-url>
/app-runtime/ansari-whatsapp/staging/META_BUSINESS_PHONE_NUMBER_ID=<phone-id>
/app-runtime/ansari-whatsapp/staging/META_ACCESS_TOKEN_FROM_SYS_USER=<token>
... (see deployment guide for full list)

Production SSM Parameters:

/app-runtime/ansari-whatsapp/production/DEPLOYMENT_TYPE=production
/app-runtime/ansari-whatsapp/production/BACKEND_SERVER_URL=<production-backend-url>
... (same variables as staging with production values)

4.5 Deployment Validation & Go-Live

• Deploy to staging environment
• Test webhook with staging Meta webhook
• Validate health checks and monitoring
• Deploy to production environment
• Update Meta webhook URL to production
• Monitor production logs
• Verify end-to-end message flow

Detailed instructions: See docs/lld/aws/deployment_guide.md

---

Breaking Changes & Migration Notes

For Existing Deployments

If you have an existing WhatsApp integration in ansari-backend:

1. Deploy ansari-backend changes first (new API endpoints, remove old webhooks)
2. Deploy ansari-whatsapp microservice (this PR)
3. Update Meta webhook URL from backend to whatsapp service
4. Verify with test message

Meta Webhook URL Changes

• Old:
  • Staging: https://staging-api.ansari.chat/whatsapp/v1
  • Production: https://api.ansari.chat/whatsapp/v1
• New:
  • Staging: https://staging-whatsapp.ansari.chat/whatsapp/v2
  • Production: https://whatsapp.ansari.chat/whatsapp/v2

Environment Variable Changes

WhatsApp credentials move from backend to this service:

• Remove from ansari-backend: META_* variables
• Add to ansari-whatsapp: META_* variables + BACKEND_SERVER_URL

No Data Migration Required

• User data stays in backend database
• Thread history preserved
• No changes to message storage or retrieval

---

Documentation

Quick Start

• README.md - Setup instructions and overview
• CLAUDE.md - Development guidelines for AI assistants

Architecture

• docs/hld/architecture.md - High-level system design
• docs/lld/implementation_guide.md - Implementation details

Deployment

• docs/lld/aws/deployment_guide.md - Complete AWS deployment guide
• docs/lld/aws/aws-cli.md - AWS CLI commands reference
• docs/lld/aws/github_actions_setup.md - CI/CD configuration

Migration

• docs/whatsapp_migration_plan/migration_plan.md - Full 4-phase plan
• Tracks progress from monolith to microservices

Testing

• tests/README.md - Test suite documentation
• Includes mock client setup and CI/CD instructions

---

Questions?

For Migration Questions:

• See migration_plan.md

For Deployment Questions:

• See docs/lld/aws/deployment_guide.md
• See docs/lld/aws/github_actions_setup.md

For Development Questions:

• See CLAUDE.md in this repository
• See tests/README.md for testing setup

For Architecture Questions:

• See docs/hld/architecture.md
• See docs/lld/implementation_guide.md

---

---

Ready for Review: This PR represents 2+ months of development work to extract WhatsApp functionality into a production-ready microservice. All code is tested, documented, and ready for AWS deployment (Phase 4).