WhatsApp Migration - Backend Separation #192

OdyAsh · 2025-10-25T07:44:10Z

Summary

This PR implements Phases 1 & 2 of the WhatsApp migration plan, transforming ansari-backend to support the new ansari-whatsapp microservice architecture. The migration extracts WhatsApp webhook handling into a dedicated service while maintaining clean API endpoints in the backend for message processing and user management.

Key Changes:

Created 6 new WhatsApp API endpoints (/whatsapp/v2/*) for the microservice
Removed legacy WhatsApp webhook implementation (main_whatsapp.py, whatsapp_presenter.py)
Eliminated 1,108 lines of legacy code
Reorganized router structure for better maintainability
Improved privacy by removing location tracking endpoint
- (Note: we never collected this data in the first place, so we just removed dead code)

Ansari-Whatsapp's PR covering phase 3: ansari-project/ansari-whatsapp#1

TOC:

PR: WhatsApp Migration - Backend Separation
- Summary
- Migration Context
  - Architecture Overview
  - Migration Phases Status
- Detailed Changes
  - Commit 1: Phase 1 Implementation (6dc24fa)
  - Commit 2: Router Improvements (aff16aa)
  - Commit 3: Merge from develop (0f31e97)
  - Commit 4: Phase 2 Implementation (6bffe3c)
  - Commit 5: V2 Endpoint Naming & Reorganization (42ea141)
- Benefits
  - 1. Microservice Architecture
  - 1. Code Quality
  - 1. Maintainability
  - 1. Privacy & Security
  - 1. Development Experience
- Testing
  - Backend Tests
  - Microservice Tests
  - Integration Tests
- Migration Progress
  - Completed (This PR)
  - Already Completed (ansari-whatsapp repo)
  - Next Steps (Post-Merge)
- Breaking Changes
  - For Backend Deployments
  - For WhatsApp Integration
  - Migration Path
- Configuration Changes
  - Backend (.env)
  - Microservice (.env)
- Documentation References
- Questions?

Migration Context

This PR is part of a comprehensive 4-phase migration plan to separate WhatsApp functionality from ansari-backend into a dedicated microservice. For complete details, see the WhatsApp Migration Plan.

Architecture Overview

Before Migration:

ansari-backend (Monolithic)
├── WhatsApp webhooks (GET/POST /whatsapp/v1)
├── WhatsApp message processing
├── WhatsApp presenter logic
└── Core Ansari functionality

After Migration (Current State):

ansari-whatsapp (Microservice)          ansari-backend (API Service)
├── WhatsApp webhooks                   ├── WhatsApp API endpoints
├── WhatsApp presenter logic       ←────┤   • User registration
└── Background task processing          │   • Thread management
                                        │   • Message processing
                                        └── Core Ansari functionality

Migration Phases Status

✅ Phase 1 (This PR): Created missing backend API endpoints
✅ Phase 2 (This PR): Cleaned up ansari-backend legacy code
✅ Phase 3 (Complete): Implemented ansari-whatsapp microservice with full test coverage
🔄 Phase 4 (Next): AWS deployment configuration

Detailed Changes

Commit 1: Phase 1 Implementation (`6dc24fa`)

Created new WhatsApp API router with 6 endpoints

Added src/ansari/app/whatsapp_api_router.py with comprehensive API endpoints for ansari-whatsapp microservice:

POST /whatsapp/v2/users/register - Register new WhatsApp users
- Accepts: phone_num, preferred_language
- Returns: user_id on success
GET /whatsapp/v2/users/exists - Check if user exists
- Query param: phone_num
- Returns: {"exists": boolean}
POST /whatsapp/v2/threads - Create new message thread
- Accepts: phone_num, title
- Returns: thread_id
GET /whatsapp/v2/threads/last - Get last active thread
- Query param: phone_num
- Returns: thread_id, last_message_time
GET /whatsapp/v2/threads/{thread_id}/history - Get thread history
- Path param: thread_id
- Query param: phone_num
- Returns: Complete message history
POST /whatsapp/v2/messages/process - Process message with AI (streaming)
- Accepts: phone_num, thread_id, message
- Returns: Streamed AI response

Files changed:

Added src/ansari/app/whatsapp_api_router.py (306 lines)
Updated src/ansari/app/main_api.py to include WhatsApp router
Updated .gitignore

Commit 2: Router Improvements (`aff16aa`)

Fixed database calls and added health endpoint

Fixed database compatibility for both SQL and MongoDB backends
Added root health endpoint (GET /) for App Runner health checks
Updated CLAUDE.md with WhatsApp migration documentation

Files changed:

src/ansari/app/whatsapp_api_router.py (improved db compatibility)
src/ansari/app/main_api.py (added root endpoint)
CLAUDE.md (documentation)

Commit 3: Merge from develop (`0f31e97`)

Merged latest changes from develop branch

Standard merge commit to keep whatsapp-migration branch up-to-date with develop.

Commit 4: Phase 2 Implementation (`6bffe3c`)

Removed legacy WhatsApp implementation

Major cleanup removing 1,108 lines of legacy code:

Deleted src/ansari/app/main_whatsapp.py (205 lines)
- Removed old FastAPI webhooks for WhatsApp (GET/POST /whatsapp/v2)
- These webhooks now live in ansari-whatsapp microservice
Deleted src/ansari/presenters/whatsapp_presenter.py (862 lines)
- Removed WhatsApp message formatting logic
- Moved to ansari-whatsapp/presenters/whatsapp_message_formatter.py
Removed location tracking endpoint
- Deleted PUT /whatsapp/v2/users/location endpoint
- Removed WhatsAppLocationRequest model
- Privacy improvement: No longer storing user locations
Cleaned up configuration
- Removed unused WhatsApp environment variables from src/ansari/config.py:
  - META_WEBHOOK_VERIFY_TOKEN
  - META_ACCESS_TOKEN_FROM_SYS_USER
  - META_BUSINESS_PHONE_NUMBER_ID
  - META_API_VERSION

Files changed:

Deleted src/ansari/app/main_whatsapp.py (-205 lines)
Deleted src/ansari/presenters/whatsapp_presenter.py (-862 lines)
Updated src/ansari/app/whatsapp_api_router.py (-35 lines: location endpoint)
Updated src/ansari/config.py (-6 lines: removed WhatsApp config)

Commit 5: V2 Endpoint Naming & Reorganization (`42ea141`)

Updated API structure and naming conventions

Confirmed all endpoints use /whatsapp/v2/* naming convention
Reorganized router location for better structure:
- Moved from src/ansari/app/whatsapp_api_router.py
- To src/ansari/routers/whatsapp_router.py
Updated imports in src/ansari/app/main_api.py
Updated .env.example to reflect changes
Fixed documentation references

Files changed:

Renamed whatsapp_api_router.py → whatsapp_router.py (moved to routers/)
Updated src/ansari/app/main_api.py
Updated .env.example
Updated documentation files

Benefits

1. Microservice Architecture

Separation of Concerns: WhatsApp webhook handling isolated from core backend
Independent Scaling: Each service can scale based on its specific load
Deployment Flexibility: Deploy and update services independently

2. Code Quality

Removed 1,108 lines of legacy code
Eliminated duplication: Single source of truth for WhatsApp logic
Better organization: Clear API boundary between services

3. Maintainability

Cleaner codebase: Removed webhook handling complexity from backend
API-driven: Well-defined contract between services
Testability: Each service can be tested independently with mocks

4. Privacy & Security

Removed location tracking: No longer collecting or storing user locations
Reduced attack surface: Webhook validation isolated to microservice
Environment isolation: WhatsApp credentials only needed in microservice

5. Development Experience

Clearer boundaries: Developers know where to find WhatsApp logic
Faster iteration: Changes to WhatsApp flow don't require backend deployment
Better CI/CD: Independent test suites and deployment pipelines

Testing

Backend Tests

The WhatsApp API endpoints are covered by comprehensive tests:

Location: ansari-backend/tests/unit/test_whatsapp_api_endpoints.py
Coverage: All 6 endpoints including streaming functionality
Test approach: Uses FastAPI TestClient with mock database

Microservice Tests

The ansari-whatsapp service has full test coverage:

Location: ansari-whatsapp/tests/test_whatsapp_service.py
Coverage: Webhook endpoints, message processing, background tasks
Test approach: Mock external dependencies (Meta API, backend API)

Integration Tests

Both services tested together to verify:

User registration and thread creation flow
Message processing with streaming responses
Error handling and edge cases

Note: Tests use mock clients to avoid external dependencies, making CI/CD reliable and fast.

Migration Progress

Completed (This PR)

✅ Phase 1: Created 6 WhatsApp API endpoints in backend
✅ Phase 2: Removed legacy WhatsApp webhook implementation
✅ Cleaned up configuration and environment variables
✅ Reorganized router structure for maintainability
✅ Updated documentation and examples

Already Completed (ansari-whatsapp repo)

✅ Phase 3: Implemented complete microservice with tests
✅ Created comprehensive test suite with mocks
✅ Set up CI/CD workflows for testing
✅ Prepared AWS deployment documentation

Next Steps (Post-Merge)

🔄 Phase 4: Execute AWS deployment
- Create ECR repositories
- Set up SSM Parameter Store
- Configure GitHub Actions secrets
- Deploy to App Runner (staging & production)
- Update Meta webhook URL to production endpoint

Breaking Changes

For Backend Deployments

Removed endpoints: GET/POST /whatsapp/v2 webhooks (moved to ansari-whatsapp)
Removed endpoint: PUT /whatsapp/v2/users/location (privacy improvement)
Environment variables: Removed unused WhatsApp config from backend

For WhatsApp Integration

Webhook URL change: Meta webhook must point to ansari-whatsapp service URL
- Old: https://backend.ansari.chat/whatsapp/v2
- New: https://whatsapp.ansari.chat/whatsapp/v2 (after Phase 4 deployment)

Migration Path

Deploy this backend change first
Ensure ansari-whatsapp is deployed (Phase 4)
Update Meta webhook URL to point to ansari-whatsapp service
Verify with test message

Note: Both services should be deployed before updating Meta webhook URL to avoid downtime.

Configuration Changes

Backend (.env)

Removed variables (no longer needed):

# These are now only in ansari-whatsapp
META_WEBHOOK_VERIFY_TOKEN=...
META_ACCESS_TOKEN_FROM_SYS_USER=...
META_BUSINESS_PHONE_NUMBER_ID=...
META_API_VERSION=...

Microservice (.env)

New service requires (see ansari-whatsapp .env.example):

# Backend integration
BACKEND_SERVER_URL=http://localhost:8000
DEPLOYMENT_TYPE=local

# WhatsApp credentials
META_ACCESS_TOKEN_FROM_SYS_USER=your_token
META_BUSINESS_PHONE_NUMBER_ID=your_phone_id
META_WEBHOOK_VERIFY_TOKEN=your_verify_token
META_API_VERSION=v21.0

# Application settings
WHATSAPP_CHAT_RETENTION_HOURS=24
WHATSAPP_MESSAGE_AGE_THRESHOLD_SECONDS=60
ALWAYS_RETURN_OK_TO_META=True

Documentation References

Migration Plan: https://github.com/OdyAsh/ansari-whatsapp/blob/770be1d905e230c083bfb4c2624329e10f805db1/docs/whatsapp_migration_plan/migration_plan.md
AWS Deployment Guide: ansari-whatsapp/docs/aws/deployment_guide.md
GitHub Actions Setup: ansari-whatsapp/docs/aws/github_actions_setup.md
AWS CLI Commands: ansari-whatsapp/docs/aws/aws-cli.md

Questions?

For questions about:

Migration plan: See migration_plan.md
Deployment: See ansari-whatsapp/docs/aws/deployment_guide.md
Testing: See CLAUDE.md in both repositories

Check https://github.com/OdyAsh/ansari-whatsapp/blob/main/docs/whatsapp_migration_plan/migration_plan.md for details

…ckend into whatsapp-migration

…resenter.py - Deleted main_whatsapp.py which contained FastAPI endpoints for WhatsApp webhook handling. - Removed whatsapp_presenter.py that provided functions for WhatsApp interactions. - Updated whatsapp_api_router.py to remove location update endpoint and related model. - Cleaned up config.py by removing WhatsApp-related configuration fields.

…tructure

OdyAsh · 2025-10-25T08:13:17Z

src/ansari/app/main_api.py

        log_level="debug",
    )

+@app.get("/")


the use of adding this root router is for ansari-whatsapp to be able to check that the backend server is live when doing pytests. reference: https://github.com/OdyAsh/ansari-whatsapp/blob/58a4370fc3802b5ac6510efe6a6c5fa86d8062ba/tests/test_whatsapp_service.py#L62

… secure requests

OdyAsh added 5 commits September 28, 2025 11:36

Implem. Phase 1 of WA migration

6dc24fa

Check https://github.com/OdyAsh/ansari-whatsapp/blob/main/docs/whatsapp_migration_plan/migration_plan.md for details

Fix db calls in WA router, add root endpoint

aff16aa

Merge branch 'develop' of https://github.com/ansari-project/ansari-ba…

0f31e97

…ckend into whatsapp-migration

refactor: update WhatsApp API endpoints to v2 and reorganize router s…

42ea141

…tructure

OdyAsh mentioned this pull request Oct 25, 2025

Ansari-WhatsApp Microservice ansari-project/ansari-whatsapp#1

Merged

19 tasks

OdyAsh commented Oct 25, 2025

View reviewed changes

OdyAsh requested a review from amrmelsayed October 29, 2025 03:58

feat: add WhatsApp service API key configuration and verification for…

d81930c

… secure requests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WhatsApp Migration - Backend Separation #192

WhatsApp Migration - Backend Separation #192

Uh oh!

OdyAsh commented Oct 25, 2025 •

edited

Loading

Uh oh!

OdyAsh Oct 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

WhatsApp Migration - Backend Separation #192

Are you sure you want to change the base?

WhatsApp Migration - Backend Separation #192

Uh oh!

Conversation

OdyAsh commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Migration Context

Architecture Overview

Migration Phases Status

Detailed Changes

Commit 1: Phase 1 Implementation (6dc24fa)

Commit 2: Router Improvements (aff16aa)

Commit 3: Merge from develop (0f31e97)

Commit 4: Phase 2 Implementation (6bffe3c)

Commit 5: V2 Endpoint Naming & Reorganization (42ea141)

Benefits

1. Microservice Architecture

2. Code Quality

3. Maintainability

4. Privacy & Security

5. Development Experience

Testing

Backend Tests

Microservice Tests

Integration Tests

Migration Progress

Completed (This PR)

Already Completed (ansari-whatsapp repo)

Next Steps (Post-Merge)

Breaking Changes

For Backend Deployments

For WhatsApp Integration

Migration Path

Configuration Changes

Backend (.env)

Microservice (.env)

Documentation References

Questions?

Uh oh!

OdyAsh Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

OdyAsh commented Oct 25, 2025 •

edited

Loading

Commit 1: Phase 1 Implementation (`6dc24fa`)

Commit 2: Router Improvements (`aff16aa`)

Commit 3: Merge from develop (`0f31e97`)

Commit 4: Phase 2 Implementation (`6bffe3c`)

Commit 5: V2 Endpoint Naming & Reorganization (`42ea141`)