Skip to content

cloudflare-setup.md

Latest commit

 

History

History
602 lines (457 loc) · 17.7 KB

cloudflare-setup.md

File metadata and controls

602 lines (457 loc) · 17.7 KB
Raw

Cloudflare Backend Setup Guide

Overview

The MCP Memory Service supports native Cloudflare integration using Vectorize for vector storage, D1 for metadata, and optional R2 for large content. This provides:

  • Vectorize: Vector database for semantic search (768-dimensional embeddings)
  • D1: SQLite database for metadata storage
  • Workers AI: Embedding generation (@cf/baai/bge-base-en-v1.5)
  • R2 (optional): Object storage for large content

This setup provides global distribution, automatic scaling, and cost-effective pay-per-use pricing.

🚀 Quick Start

For users who want to get started immediately:

Prerequisites

  1. Cloudflare Account: You need a Cloudflare account with Workers/D1/Vectorize access
  2. API Token: Create an API token with these permissions:
    • Vectorize Edit (for creating and managing vector indexes)
    • D1 Edit (for creating and managing databases)
    • R2 Edit (optional, for large content storage)
    • Workers AI Read (for embedding generation)

Quick Setup Commands

# 1. Install dependencies
pip install httpx>=0.24.0

# 2. Create Cloudflare resources (requires wrangler CLI)
wrangler vectorize create mcp-memory-index --dimensions=768 --metric=cosine
wrangler d1 create mcp-memory-db
wrangler r2 bucket create mcp-memory-content  # Optional

# 3. Configure environment
export MCP_MEMORY_STORAGE_BACKEND=cloudflare
export CLOUDFLARE_API_TOKEN="your-api-token"
export CLOUDFLARE_ACCOUNT_ID="your-account-id"
export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index"
export CLOUDFLARE_D1_DATABASE_ID="your-d1-database-id"
export CLOUDFLARE_R2_BUCKET="mcp-memory-content"  # Optional

# 4. Test and start
memory launch                   # HTTP server (background, recommended)

# Alternative startup methods:
# memory server                 # MCP stdio (for Claude Desktop)
# python -m mcp_memory_service.server  # MCP stdio (module form)

⚠️ Important: Cloudflare backend uses Workers AI for embedding generation, so do NOT use scripts/memory_offline.py which sets offline mode. Use the standard startup methods above instead.

Prerequisites

  1. Cloudflare Account: Sign up at cloudflare.com
  2. Cloudflare Services: Access to Vectorize, D1, and optionally R2
  3. API Token: With appropriate permissions

Step 1: Create Cloudflare Resources

1.1 Create Vectorize Index

# Install Wrangler CLI
npm install -g wrangler

# Login to Cloudflare
wrangler login

# Create Vectorize index (768 dimensions for BGE embeddings)
wrangler vectorize create mcp-memory-index --dimensions=768 --metric=cosine

1.2 Create D1 Database

# Create D1 database
wrangler d1 create mcp-memory-db

# Note the database ID from the output

1.3 Create R2 Bucket (Optional)

# Create R2 bucket for large content storage
wrangler r2 bucket create mcp-memory-content

Step 2: Configure API Token

2.1 Create API Token

  1. Go to Cloudflare Dashboard → My Profile → API Tokens
  2. Click "Create Token"
  3. Use "Custom Token" template
  4. Configure permissions:
    • Account: Read (to access account resources)
    • Vectorize: Edit (to manage vector operations)
    • D1: Edit (to manage database operations)
    • R2: Edit (if using R2 for large content)
    • Workers AI: Read (for embedding generation)

2.2 Get Account ID

  1. Go to Cloudflare Dashboard
  2. Select your domain or go to overview
  3. Copy the Account ID from the right sidebar

2.3 Manual Resource Creation (Alternative)

If you prefer manual creation via the Cloudflare Dashboard or encounter authentication issues:

Create Vectorize Index via Dashboard:

  1. Go to Cloudflare Dashboard → Vectorize
  2. Click "Create Index"
  3. Name: mcp-memory-index
  4. Dimensions: 768
  5. Metric: cosine

Create D1 Database via Dashboard:

  1. Go to Cloudflare Dashboard → D1
  2. Click "Create Database"
  3. Name: mcp-memory-db
  4. Copy the Database ID from the overview page

Create R2 Bucket via Dashboard (Optional):

  1. Go to Cloudflare Dashboard → R2
  2. Click "Create Bucket"
  3. Name: mcp-memory-content
  4. Choose region closest to your location

Alternative API Creation:

# Create Vectorize index via API
curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/vectorize/indexes" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mcp-memory-index",
    "config": {
      "dimensions": 768,
      "metric": "cosine"
    }
  }'

# Create D1 database via API
curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/d1/database" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mcp-memory-db"
  }'

# Create R2 bucket via API (optional)
curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/r2/buckets" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mcp-memory-content"
  }'

Step 3: Configure Environment Variables

Set the following environment variables:

# Required Configuration
export MCP_MEMORY_STORAGE_BACKEND=cloudflare
export CLOUDFLARE_API_TOKEN="your-api-token-here"
export CLOUDFLARE_ACCOUNT_ID="your-account-id-here"
export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index"
export CLOUDFLARE_D1_DATABASE_ID="your-d1-database-id"

# Optional Configuration
export CLOUDFLARE_R2_BUCKET="mcp-memory-content"  # For large content
export CLOUDFLARE_EMBEDDING_MODEL="@cf/baai/bge-base-en-v1.5"  # Default
export CLOUDFLARE_LARGE_CONTENT_THRESHOLD="1048576"  # 1MB threshold
export CLOUDFLARE_MAX_RETRIES="3"  # API retry attempts
export CLOUDFLARE_BASE_DELAY="1.0"  # Retry delay in seconds

Configuration File Example

Create a .env file in your project root:

# Cloudflare Backend Configuration
MCP_MEMORY_STORAGE_BACKEND=cloudflare

# Required Cloudflare Settings
CLOUDFLARE_API_TOKEN=your-api-token-here
CLOUDFLARE_ACCOUNT_ID=your-account-id-here
CLOUDFLARE_VECTORIZE_INDEX=mcp-memory-index
CLOUDFLARE_D1_DATABASE_ID=your-d1-database-id

# Optional Settings
CLOUDFLARE_R2_BUCKET=mcp-memory-content
CLOUDFLARE_EMBEDDING_MODEL=@cf/baai/bge-base-en-v1.5
CLOUDFLARE_LARGE_CONTENT_THRESHOLD=1048576
CLOUDFLARE_MAX_RETRIES=3
CLOUDFLARE_BASE_DELAY=1.0

# Logging
LOG_LEVEL=INFO

Step 4: Install Dependencies

The Cloudflare backend requires additional dependencies:

# Install additional requirements
pip install -r requirements-cloudflare.txt

# Or install manually
pip install httpx>=0.24.0

Step 5: Initialize and Test

5.1 Start the Service

# Start MCP Memory Service with Cloudflare backend
python -m src.mcp_memory_service.server

5.2 Verify Configuration

The service will automatically:

  1. Initialize the D1 database schema
  2. Verify access to the Vectorize index
  3. Check R2 bucket access (if configured)

Look for these success messages in the logs:

INFO:mcp_memory_service.config:Using Cloudflare backend with:
INFO:mcp_memory_service.config:  Vectorize Index: mcp-memory-index
INFO:mcp_memory_service.config:  D1 Database: your-d1-database-id
INFO:mcp_memory_service.server:Created Cloudflare storage with Vectorize index: mcp-memory-index
INFO:mcp_memory_service.storage.cloudflare:Cloudflare storage backend initialized successfully

5.3 Test Basic Operations

Option A: Comprehensive Test Suite

# Run comprehensive automated tests
python scripts/test_cloudflare_backend.py

Option B: Manual API Testing

# Store a test memory
curl -X POST http://localhost:8000/api/memories \
  -H "Content-Type: application/json" \
  -d '{
    "content": "This is a test memory for Cloudflare backend",
    "tags": ["test", "cloudflare"]
  }'

# Search memories
curl -X POST http://localhost:8000/api/memories/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "test memory",
    "n_results": 5
  }'

# Get statistics
curl http://localhost:8000/api/stats

Option C: Automated Resource Setup

# Set up Cloudflare resources automatically
python scripts/setup_cloudflare_resources.py

Architecture Details

Data Flow

  1. Content Storage:

    • Small content (<1MB): Stored directly in D1
    • Large content (>1MB): Stored in R2, referenced in D1

  2. Vector Processing:

    • Content → Workers AI → Embedding Vector
    • Vector stored in Vectorize with metadata
    • Semantic search via Vectorize similarity

  3. Metadata Management:

    • Memory metadata stored in D1 SQLite
    • Tags stored in relational tables
    • Full ACID compliance for data integrity

Performance Optimizations

  • Connection Pooling: Reused HTTP connections
  • Embedding Caching: 1000-entry LRU cache
  • Batch Operations: Bulk vector operations
  • Smart Retries: Exponential backoff for rate limits
  • Async Operations: Non-blocking I/O throughout

Security Features

  • API Key Security: Never logged or exposed
  • Input Validation: SQL injection prevention
  • Rate Limiting: Built-in protection
  • Secure Headers: Proper HTTP security

Migration from Other Backends

From SQLite-vec

# Export existing data
python scripts/export_sqlite_vec.py --output cloudflare_export.json

# Switch to Cloudflare backend
export MCP_MEMORY_STORAGE_BACKEND=cloudflare

# Import data
python scripts/import_to_cloudflare.py --input cloudflare_export.json

From Legacy ChromaDB Data

ChromaDB was removed in v8.0.0. If you still have ChromaDB data, export it from the chromadb-legacy branch first (see guides/chromadb-migration.md) and then import the resulting JSON into Cloudflare:

# On the chromadb-legacy branch — produce a backup JSON
git checkout chromadb-legacy
python scripts/migration/migrate_chroma_to_sqlite.py --backup ~/chromadb_backup.json

# Back on main — switch to Cloudflare and import
git checkout main
export MCP_MEMORY_STORAGE_BACKEND=cloudflare
python scripts/import_to_cloudflare.py --input ~/chromadb_backup.json

Troubleshooting

Common Issues

1. Authentication Errors (401)

ERROR: Missing required environment variables for Cloudflare backend: CLOUDFLARE_API_TOKEN
ERROR: Unauthorized - Invalid API token

Solution:

  • Verify all required environment variables are set
  • Check API token has correct permissions (Vectorize:Edit, D1:Edit, Workers AI:Read)
  • Ensure token is not expired
  • Verify account ID is correct

2. Resource Not Found (404)

ValueError: Vectorize index 'mcp-memory-index' not found
ValueError: D1 database not found

Solution:

  • Create the Vectorize index or verify the index name is correct
  • Check that resources were created in the correct account
  • Confirm resource IDs/names match exactly
  • Verify resource names match exactly

3. Vector Storage Errors (400)

ValueError: Failed to store vector data
HTTP 400: Invalid vector data format

Solution:

  • Check vector dimensions (must be 768)
  • Verify NDJSON format for vector data
  • Ensure metadata values are properly serialized
  • Validate input data types

4. D1 Database Access Issues

ValueError: Failed to initialize D1 schema
HTTP 403: Insufficient permissions

Solution:

  • Verify D1 database ID and API token permissions
  • Ensure database exists and is accessible
  • Check API token has D1:Edit permissions

5. API Rate Limits (429)

Rate limited after 3 retries
HTTP 429: Too Many Requests

Solution:

  • Increase CLOUDFLARE_MAX_RETRIES or CLOUDFLARE_BASE_DELAY for more conservative retry behavior
  • Implement exponential backoff (already included)
  • Monitor API usage through Cloudflare dashboard
  • Consider implementing request caching for high-volume usage

Debug Mode

Enable detailed logging:

export LOG_LEVEL=DEBUG
python -m src.mcp_memory_service.server --debug

Health Check

# Check backend health
curl http://localhost:8000/api/health

# Get detailed statistics
curl http://localhost:8000/api/stats

Limitations

Current Limitations

  • Embedding Model: Fixed to Workers AI BGE model (768 dimensions)
  • Content Size: R2 storage recommended for content >1MB
  • Rate Limits: Subject to Cloudflare service limits
  • Region: Embedding generation uses Cloudflare's global network

Planned Improvements

  • Local Embedding Fallback: For offline or restricted environments
  • Custom Embedding Models: Support for other embedding models
  • Enhanced Caching: Multi-level caching strategy
  • Batch Import Tools: Efficient migration utilities

🔄 Multi-Machine Bidirectional Sync

New in v6.13.7: Cloudflare backend now supports seamless bidirectional sync between multiple machines, making it ideal for distributed teams or as a replacement for failed centralized servers.

Use Cases

  1. Failed Server Recovery: Replace a failed narrowbox/central server with Cloudflare
  2. Multi-Machine Development: Sync memories across multiple development machines
  3. Team Collaboration: Share memory context across team members
  4. Backup Strategy: Cloudflare as primary with local sqlite_vec as backup

Architecture

┌─────────────────┐    ┌─────────────────┐
│   Machine A     │    │   Machine B     │
│                 │    │                 │
│ Claude Desktop  │    │ Claude Desktop  │
│      ↕          │    │       ↕         │
│ sqlite_vec      │    │ sqlite_vec      │
│   (backup)      │    │   (backup)      │
└─────────┬───────┘    └─────────┬───────┘
          │                      │
          └─────────┬────────────┘
                    ↕
          ┌─────────────────┐
          │   Cloudflare    │
          │                 │
          │ D1 Database     │
          │ Vectorize Index │
          │ Workers AI      │
          └─────────────────┘

Setup Process

1. Initial Migration (from failed server)

# Export memories from existing machine
memory export /path/to/export.json

# Set up Cloudflare environment
export CLOUDFLARE_API_TOKEN="your-token"
export CLOUDFLARE_ACCOUNT_ID="your-account"
export CLOUDFLARE_D1_DATABASE_ID="your-d1-id"
export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index"
export MCP_MEMORY_STORAGE_BACKEND="cloudflare"

# Import to Cloudflare
python scripts/import_to_cloudflare.py /path/to/export.json

2. Configure Each Machine

Claude Desktop Configuration (claude_desktop_config.json):

{
  "mcpServers": {
    "memory": {
      "command": "/path/to/memory",
      "args": ["server"],
      "env": {
        "MCP_MEMORY_STORAGE_BACKEND": "cloudflare",
        "MCP_MEMORY_SQLITE_PATH": "/local/backup/path/sqlite_vec.db",
        "CLOUDFLARE_API_TOKEN": "your-token",
        "CLOUDFLARE_ACCOUNT_ID": "your-account",
        "CLOUDFLARE_D1_DATABASE_ID": "your-d1-id",
        "CLOUDFLARE_VECTORIZE_INDEX": "mcp-memory-index"
      }
    }
  }
}

3. Verification Testing

Test bidirectional sync by storing and retrieving memories from each machine:

# Test script for verification
import asyncio
from mcp_memory_service.storage.cloudflare import CloudflareStorage

async def test_sync():
    storage = CloudflareStorage(...)
    await storage.initialize()

    # Store test memory
    test_memory = Memory(content="Test from Machine A", tags=["sync-test"])
    success, message = await storage.store(test_memory)

    # Verify from other machine
    results = await storage.retrieve("Test from Machine A")
    print(f"Sync verified: {len(results)} results found")

Important Notes

  • v6.13.7 Required: This version fixes the critical Vectorize ID length issue
  • Breaking Change: Vector IDs changed format from v6.13.6 (removed "mem_" prefix)
  • Backup Strategy: Local sqlite_vec files are maintained for fallback
  • Migration Time: Allow extra time for initial memory migration to Cloudflare

Troubleshooting Sync Issues

Vector ID Length Error (Fixed in v6.13.7)

Error: "id too long; max is 64 bytes, got 68 bytes"

Solution: Update to v6.13.7 or later

Environment Variable Issues

Problem: Memories not syncing between machines Solution:

  • Verify identical environment variables on all machines
  • Check Claude Desktop configuration matches exactly
  • Restart Claude Desktop after config changes

Sync Verification

# Check memory count on each machine
memory status

# Test cross-machine visibility
memory retrieve "test query from other machine"

Support

For issues and questions:

  1. Documentation: Check this guide and API documentation
  2. GitHub Issues: Report bugs at the project repository
  3. Cloudflare Support: For Cloudflare service-specific issues
  4. Community: Join the project Discord/community channels

Performance Benchmarks

Typical Performance

  • Storage: ~200ms per memory (including embedding generation)
  • Search: ~100ms for semantic search (5 results)
  • Batch Operations: ~50ms per memory in batches of 100
  • Global Latency: <100ms from most global locations

Optimization Tips

  1. Batch Operations: Use bulk operations when possible
  2. Content Strategy: Use R2 for large content
  3. Caching: Enable embedding caching
  4. Connection Pooling: Reuse HTTP connections
  5. Regional Deployment: Deploy close to your users