Production-grade Email Scheduler

A production-grade, asynchronous email scheduling system capable of handling high throughput, enforcing strict rate limits, and surviving server restarts without data loss.

Explanation Video: Watch on Loom

Table of Contents

• Quick Start (Run Locally)
• Configuration
• --Architecture & Logic--
• --Deployment--
• Troubleshooting

---

Workflow & Architecture

Quick Start (Run Locally)

Follow these steps to get the application running on your local machine.

Prerequisites

• Docker & Docker Compose (for Database & Redis)
• Node.js (v18+)
• npm or yarn

1. Clone & Install

git clone https://github.com/your-username/Email_service-and-Dashboard.git
cd Email_service-and-Dashboard
npm install

2. Start Infrastructure

Start PostgreSQL and Redis containers:

docker-compose up -d

3. Setup Environment

Copy the example environment files:

# Backend
cp backend/.env.example backend/.env

# Frontend
cp frontend/.env.example frontend/.env

Tip: The default DATABASE_URL in .env works out-of-the-box with the Docker setup.

4. Initialize Database

Create the database tables:

cd backend
npx prisma migrate dev --name init
cd ..

5. Start the Application

You will need 3 terminal windows:

Terminal 1 (Backend API):

npm run dev --workspace=backend

Terminal 2 (Background Worker):

npm run worker --workspace=backend

Terminal 3 (Frontend Dashboard):

npm run dev --workspace=frontend

6. Access

• Dashboard: http://localhost:5173
• API: http://localhost:3000
• Mail (Ethereal): Check console logs for login URL to view sent emails.

---

Configuration

Key environment variables to configure in backend/.env.

Variable	Description	Default / Example
DATABASE_URL	Postgres Connection String	postgresql://user:password@localhost:5432/email_db
REDIS_URL	Redis Connection String	redis://localhost:6379
RATE_LIMIT_PER_HOUR	Max emails per user/hour	250
SMTP_USER	SMTP Username	Get from Ethereal
SMTP_PASS	SMTP Password	Get from Ethereal
GOOGLE_CLIENT_ID	OAuth Client ID	Google Cloud Console

---

🏗️ Architecture & Logic

System Architecture

The system follows a Producer-Consumer pattern:

1. Client: Dashboard for scheduling campaigns.
2. API (Producer): Validates requests, saves to DB (PENDING), adds jobs to BullMQ.
3. Queue: Redis-backed queue acting as a buffer and time-keeper.
4. Worker (Consumer): Processes jobs, enforces rate limits, sends emails.
5. DB: Postgres for persistent history; Redis for short-term state/counters.

Click to expand Core Logic (Scheduling, Rate Limiting)

A. Scheduling

• Uses BullMQ's delay feature to schedule jobs.
• Jobs reside in Redis until their execution time, surviving server restarts.

B. Multi-Tenant Rate Limiting

• Isolation: Each user has their own hourly limit (e.g., 250 emails/hr).
• Behavior:
  • If user is under limit: Send immediately.
  • If over limit: Delay the job to the next hour window.
  • Never Drop: Jobs are re-scheduled, not rejected.
• Implementation: Uses Redis INCR keys like sends:{userId}:{hourWindow}.

C. Concurrency & Idempotency

• Concurrency: Configurable worker concurrency to process multiple jobs in parallel.
• Idempotency: Checks "Sent" table before processing to prevent duplicate sends.

Click to expand Technology Stack

• Backend: Node.js, Express, TypeScript, BullMQ, Prisma
• Frontend: React, TailwindCSS, Vite
• Database: PostgreSQL
• Queue/Cache: Redis
• Infrastructure: Docker Compose

---

Deployment

Database & Redis

Use a managed provider like Railway or Supabase for Postgres and Redis. Update DATABASE_URL and REDIS_URL in your production env vars.

Backend & Worker

Deploy the backend code to a platform like Railway or Render.

• Start Command (API): npm run start
• Start Command (Worker): npm run worker:prod
  • Note: The worker is a separate process. You may need to deploy a second service or use a process manager (like PM2) to run both in one container.

Frontend

Deploy to Vercel or Netlify.

• Build Command: npm run build
• Output Directory: dist

---

🐞 Troubleshooting

Common Issues & Fixes

1. Redis Connection Refused

Error: connect ECONNREFUSED 127.0.0.1:6379

• Fix: Ensure Docker is running (docker-compose up -d).
• If running inside Docker (production), use the service name redis instead of localhost.

2. Database Migration Fails

• Ensure the database container is healthy: docker ps.
• Check credentials in DATABASE_URL.

3. Deployment: Worker crashes

• Make sure REDIS_URL is set correctly in production variables.
• Ensure the worker process has access to the internet for SMTP (port 587/465).

4. Viewing Database Data

Use Prisma Studio to visually inspect your database:

cd backend
npx prisma studio

---

📁 Project Structure

View Directory Structure

Email_service-and-Dashboard/
├── backend/          # Express API & Background Worker
│   ├── src/
│   │   ├── services/ # Business logic (Rate limiter, Email)
│   │   ├── worker.ts # Background job processor
│   │   └── index.ts  # API Entry point
├── frontend/         # React Dashboard
├── docs/             # Diagrams & Documentation
├── docker-compose.yml
└── README.md