warpspeedopen-source · elevasyncsolutions-jpg · Jul 13, 2026
diff --git a/bounty-1/DESIGN.md b/bounty-1/DESIGN.md
@@ -0,0 +1,140 @@
+# Attachment Summarizer — Architecture Design
+
+## Overview
+
+A Node.js background service that processes email attachment events from an SQS queue,
+downloads the attachments from Google Cloud Storage, extracts text content from PDF, DOCX,
+TXT, and image files, and generates natural-language summaries using a self-hosted
+open-source LLM (Llama 3 / Mistral via Ollama).
+
+## System Flow
+
+```
+Email System
+    │
+    │ (S3/GCS event notification → SQS)
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│                    SQS Queue                            │
+│            AttachmentEvent { token, gcsUri }            │
+└─────────────────────────────────────────────────────────┘
+    │
+    │ Poll (long-poll, batch size 10)
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│              sqsConsumer.ts                             │
+│  - Receives SQS messages                                │
+│  - Parses AttachmentEvent                               │
+│  - Passes to orchestrator                               │
+│  - Deletes message on success / DLQ on failure          │
+└─────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│              gcsDownloader.ts                           │
+│  - Authenticates via GCP service account (ADC)          │
+│  - Downloads file to local temp storage                 │
+│  - Returns file path + content-type                     │
+└─────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│              contentExtractor.ts                        │
+│  - Routes on MIME type                                  │
+│    ├─ application/pdf  → pdf-parse                     │
+│    ├─ application/vnd.openxmlformats-officedocument     │
+│    │   .wordprocessingml.document → mammoth            │
+│    ├─ text/plain       → fs.readFile                   │
+│    └─ image/*          → tesseract OCR                 │
+│  - Returns ExtractionResult { text, metadata }          │
+└─────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│              summarizer.ts                              │
+│  - Calls self-hosted LLM (Ollama API)                  │
+│  - Constructs prompt with extracted text               │
+│  - Returns Summary { id, summary, createdAt }           │
+└─────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌─────────────────────────────────────────────────────────┐
+│              PostgreSQL via Prisma                      │
+│  - Stores Attachment, ExtractionResult, Summary         │
+│  - Enables idempotency dedup by SQS message ID          │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Component Responsibilities
+
+### 1. SQS Consumer (`sqsConsumer.ts`)
+- Long-polls SQS in an infinite loop (configurable interval).
+- Parses `AttachmentEvent` from message body.
+- Runs the full pipeline: download → extract → summarize → persist.
+- On success: deletes the message from SQS.
+- On failure after retries: sends to DLQ (configured at queue level).
+- Uses `@aws-sdk/client-sqs`.
+
+### 2. GCS Downloader (`gcsDownloader.ts`)
+- Parses `gs://bucket/path` URIs.
+- Streams files to a temp directory using `@google-cloud/storage`.
+- Cleans up temp files after processing.
+- Returns local path and detected MIME type.
+
+### 3. Content Extractor (`contentExtractor.ts`)
+- Routes by MIME type to the appropriate extraction engine.
+- PDF: `pdf-parse` (PDF.js-based, extracts text).
+- DOCX: `mammoth` (converts DOCX to raw text).
+- TXT: Direct `fs.readFile` + UTF-8 decode.
+- Images (JPEG, PNG, TIFF): `tesseract.js` OCR.
+- Returns a normalized `ExtractionResult`.
+
+### 4. Summarizer (`summarizer.ts`)
+- Constructs a prompt with the extracted text and a summarization instruction.
+- Calls Ollama REST API (`http://localhost:11434/api/generate`).
+- Model: `llama3.1:8b` or `mistral:7b` (configurable).
+- Configures max tokens, temperature, prompt template.
+- Parses and returns the generated summary text.
+
+### 5. Database (Prisma + PostgreSQL)
+- Tracks each attachment's processing state.
+- Avoids duplicate processing via SQS message dedup ID.
+- Stores raw extracted text and the final summary.
+
+## Data Flow (Sequence)
+
+```
+SQS ──► sqsConsumer ──► gcsDownloader ──► contentExtractor ──► summarizer ──► Prisma
+  │         │               │                   │                   │            │
+  │    parse event      download file       extract text         call LLM    persist
+  │    delete msg       cleanup temp        return text          return       ack
+```
+
+## Error Handling & Retry
+
+| Layer | Strategy |
+|-------|----------|
+| SQS   | Visibility timeout (30s); message goes to DLQ after 3 receives |
+| GCS   | Retry with exponential backoff (3 attempts) |
+| Extraction | Catch per-file errors; skip unsupported types gracefully |
+| LLM   | Retry on 5xx / timeout (2 attempts); fallback to truncated summary |
+
+## Security Considerations
+
+- GCS access via Workload Identity Federation or service account key (ADC).
+- SQS access via IAM role with least-privilege policy.
+- LLM runs on localhost — no network exposure.
+- Temp files cleaned immediately after extraction.
+- No secrets in code — all via environment variables.
+
+## Deployment
+
+- Docker container deployed to Cloud Run (or ECS Fargate).
+- Single-process, no HTTP server needed (background worker).
+- Configure `SQS_QUEUE_URL`, `GOOGLE_APPLICATION_CREDENTIALS`, `DATABASE_URL`, `OLLAMA_BASE_URL`.
+
+## Scaling
+
+- Horizontally scalable: each instance polls the same SQS queue.
+- SQS visibility timeout must exceed max processing time per message.
+- Use SQS batch size of 10 for throughput.
diff --git a/bounty-1/README.md b/bounty-1/README.md
@@ -0,0 +1,125 @@
+# Attachment Summarizer
+
+Consumes email attachment events from AWS SQS, downloads files from Google Cloud Storage, extracts text (PDF, DOCX, TXT, images), and generates summaries via a self-hosted open-source LLM (Ollama).
+
+## Prerequisites
+
+- Node.js 20+
+- PostgreSQL 15+
+- AWS SQS queue + IAM credentials
+- Google Cloud Storage bucket + service account
+- Ollama with a model pulled (`llama3.1:8b` or `mistral:7b`)
+
+## Quick Start
+
+```bash
+# Install dependencies
+npm install
+
+# Generate Prisma client
+npx prisma generate
+
+# Run migrations
+npx prisma migrate deploy
+
+# Start the consumer
+node -r dotenv/config dist/index.js
+```
+
+## Environment Variables
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `SQS_QUEUE_URL` | yes | — | SQS queue URL |
+| `AWS_REGION` | no | `us-east-1` | AWS region |
+| `GOOGLE_APPLICATION_CREDENTIALS` | no | — | Path to GCP service account JSON |
+| `DATABASE_URL` | yes | — | PostgreSQL connection string |
+| `OLLAMA_BASE_URL` | no | `http://localhost:11434` | Ollama API base URL |
+| `LLM_MODEL` | no | `llama3.1:8b` | Ollama model name |
+| `MAX_INPUT_TOKENS` | no | `3072` | Max tokens to send to LLM |
+| `MAX_OUTPUT_TOKENS` | no | `512` | Max tokens in generated summary |
+
+## SQS Message Format
+
+Send messages with this JSON body:
+
+```json
+{
+  "attachmentId": "uuid-or-id",
+  "gcsUri": "gs://bucket-name/path/to/file.pdf",
+  "fileName": "invoice.pdf",
+  "mimeType": "application/pdf",
+  "receivedAt": "2026-07-14T12:00:00Z"
+}
+```
+
+## Architecture
+
+See [DESIGN.md](./DESIGN.md) for full architecture details.
+
+### Directory Layout
+
+```
+├── types.ts                    # TypeScript type definitions
+├── services/
+│   ├── sqsConsumer.ts          # SQS long-poll consumer
+│   ├── gcsDownloader.ts        # GCS file downloader
+│   ├── contentExtractor.ts     # PDF/DOCX/TXT/image extraction
+│   └── summarizer.ts           # Ollama LLM summarization
+├── schema.prisma               # Prisma schema (PostgreSQL)
+├── DESIGN.md                   # Architecture document
+└── README.md                   # This file
+```
+
+## Docker
+
+```dockerfile
+FROM node:20-slim
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    tesseract-ocr tesseract-ocr-eng poppler-utils && \
+    rm -rf /var/lib/apt/lists/*
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+COPY dist/ ./dist/
+COPY prisma/ ./prisma/
+RUN npx prisma generate
+CMD ["node", "dist/index.js"]
+```
+
+## Deployment (Cloud Run)
+
+```bash
+gcloud builds submit --tag gcr.io/$PROJECT_ID/attachment-summarizer
+gcloud run deploy attachment-summarizer \
+  --image gcr.io/$PROJECT_ID/attachment-summarizer \
+  --platform managed \
+  --region us-central1 \
+  --set-env-vars="SQS_QUEUE_URL=...,DATABASE_URL=...,OLLAMA_BASE_URL=..."
+```
+
+## Development
+
+```bash
+cp .env.example .env
+npm run dev        # tsx watch src/index.ts
+npm run build      # tsc
+npm run lint       # biome check
+```
+
+## Supported File Types
+
+| Type | Extension | Engine |
+|---|---|---|
+| PDF | `.pdf` | `pdf-parse` |
+| DOCX | `.docx` | `mammoth` |
+| TXT | `.txt`, `.csv`, `.log`, `.md` | `fs.readFile` |
+| JPEG | `.jpg`, `.jpeg` | `tesseract.js` |
+| PNG | `.png` | `tesseract.js` |
+| TIFF | `.tif`, `.tiff` | `tesseract.js` |
+
+## Monitoring
+
+- Logs: structured JSON via `pino` (recommended).
+- Metrics: expose Prometheus gauge for queue depth and processing latency.
+- Alerts: DLQ message count > 0 should trigger an alert.
diff --git a/bounty-1/lib/prisma.ts b/bounty-1/lib/prisma.ts
@@ -0,0 +1,8 @@
+import { PrismaClient } from '@prisma/client';
+
+export const prisma = new PrismaClient({
+  log:
+    process.env.NODE_ENV === 'development'
+      ? ['query', 'warn', 'error']
+      : ['warn', 'error'],
+});
diff --git a/bounty-1/schema.prisma b/bounty-1/schema.prisma
@@ -0,0 +1,39 @@
+generator client {
+  provider = "prisma-client-js"
+}
+
+datasource db {
+  provider = "postgresql"
+  url      = env("DATABASE_URL")
+}
+
+enum AttachmentStatus {
+  PENDING
+  PROCESSING
+  COMPLETED
+  FAILED
+}
+
+model Attachment {
+  id           String           @id @default(uuid())
+  attachmentId String           @unique
+  fileName     String
+  mimeType     String
+  gcsUri       String
+  sqsMessageId String?
+  status       AttachmentStatus @default(PENDING)
+
+  extractedText String?
+  summaryText   String?
+  summaryModel  String?
+
+  errorMessage String?
+  retryCount   Int      @default(0)
+
+  createdAt   DateTime @default(now())
+  processedAt DateTime?
+
+  @@index([attachmentId])
+  @@index([status])
+  @@index([createdAt])
+}