Post-Deployment Sentry 🛡️

Real-time deployment monitoring with AI-powered risk analysis

A streaming system that monitors GitHub Actions deployments, detects post-deployment risks using live metrics and logs, and generates AI-powered explanations using RAG (Retrieval-Augmented Generation) with Google Gemini.

Key Features

• Real-time GitHub Webhook Ingestion - Receives deployment events from GitHub Actions
• Pathway Streaming Pipeline - Processes events in a live, incremental dataflow
• Live Metrics Integration - Queries Prometheus for error rates and latency
• Log Analysis - Fetches recent logs from Loki for context
• Risk Score Computation - Calculates deployment risk in real-time
• RAG + LLM Explanations - Uses ChromaDB + Gemini to generate contextual explanations
• Intelligent Alerts - Sends enriched Slack notifications with AI insights
• Full Observability - Prometheus, Grafana, Loki dashboard

Architecture

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  GitHub Actions  │────▶│  FastAPI Webhook │────▶│   JSONL File     │
└──────────────────┘     └──────────────────┘     └────────┬─────────┘
                                                           │
                         ┌─────────────────────────────────▼─────────────────────────────────┐
                         │                    PATHWAY STREAMING PIPELINE                      │
                         │  ┌──────────────────────────────────────────────────────────────┐ │
                         │  │  UDF 1: Parse Event + Fetch Prometheus Metrics + Loki Logs   │ │
                         │  └──────────────────────────────┬───────────────────────────────┘ │
                         │                                 ▼                                  │
                         │  ┌──────────────────────────────────────────────────────────────┐ │
                         │  │  UDF 2: Compute Risk Score                                   │ │
                         │  └──────────────────────────────┬───────────────────────────────┘ │
                         │                                 ▼                                  │
                         │  ┌──────────────────────────────────────────────────────────────┐ │
                         │  │  UDF 3: RAG + Gemini LLM Explanation (high-risk only)        │ │
                         │  │         - ChromaDB retrieves relevant docs                   │ │
                         │  │         - Gemini generates contextual explanation            │ │
                         │  └──────────────────────────────┬───────────────────────────────┘ │
                         │                                 ▼                                  │
                         │  ┌──────────────────────────────────────────────────────────────┐ │
                         │  │  OUTPUT: Alert with AI Explanation                          │ │
                         │  └──────────────────────────────────────────────────────────────┘ │
                         └───────────────────────────────────────────────────────────────────┘
                                                           │
                    ┌──────────────────────────────────────┼──────────────────────────────────┐
                    ▼                                      ▼                                  ▼
           ┌──────────────┐                       ┌──────────────┐                   ┌──────────────┐
           │    Slack     │                       │  PostgreSQL  │                   │  Dashboard   │
           │    Alert     │                       │  (Incidents) │                   │   (React)    │
           └──────────────┘                       └──────────────┘                   └──────────────┘

Quick Start

Prerequisites

• Docker & Docker Compose
• Gemini API Key (free from aistudio.google.com)

1. Clone and Configure

git clone https://github.com/yourname/post-deploy-sentry.git
cd post-deploy-sentry

# Copy and edit environment
cp .env.example .env
# Add your GOOGLE_API_KEY to .env

2. Start All Services

docker-compose up -d

3. View Logs

# Watch the Pathway stream
docker logs -f post-deploy-sentry-pathway

4. Trigger a Test Event

# Option A: Send webhook to API
curl -X POST http://localhost:8000/webhook/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: workflow_run" \
  -d @data/sample_webhook.json

# Option B: Append directly to JSONL (Pathway watches this)
cat data/sample_webhook.json >> data/deploy_events.jsonl

5. Watch the Magic!

======================================================================
DEPLOYMENT RISK ALERT WITH AI EXPLANATION
======================================================================
🔴 *HIGH RISK DEPLOYMENT DETECTED*

*Repository:* `myorg/production-api`
*Commit:* `abc123d`
*Status:* failure
*Risk Score:* *72.5* / 100
*Error Rate:* 8.50%
*Latency:* 450ms

---
*AI Analysis:*
**Likely Issue:** The deployment failed with elevated error rates, suggesting
the new code may have introduced breaking changes.

**Investigate First:**
- Check GitHub Actions logs for the failure reason
- Review error logs in Loki for specific exceptions
- Compare metrics before/after deployment

**Recommended Action:** Roll back to the previous version immediately while
investigating the root cause.
======================================================================

Project Structure

post-deploy-sentry/
├── backend/                  # FastAPI application
│   └── app/
│       ├── main.py          # Webhook endpoint
│       ├── logs.py          # Loki integration
│       ├── projects.py      # External project monitoring
│       └── insights.py      # Real-time insights
│
├── worker/                   # Background workers
│   ├── pathway_stream.py    # 🌟 MAIN: Pathway pipeline with RAG
│   ├── rag/                 # RAG module
│   │   ├── vector_store.py  # ChromaDB integration
│   │   ├── document_loader.py # Sample docs loader
│   │   └── llm_explainer.py # Gemini LLM integration
│   ├── worker.py            # Redis-based worker (legacy)
│   └── risk_score.py        # Risk computation
│
├── RagMod/frontend/         # React dashboard
│   └── src/
│       ├── pages/
│       │   ├── Dashboard.jsx
│       │   ├── LiveLogs.jsx
│       │   ├── ProjectSetup.jsx
│       │   └── ProjectInsights.jsx
│
├── sample_app/              # Demo app for generating metrics
├── config/                  # Prometheus, Loki, Promtail configs
├── data/                    # JSONL events (Pathway watches this)
└── docker-compose.yml       # Full stack orchestration

API Endpoints

Endpoint	Method	Description
/webhook/github	POST	Receive GitHub workflow events
/health	GET	Service health check
/logs/live	GET	Live logs from Loki
/projects	GET/POST	Manage monitored projects
/projects/{id}/insights	GET	Real-time project insights
/ingest/{token}	POST	External log ingestion webhook

Demo Script

# 1. Start the system
docker-compose up -d

# 2. Wait for services (30 seconds)
sleep 30

# 3. Generate some traffic to create metrics
curl http://localhost:8001/slow
curl http://localhost:8001/error
curl http://localhost:8001/error

# 4. Trigger a deployment event
curl -X POST http://localhost:8000/webhook/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: workflow_run" \
  -d @data/sample_webhook.json

# 5. Watch Pathway process and generate AI explanation
docker logs -f post-deploy-sentry-pathway

# 6. Open dashboard
# Visit http://localhost:5173

📊 Observability URLs

Service	URL
FastAPI Backend	http://localhost:8000
React Dashboard	http://localhost:5173
Sample App	http://localhost:8001
Prometheus	http://localhost:9090
Grafana	http://localhost:3000
Loki (query)	http://localhost:3100