Golden Signal Dashboard

Student Name: Bhavya Jain
Registration No: 23FE10CSE00831
Course: CSE3253 DevOps [PE6]
Semester: VI (2025–2026)
Project Type: Puppet / Monitoring
Difficulty: Intermediate

Project Overview

The Golden Signal Dashboard is a Python Streamlit application that visualizes the four SRE golden signals—latency, traffic, errors, and saturation—from live Prometheus metrics or reproducible CSV sample data. It is designed for academic submission under the Puppet / Monitoring category and includes Docker, automated tests, GitHub Actions CI, and minimal Puppet manifests for configuration alignment.

Problem Statement

Operators and developers often struggle to interpret scattered metrics. This project provides a single, lightweight dashboard that queries or loads service-wise time series, applies clear thresholds, and surfaces Healthy / Warning / Critical health states alongside interactive charts.

Objectives

• Visualize latency, traffic, errors, and saturation with charts, summary cards, and threshold-based health states
• Provide reproducible sample data, automated tests, and GitHub Actions CI
• Align with the Puppet / Monitoring category using Docker, documentation, and Puppet configuration samples

Key Features

Feature 1 — Interactive Plotly line charts for latency, traffic, errors, and saturation, with warning/critical reference lines where applicable.

Feature 2 — Sidebar filters for services and time window, plus per-service health derived from thresholds.

Feature 3 — Reproducible sample data via utils/generate_sample_data.py, pytest checks, and a GitHub Actions workflow.

Feature 4 — Strict CSV schema and range validation with clear UI errors, plus an optional DASHBOARD_DATA_PATH override for external datasets.

Feature 5 — Live Prometheus range queries, automatic dashboard refresh, and a self-contained metrics exporter for local demonstrations.

Technology Stack

Core Technologies

Programming Language: Python
Framework: Streamlit
Metrics storage: Prometheus TSDB (live mode) or CSV fallback

DevOps Tools

Version Control: Git
CI/CD: GitHub Actions
Containerization: Docker
Orchestration: Kubernetes (not used in this submission)
Configuration Management: Puppet (sample manifests and EPP template)
Monitoring: Prometheus, PromQL, and application-level golden-signal dashboard

Getting Started

Prerequisites

• Docker Desktop v20.10+ (recommended)
• Git 2.30+
• Python 3.11+ (for local runs without Docker)

Installation

1. Clone the repository:

git clone https://github.com/bhavyajain0810/devopsprojectgoldensignaldashboard.git
cd devopsprojectgoldensignaldashboard

2. Start the complete live monitoring stack:

docker compose up --build

This starts the live metrics exporter, Prometheus, and Streamlit dashboard. For CSV-only mode, build and run just the dashboard image:

docker build -t devopsprojectgoldensignaldashboard:latest .
docker run --rm -p 8501:8501 devopsprojectgoldensignaldashboard:latest

3. Access the application:

Web Interface: http://localhost:8501

Prometheus Interface: http://localhost:9090

Alternative Installation (Without Docker)

cd devopsprojectgoldensignaldashboard
python -m venv .venv
# Windows:
.venv\Scripts\activate
# Linux/macOS:
# source .venv/bin/activate
pip install -r requirements.txt
streamlit run app.py

For development and tests, install requirements-dev.txt instead.

Open http://localhost:8501. If data/sample_metrics.csv is missing, run:

python utils/generate_sample_data.py

Project Structure

devopsprojectgoldensignaldashboard/
├── README.md
├── app.py
├── metrics.py
├── prometheus_source.py
├── requirements.txt
├── requirements-dev.txt
├── Dockerfile
├── docker-compose.yml
├── .dockerignore
├── .gitignore
├── LICENSE
├── data/
│   └── sample_metrics.csv
├── monitoring/
│   ├── exporter.py
│   └── prometheus.yml
├── utils/
│   └── generate_sample_data.py
├── tests/
│   ├── test_app.py
│   ├── test_metrics_file.py
│   └── test_prometheus_source.py
├── docs/
│   ├── architecture.md
│   ├── implementation.md
│   └── project_report.md
├── .github/
│   └── workflows/
│       └── ci.yml
├── .devcontainer/
│   └── devcontainer.json
├── screenshots/
│   └── dashboard.png
└── puppet/
    ├── manifests/
    │   └── init.pp
    └── templates/
        └── dashboard.conf.epp

Configuration

Environment Variables

The default source is CSV. Select Prometheus and configure its endpoint with:

$env:METRICS_SOURCE = "prometheus"
$env:PROMETHEUS_URL = "http://localhost:9090"
streamlit run app.py

METRICS_SOURCE=prometheus PROMETHEUS_URL=http://localhost:9090 streamlit run app.py

Optional settings include PROMETHEUS_LOOKBACK_MINUTES, PROMETHEUS_STEP_SECONDS, PROMETHEUS_TIMEOUT_SECONDS, DASHBOARD_REFRESH_SECONDS, and the four PROMETHEUS_*_QUERY overrides. DASHBOARD_DATA_PATH still selects a different CSV file in CSV mode.

Key Configuration Files

1. app.py — Streamlit dashboard entrypoint
2. metrics.py — Shared data validation and health logic
3. prometheus_source.py — Prometheus HTTP API and PromQL adapter
4. monitoring/prometheus.yml — Prometheus scrape configuration
5. monitoring/exporter.py — Live demonstration metrics target
6. data/sample_metrics.csv — Offline fallback metrics
7. Dockerfile — Non-root application image

CI/CD Pipeline

Pipeline Stages

1. Checkout — Clone repository on GitHub-hosted runner
2. Setup Python — Python 3.11
3. Install dependencies — pip install -r requirements-dev.txt
4. Test — pytest tests/ -v
5. Container Build — Build the Docker image to catch packaging regressions
6. Compose Validation — Validate the three-service monitoring stack
7. Security Scan — Not configured in this submission (future scope: e.g. Trivy on image)
8. Deploy — Not configured in this submission (manual/local Docker only)

Pipeline Status

Testing

Test Types

Unit Tests: pytest (python -m pytest tests/ -v) Integration Tests: Streamlit render smoke test E2E Tests: Not included

Test Coverage

Coverage reporting is not configured; the suite covers sample-data compatibility, validation failures, environment configuration, and health-threshold boundaries.

Monitoring

Monitoring Setup

Dashboard: Streamlit UI querying Prometheus or reading the CSV fallback. Custom metrics: A live exporter exposes per-service counters, histograms, and CPU gauges. Prometheus: Scrapes the exporter every five seconds and retains local data for 24 hours. Alerts: Not implemented (thresholds are visual only in the UI).

Docker

Build and run with Docker Compose

docker compose up --build

Open the dashboard at http://localhost:8501 and Prometheus at http://localhost:9090. Stop the stack with docker compose down.

Docker Images

# Build image
docker build -t devopsprojectgoldensignaldashboard:latest .

# Run container
docker run --rm -p 8501:8501 devopsprojectgoldensignaldashboard:latest

🚀 Live Demo

Open the hosted dashboard

The hosted demo uses CSV fallback because it cannot reach Prometheus running on your local Docker network.

Documentation

User Documentation

Project Report

Prometheus API integration is documented in the Implementation Guide.

Technical Documentation

Architecture Overview
Implementation Guide

DevOps Documentation

Docker, CI, and Puppet notes: Implementation Guide
Troubleshooting: Implementation Guide — Troubleshooting

Development Workflow

Git Branching Strategy

main
├── develop
│   ├── feature/latency-charts
│   ├── feature/puppet-config
│   └── fix/csv-validation
└── release/v1.0.0

Commit Convention

• feat: New feature
• fix: Bug fix
• docs: Documentation
• test: Test-related
• refactor: Code refactoring
• chore: Maintenance tasks

Security

Security Measures Implemented

• Input validation and sanitization (required schema, timestamps, numeric types, bounds, and service names)
• Authentication and authorization (not in scope)
• Environment-based data-source configuration
• Non-root container runtime
• Regular dependency updates (recommended practice for maintainers)
• Security headers in web applications (reverse-proxy concern for production)

Security Scanning

# Example (not run in CI for this submission):
# trivy image devopsprojectgoldensignaldashboard:latest

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License — see the LICENSE file for details.

Project Challenges

1. Balancing template breadth vs. repo simplicity — The course template describes a large multi-folder layout; this submission uses the prescribed flat academic tree while keeping README sections aligned with the template.
2. Health logic vs. traffic — Traffic spikes are not automatically unhealthy; health emphasizes latency, errors, and saturation.
3. Puppet without a full control repo — Manifests are minimal and documented so they can be relocated into a proper module path when used with a real Puppet server.

Learnings

• Golden signals provide a compact mental model for service health.
• Container images make grading and demos repeatable across machines.
• A pluggable metrics source keeps live monitoring and deterministic tests compatible.

Acknowledgments

Course Instructor: Mr. Jay Shankar Sharma
Reference materials and tutorials
Open-source tools and libraries

Contact

Student: Bhavya Jain GitHub: bhavyajain0810 Course Coordinator: Mr. Jay Shankar Sharma Consultation Hours: Thursday & Friday, 5–6 PM, LHC 308F