Skip to content

Latest commit

 

History

History
429 lines (349 loc) · 15 KB

File metadata and controls

429 lines (349 loc) · 15 KB

📊 QuickLogs — Edge-First SIEM

Open-source SIEM with 98% cost reduction vs Splunk — Rust + ClickHouse + ML log summarization

Enterprise security monitoring at 1/50th the cost. Edge aggregation + 99% data compression means a $18,000/mo Splunk workload runs for $367/mo.

Cybersecurity SIEM License: Apache 2.0 Built by Networkers Home


🏛️ Built by Networkers Home

QuickLogs was built by Networkers Home — India's leading Cisco + cybersecurity training institute (Bengaluru, since 2005). It's the same SIEM stack our students learn in the Cybersecurity Pro program — and used by our 4-month paid SOC internship cohort to triage real customer logs.

The reason most SOC analyst training stops at "watch Splunk demos" is cost. QuickLogs is the open-source SIEM Networkers Home uses to give every student real hands-on time on log analysis without burning a $50K Splunk POC budget. Book a demo class →

Compare top cybersecurity institutes: Top 10 Cybersecurity India · Top 10 SOC Analyst Bangalore · Top 10 Cloud Security India


EdgeAI Telemetry - Edge-First SIEM

Enterprise-grade security monitoring with 98% cost reduction through edge aggregation and 99% data compression

🎯 What Makes This Different

Traditional SIEMs send every raw event to the cloud. EdgeAI sends only summaries.

Metric Traditional SIEM EdgeAI Telemetry Savings
Network Traffic 10,000 events/sec 100 summaries/sec 99%
Storage 100% raw events 1% summaries only 99%
Compute Parse 10k events Parse 100 summaries 99%
Monthly Cost $18,000 $367 98%

✨ Enterprise Features

🔐 Authentication & Security

  • SSO/OIDC - Azure AD, Okta, Google Workspace, Auth0, Keycloak
  • MFA/TOTP - Google Authenticator, Authy support
  • RBAC - Granular role-based access control
  • API Keys - Scoped access for integrations
  • Audit Logging - Complete activity trail with integrity verification

🔔 Alerting & Notifications

  • Multi-channel alerts - Email, Slack, Webhook, PagerDuty
  • Smart rules - Risk score thresholds, incident-based, anomaly detection
  • Rate limiting - Cooldown periods and max alerts per hour
  • Alert management - Acknowledge, resolve, track history

👥 User Management

  • Role management - Admin, Analyst, Viewer, Security Engineer
  • User provisioning - Manual or SSO JIT (Just-in-Time)
  • Permission system - Resource-level access control
  • Session management - Secure JWT tokens with expiration

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    SINGLE VM (Phase 1 - MVP)                    │
│  ┌─────────────────┐        ┌───────────────────────────────┐   │
│  │  Edge Agent     │        │  Cloud Server                 │   │
│  │  (Same VM)      │───────▶│  - gRPC Receiver              │   │
│  │                 │ 1%     │  - Summary Storage            │   │
│  │  • 60s Windows  │ traffic│  - AI Correlation             │   │
│  │  • Local ML     │        │  - Graph Analysis             │   │
│  │  • zstd (5:1)   │        │  - Web Dashboard              │   │
│  └─────────────────┘        │  - SSO/OIDC Auth              │   │
│                             │  - Alerting Engine            │   │
│                             │  - Audit Logging              │   │
│                             └───────────────────────────────┘   │
│                                                              │
│  Test: cat events.jsonl | ./edge-agent --test-file /data/...   │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Phase 2 (Client Testing)
                    ┌─────────────────┐
                    │ Client Server   │  (Separate Linux VM)
                    │  - Edge Agent   │  • Runs edge binary
                    │  - Sends 1%     │  • Buffer on disconnect
                    └─────────────────┘  • Auto-reconnect

🚀 Quick Start

Prerequisites

  • Docker & Docker Compose
  • 4 vCPU, 8GB RAM VM
  • PostgreSQL 14+

Single VM Deployment

cd "siem ai"

# Run database migrations
docker-compose exec postgres psql -U edgeai -d edgeai -f /migrations/001_initial_schema.sql
docker-compose exec postgres psql -U edgeai -d edgeai -f /migrations/003_sso_alerting_audit.sql

# Start all services
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f cloud-server
docker-compose logs -f edge-agent

# Access dashboard
open http://localhost:3000

Default Login

  • Email: admin@example.com
  • Password: admin123
  • ⚠️ Change immediately in production!

📊 Cost Breakdown

Traditional SIEM (10k events/sec)

Network Ingress:  $5,000/mo  (bandwidth)
Storage:         $10,000/mo  (raw events)
Compute:          $3,000/mo  (parsing)
─────────────────────────────────
Total:           $18,000/mo

EdgeAI Telemetry

Network Ingress:     $18/mo  (1% of traffic)
Storage:            $150/mo  (summaries only)
Compute:            $199/mo  (correlation only)
─────────────────────────────────
Total:              $367/mo

SAVINGS: 98% ($17,633/mo)

🔧 Components

Edge Agent (edge-agent/)

  • Binary Size: <10MB
  • Memory: <50MB RAM
  • CPU: <5% single core
  • Language: Rust
// 60-second tumbling window aggregation
WindowAggregator::new(window_seconds: 60)

// Local anomaly detection (Isolation Forest)
AnomalyDetector::new(threshold: 70.0)

// Offline resilience (SQLite buffer)
LocalBuffer::new("/var/lib/edgeai/buffer.db")

Cloud Server (cloud-server/)

  • HTTP API: REST for dashboard (port 8080)
  • gRPC: Binary protocol for agents (port 50051)
  • Storage: PostgreSQL for summaries
  • Correlation: Rule-based AI engine
  • Auth: JWT + SSO/OIDC + MFA

Protocol (gRPC + Protobuf)

message EventSummary {
  bytes identity_hash = 1;      // 16 bytes (not string)
  uint32 event_type_id = 2;     // 4 bytes (lookup)
  uint32 count = 3;             // 4 bytes
  uint64 window_start = 4;      // 8 bytes
  map<uint32, uint32> severity_dist = 6;
  bytes compressed_samples = 8; // zstd compressed
}
// Total: ~50 bytes vs 500 bytes JSON = 10x smaller

🔐 Authentication & SSO

Supported Providers

  • Azure Active Directory / Microsoft Entra ID
  • Okta
  • Google Workspace
  • Auth0
  • Keycloak
  • Any OIDC-compliant provider

Setup SSO

# 1. Configure provider in database
psql -U edgeai -d edgeai -c "
INSERT INTO sso_providers (name, provider_type, client_id, client_secret, issuer_url, scopes, is_active, is_default)
VALUES ('Azure AD', 'oidc', 'your-client-id', 'your-secret', 'https://login.microsoftonline.com/{tenant}/v2.0', ARRAY['openid','email','profile'], true, true);
"

# 2. Enable SSO
export ENABLE_SSO=true
export JWT_SECRET=your-256-bit-secret

See SSO_OIDC_SETUP.md for detailed configuration.

🔔 Alerting Configuration

Create Alert Rule

curl -X POST http://localhost:8080/api/v1/alerting/rules \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High Risk Score Alert",
    "condition_type": "risk_score_threshold",
    "risk_score_min": 70,
    "cooldown_minutes": 60,
    "channel_ids": ["uuid-of-slack-channel"]
  }'

Supported Channels

  • Email (SMTP)
  • Slack (webhooks)
  • Microsoft Teams
  • PagerDuty
  • Generic Webhook

🧪 Testing

Load Test

# Generate 1 million test events
cd test-data
python3 generate_events.py --count 1000000 --output events.jsonl

# Run through edge agent
docker-compose exec edge-agent ./edgeai-agent \
  --test-file /data/events.jsonl \
  --server http://cloud-server:50051

# Expected: 10,000 raw events → 100 summaries (100:1 reduction)

Verify Data Reduction

# Check metrics endpoint
curl http://localhost:8080/api/v1/dashboard/metrics | jq

# Expected response:
{
  "success": true,
  "data": {
    "data_reduction_percent": 99.0,
    "summaries_received": 100,
    "anomalies_received": 5
  }
}

📁 File Structure

siem-ai/
├── edge-agent/           # Edge aggregation agent (Rust)
│   ├── src/
│   │   ├── aggregator.rs    # 60s window aggregation
│   │   ├── detector.rs      # Isolation Forest ML
│   │   ├── buffer.rs        # SQLite offline buffer
│   │   ├── sender.rs        # gRPC client
│   │   └── main.rs
│   └── Cargo.toml
│
├── cloud-server/         # Cloud aggregation receiver (Rust)
│   ├── src/
│   │   ├── grpc.rs          # gRPC service
│   │   ├── correlation.rs   # AI correlation engine
│   │   ├── storage.rs       # Summary storage
│   │   ├── auth/            # Authentication & SSO
│   │   │   ├── mod.rs       # JWT & middleware
│   │   │   ├── oidc.rs      # OIDC/OAuth2 client
│   │   │   ├── rbac.rs      # Role-based access control
│   │   │   └── mfa.rs       # TOTP MFA
│   │   ├── alerting/        # Alert rules & notifications
│   │   ├── audit/           # Audit logging
│   │   └── api/             # REST API endpoints
│   └── Cargo.toml
│
├── shared/proto/         # Protobuf definitions
│   └── telemetry.proto
│
├── frontend/             # React + TypeScript dashboard
│   ├── src/
│   │   ├── pages/
│   │   │   ├── Dashboard.tsx
│   │   │   ├── Incidents.tsx
│   │   │   ├── UserManagement.tsx   # User & role management
│   │   │   ├── AlertConfig.tsx      # Alert rules & channels
│   │   │   ├── AuditLogs.tsx        # Audit log viewer
│   │   │   ├── VendorComparison.tsx
│   │   │   └── UserGuide.tsx
│   │   ├── components/
│   │   └── App.tsx
│   └── package.json
│
├── migrations/           # PostgreSQL schema
│   ├── 001_initial_schema.sql
│   └── 003_sso_alerting_audit.sql
│
├── test-data/            # Test event generators
├── docker-compose.yml    # Single VM orchestration
└── SSO_OIDC_SETUP.md     # SSO configuration guide

🛣️ Migration Path

Phase Timeline Change
Phase 1 Now Single VM, co-located agent (MVP)
Phase 2 Week 5-6 Agent moves to client servers
Phase 3 Month 3 Kafka streaming, multiple agents
Phase 4 Month 6 Distributed graph DB

🔒 Security Features

Feature Description
Identity Hashing Blake3 (16-byte hashes, not plain text)
Transport gRPC with mTLS
Compression zstd with bound memory
Buffer Local SQLite encrypted at rest
Authentication JWT tokens with expiration
SSO/OIDC Azure AD, Okta, Google, Auth0, Keycloak
MFA TOTP (Google Authenticator, Authy)
RBAC Granular resource-level permissions
Audit Tamper-evident activity logging
API Keys Scoped access tokens

📈 Monitoring

Metric Target Dashboard
Data Reduction 99% ✅ Shows real-time %
Events/Sec 10k → 100 ✅ Before/after
Agent CPU <5% 🔄 Via heartbeat
Agent Memory <50MB 🔄 Via heartbeat
Active Users - ✅ User management
Alert Status - ✅ Alert dashboard
Audit Events - ✅ Audit log viewer

🆚 Comparison with Traditional SIEM

Feature Splunk Datadog EdgeAI (This)
Data Ingestion $$$/GB $$$/GB 99% less
Raw Storage Required Required Not stored
Edge Processing ✅ Core feature
Local ML ✅ Isolation Forest
Offline Buffer ✅ SQLite
SSO/SAML ✅ OIDC
MFA ✅ TOTP
RBAC ✅ Granular
Alerting ✅ Multi-channel
Audit Logging ✅ Integrity hashed
Cost at 10k EPS $20k/mo $18k/mo $367/mo

📚 Documentation

🔌 API Reference

Authentication

POST /api/v1/auth/login
POST /api/v1/auth/sso/login/:provider_id
POST /api/v1/auth/sso/callback/:provider_id
POST /api/v1/auth/mfa/setup
POST /api/v1/auth/mfa/enable

Users & Roles

GET    /api/v1/users
POST   /api/v1/users
GET    /api/v1/roles
GET    /api/v1/permissions

Alerting

GET    /api/v1/alerting/rules
POST   /api/v1/alerting/rules
GET    /api/v1/alerting/channels
POST   /api/v1/alerting/alerts/:id/acknowledge

Audit

GET    /api/v1/audit/logs
GET    /api/v1/audit/stats
GET    /api/v1/audit/export

🤝 Contributing

  1. Test with: docker-compose up
  2. Check cost metrics on dashboard
  3. Verify 99% data reduction
  4. Run migrations for new features

📄 License

MIT - See LICENSE file


Built for 5-year production survival with 98% cost savings.