Skip to content

Monitoring_Wiki.md

Latest commit

 

History

History
148 lines (114 loc) · 7.46 KB

Monitoring_Wiki.md

File metadata and controls

148 lines (114 loc) · 7.46 KB

Monitoring

Definition

Monitoring in software systems is the act of systematically collecting, analyzing, and presenting data about the performance, health, and behavior of a system. It involves setting up tools and processes to continuously observe various aspects of an application and its infrastructure, providing insights into its operational state.

While closely related to Observability, Monitoring typically focuses on known unknowns – predefined metrics and logs that indicate whether a system is working as expected. Observability, on the other hand, aims to understand unknown unknowns – allowing you to ask arbitrary questions about the system's internal state.

Key Aspects of Monitoring

  1. What to Monitor:

    • Infrastructure Metrics: CPU usage, memory consumption, disk I/O, network traffic, process counts.
    • Application Metrics: Request rates, error rates, latency, throughput, queue sizes, garbage collection times, database connection pools.
    • Business Metrics: User sign-ups, conversion rates, transaction volumes, revenue.
    • Logs: Application logs, server logs, access logs, security logs.
    • Dependencies: Health and performance of external services, databases, and APIs.

  2. How to Monitor:

    • Agents: Software installed on servers to collect metrics and logs.
    • Exporters: Components that expose metrics in a format consumable by monitoring systems (e.g., Prometheus exporters).
    • APIs: Direct integration with cloud provider APIs or application-specific APIs.
    • Synthetic Monitoring: Simulating user interactions from outside the system to check availability and performance.
    • Real User Monitoring (RUM): Collecting data from actual user browsers or mobile apps.

  3. Monitoring Tools:

    • Metrics Collection & Storage: Prometheus, Graphite, InfluxDB, Datadog, New Relic.
    • Visualization & Dashboards: Grafana, Kibana, Datadog, New Relic.
    • Alerting: Alertmanager (for Prometheus), PagerDuty, Opsgenie, custom scripts.
    • Log Management: ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Loki, Sumo Logic.

Types of Monitoring

  • Black-box Monitoring: Observing the system from the outside, as a user would. Focuses on external behavior (e.g., HTTP response codes, latency from a client perspective).
  • White-box Monitoring: Observing the system from the inside, using metrics and logs exposed by the application's internals (e.g., CPU usage of a specific process, internal queue lengths).
  • Synthetic Monitoring: Proactive monitoring by simulating user transactions at regular intervals from various locations.
  • Real User Monitoring (RUM): Passive monitoring that collects data from actual user interactions with the application.

Alerting

Monitoring is incomplete without effective alerting. Alerts notify relevant teams when predefined thresholds are crossed or anomalies are detected, indicating a potential or actual problem.

  • Actionable Alerts: Alerts should be clear, concise, and actionable. They should tell you what is wrong, where it's wrong, and ideally, what to do about it.
  • Avoid Alert Fatigue: Too many non-actionable alerts lead to alert fatigue, where teams start ignoring warnings.
  • On-Call Rotation: Alerts are typically routed to an on-call rotation.

Rust in Monitoring

Rust's features make it an excellent choice for building monitoring components due to its performance, reliability, and control over system resources.

  • High-Performance Agents/Exporters: Rust can be used to write highly efficient agents that collect metrics with minimal overhead, or custom Prometheus exporters.
  • Reliable Data Collection: Rust's memory safety and strong type system ensure that monitoring components are robust and less likely to crash, guaranteeing continuous data collection.
  • Low-Level System Access: Rust provides the necessary control for interacting with operating system APIs to gather detailed system metrics.
  • Concurrency: Ideal for building concurrent monitoring services that can handle many data streams.
  • Tooling: cargo simplifies dependency management and building.

Rust Monitoring Example (Conceptual: Basic CPU Usage Exporter)

This example shows a very basic Rust program that could expose CPU usage as a Prometheus metric.

1. Setup Cargo.toml:

[package]
name = "rust_cpu_exporter"
version = "0.1.0"
edition = "2021"

[dependencies]
prometheus = "0.13"
lazy_static = "1.4"
sysinfo = "0.29" # For system information
actix-web = "4"
tokio = { version = "1", features = ["full"] }

2. Example Code (src/main.rs)

// src/main.rs
use prometheus::{Encoder, Gauge, Opts, Registry, TextEncoder};
use actix_web::{get, web, App, HttpResponse, HttpServer, Responder};
use std::sync::Arc;
use sysinfo::{System, SystemExt};
use tokio::time::{sleep, Duration};

// Define a custom metric for CPU usage
lazy_static::lazy_static! {
    static ref CPU_USAGE_PERCENT: Gauge =
        Gauge::new("system_cpu_usage_percent", "Current CPU usage percentage").unwrap();
}

// Register metrics with a Prometheus Registry
fn register_metrics(registry: &Registry) {
    registry.register(Box::new(CPU_USAGE_PERCENT.clone())).unwrap();
}

// Function to update CPU usage metric
async fn update_cpu_metric(sys: Arc<Mutex<System>>) {
    loop {
        // Refresh system information
        sys.lock().unwrap().refresh_cpu();
        // Get average CPU usage across all cores
        let cpu_usage = sys.lock().unwrap().global_cpu_info().cpu_usage();
        CPU_USAGE_PERCENT.set(cpu_usage as f64);
        // println!("CPU Usage: {}%", cpu_usage); // For debugging

        sleep(Duration::from_secs(5)).await; // Update every 5 seconds
    }
}

// Endpoint to expose metrics for Prometheus to scrape
#[get("/metrics")]
async fn metrics_handler(registry: web::Data<Arc<Registry>>) -> impl Responder {
    let mut buffer = vec![];
    let encoder = TextEncoder::new();
    let metric_families = registry.gather();
    encoder.encode(&metric_families, &mut buffer).unwrap();

    HttpResponse::Ok().body(buffer)
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    // Create a new Prometheus Registry
    let registry = Arc::new(Registry::new());
    register_metrics(&registry);

    // Create a shared System instance for CPU info
    let sys_arc = Arc::new(Mutex::new(System::new_all()));

    // Spawn a background task to update the CPU metric
    tokio::spawn(update_cpu_metric(sys_arc.clone()));

    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(registry.clone()))
            .service(metrics_handler)
    })
    .bind("127.0.0.1:9000")? // Exporter runs on port 9000
    .run()
    .await
}

To run: cargo run. Then access http://127.0.0.1:9000/metrics to see the exposed CPU usage.

Conclusion

Monitoring is a foundational practice for ensuring the health, performance, and reliability of software systems. By systematically collecting and analyzing data, it provides the necessary visibility to detect issues, understand trends, and make informed operational decisions. While Observability offers deeper insights into unknown unknowns, robust Monitoring remains essential for tracking known behaviors and alerting on deviations. Rust's capabilities make it an excellent choice for building efficient and reliable monitoring components, contributing significantly to the operational excellence of any software system.