OMET

OpenMetrics Editing Tool - A command-line utility for reading, modifying, and writing Prometheus/OpenMetrics format data.

Quick Start

# Install
go install github.com/alexkarp-umd/omet@latest

# Basic usage - pipeline mode (default)
echo "5" | omet -f metrics.txt request_count inc > new_metrics.txt

# In-place editing with locking
omet -i -f metrics.txt -l service=api -l env=prod cpu_usage set 75.5

# Check health
omet-healthcheck -f metrics.txt --max-age=300s

Overview

OMET is a lightweight, fast tool for manipulating Prometheus metrics files. Built using the official Prometheus Go libraries, it provides a simple command-line interface for common metric operations like incrementing counters, setting gauges, and adding labels.

Perfect for:

• 🔧 DevOps automation and scripting
• 📊 Metric preprocessing and transformation
• 🧪 Testing and development workflows
• 📈 Custom metric collection pipelines

Why OMET?

Before OMET:

# Complex, error-prone metric manipulation
awk '/^request_count/ {print $1, $2+5}' metrics.txt > temp.txt
# Risk of corrupting metrics format
# No label support
# No type safety

With OMET:

# Simple, safe, and reliable - pipeline mode
echo "5" | omet -f metrics.txt request_count inc > new_metrics.txt

# Or in-place editing with locking
omet -i -f metrics.txt request_count inc 5
# Preserves format
# Full label support  
# Type validation
# Self-monitoring

Features

• ✅ Production Ready: Uses official Prometheus client libraries
• ⚡ High Performance: Process 10K+ metrics in milliseconds
• 🏷️ Rich Labels: Full label support with validation
• 📁 Flexible I/O: Files, stdin/stdout, or pipes
• 🎯 Type Safety: Automatic metric type validation
• 🔍 Health Monitoring: Built-in health check tool
• 📊 Self-Monitoring: Automatic operational metrics
• 🛡️ Error Resilience: Continues operation despite errors
• 🐳 Container Ready: Single binary, no dependencies

Installation

From Source

git clone https://github.com/alexkarp-umd/omet.git
cd omet

# Build main tool
go build -o omet

# Build health check tool
go build -o omet-healthcheck ./cmd/omet-healthcheck

Using Go Install

# Install main tool
go install github.com/alexkarp-umd/omet@latest

# Install health check tool
go install github.com/alexkarp-umd/omet/cmd/omet-healthcheck@latest

Usage

omet [OPTIONS] <metric_name> <operation> [value]

Options

Flag	Description
-f, --file <FILE>	Input metrics file (default: stdin)
-l, --label <KEY=VALUE>	Add label (can be repeated)
-i, --in-place	Edit file in-place (default: write to stdout)
-v, --verbose	Enable verbose logging
-h, --help	Show help

Operations

Operation	Description	Example
inc [VALUE]	Increment counter (default: 1)	omet requests_total inc 5
set <VALUE>	Set gauge value	omet cpu_usage set 85.5
observe <VALUE>	Add histogram observation	omet response_time observe 0.123

Comparison

Feature	Manual Scripts	Prometheus Tools	OMET
Metric Type Safety	❌	✅	✅
Label Support	❌	✅	✅
Pipeline Friendly	⚠️	❌	✅
Single Binary	✅	❌	✅
Health Checking	❌	❌	✅
Self-Monitoring	❌	❌	✅
Error Resilience	❌	⚠️	✅

Examples

Basic Operations

# Pipeline mode (default) - writes to stdout
omet -f metrics.txt request_count inc 1 > new_metrics.txt

# In-place editing - modifies file directly with locking
omet -i -f metrics.txt memory_usage_bytes set 1048576

# Pipeline with labels
omet -f metrics.txt -l region=us-east -l env=prod request_count inc > updated.txt

Pipeline Usage

# Count errors from log file
grep ERROR app.log | wc -l | omet -f metrics.txt -l level=error error_count set > updated.txt

# Monitor queue depth  
echo "42" | omet -f metrics.txt -l queue=processing queue_depth set > queue_metrics.txt

# Process multiple metrics in pipeline
cat raw_metrics.txt | omet -l service=api | omet -l version=v2.1 > processed_metrics.txt

# In-place updates for production
grep ERROR app.log | wc -l | omet -i -f /var/lib/node_exporter/errors.prom -l level=error error_count set

Real-world Scenarios

# DevOps: Update deployment metrics in-place
omet -i -f /var/lib/node_exporter/deploy.prom \
     -l version=$(git rev-parse --short HEAD) \
     deployment_timestamp set $(date +%s)

# Monitoring: Custom business metrics pipeline
curl -s https://api.example.com/stats | jq -r '.active_users' | \
omet -f business_metrics.prom -l product=web active_users_total set > updated_business.prom

# Testing: Generate test data in-place
for i in {1..10}; do
  echo $((RANDOM % 100)) | omet -i -f test_metrics.prom -l instance=server$i cpu_usage set
done

Use Cases

🚀 CI/CD Pipelines

# Track deployment metrics in-place
omet -i -f /shared/metrics.prom \
     -l version=$(git rev-parse --short HEAD) \
     -l environment=$ENV \
     deployments_total inc

📊 Custom Business Metrics

# Daily revenue tracking in-place
curl -s "$API/revenue" | jq -r '.today' | \
omet -i -f business.prom -l date=$(date +%Y-%m-%d) revenue_dollars set

🔍 Log Processing

# Real-time error monitoring in-place
tail -f app.log | grep ERROR | \
while read line; do
  echo "1" | omet -i -f errors.prom -l severity=error error_count inc
done

Input Format

OMET expects standard Prometheus exposition format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total 100
http_requests_total{method="GET",status="200"} 85
http_requests_total{method="POST",status="201"} 15

# HELP memory_usage_bytes Current memory usage
# TYPE memory_usage_bytes gauge
memory_usage_bytes 1048576
memory_usage_bytes{process="app1"} 2097152

Output Format

OMET outputs valid Prometheus exposition format that can be:

• Consumed by Prometheus server
• Processed by other tools in the ecosystem
• Used with node_exporter textfile collector
• Piped to additional OMET commands

Advanced Usage

Chaining Operations

# Multiple transformations in pipeline
cat base_metrics.txt | \
  omet -l datacenter=us-west request_count inc 100 | \
  omet -l environment=production memory_usage set 2048 | \
  omet -l service=web-api response_time observe 0.250 > final_metrics.txt

Integration with Node Exporter

# Generate metrics for node_exporter textfile collector
#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfile"

# Custom application metrics
omet -f /dev/null -l app=myapp -l version=1.2.3 app_uptime set $UPTIME_SECONDS > "$TEXTFILE_DIR/app_metrics.prom"

# Business metrics
curl -s "$API_ENDPOINT/metrics" | \
omet -f /dev/null -l source=api business_revenue set > "$TEXTFILE_DIR/business_metrics.prom"

Automation Scripts

#!/bin/bash
# Update metrics from system stats

# CPU usage
cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
echo "$cpu_usage" | omet -i -f system_metrics.prom -l host=$(hostname) cpu_usage_percent set

# Disk usage  
disk_usage=$(df -h / | awk 'NR==2 {print $5}' | cut -d'%' -f1)
echo "$disk_usage" | omet -i -f system_metrics.prom -l host=$(hostname) -l mount=root disk_usage_percent set

# Active connections
active_conns=$(netstat -an | grep ESTABLISHED | wc -l)
echo "$active_conns" | omet -i -f system_metrics.prom -l host=$(hostname) network_connections_active set

Docker Usage

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o omet
RUN go build -o omet-healthcheck ./cmd/omet-healthcheck

FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /app/omet /usr/local/bin/
COPY --from=builder /app/omet-healthcheck /usr/local/bin/

# Use in containers - in-place editing
docker run --rm -v /metrics:/data myapp/omet \
  -i -f /data/metrics.prom request_count inc 1

# Use in containers - pipeline mode
docker run --rm -v /metrics:/data myapp/omet \
  -f /data/metrics.prom request_count inc 1 > /data/new_metrics.prom

Metric Types

Counters

• Always increment (never decrease)
• Automatically resets to 0 when process restarts
• Good for: request counts, error counts, bytes transferred

# Pipeline mode
omet -f metrics.txt requests_total inc 1 > updated.txt

# In-place mode  
omet -i -f metrics.txt -l status=error errors_total inc

Gauges

• Can go up or down
• Represents current state
• Good for: CPU usage, memory usage, queue length

# Pipeline mode
omet -f metrics.txt cpu_usage_percent set 75.5 > updated.txt

# In-place mode
omet -i -f metrics.txt -l queue=jobs queue_length set 42

Histograms

• Track distributions of values
• Automatically creates buckets, count, and sum
• Good for: response times, request sizes

# Pipeline mode
omet -f metrics.txt response_time_seconds observe 0.123 > updated.txt

# In-place mode
omet -i -f metrics.txt response_time_seconds observe 0.123

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

Development Setup

# Clone the repository
git clone https://github.com/alexkarp-umd/omet.git
cd omet

# Install dependencies
go mod download

# Run tests
go test ./...

# Build
go build -o omet

Running Tests

# Unit tests
go test -v

# Integration tests with sample data
./test/integration_test.sh

# Benchmark tests
go test -bench=.

Performance

OMET is designed for high performance:

• Memory efficient: Streams data without loading entire files
• Fast parsing: Uses optimized Prometheus libraries
• Low overhead: Single binary with minimal dependencies
• Concurrent safe: Can be used in parallel pipelines

Benchmarks on a MacBook Pro M1:

• Parse 10K metrics: ~2ms
• Transform 10K metrics: ~5ms
• Memory usage: <10MB

Health Checking

OMET includes omet-healthcheck - a fast, reliable health check tool for monitoring OMET-generated metrics files.

Usage

# Basic health check
omet-healthcheck /shared/metrics.prom

# Check if metrics were written recently
omet-healthcheck /shared/metrics.prom --max-age=300s

# Check consecutive error count
omet-healthcheck /shared/metrics.prom --max-consecutive-errors=10

# Check if specific metric exists
omet-healthcheck /shared/metrics.prom --metric-exists=omet_last_write

# Multiple checks (all must pass)
omet-healthcheck /shared/metrics.prom --max-age=300s --max-consecutive-errors=5

# JSON output for structured logging
omet-healthcheck /shared/metrics.prom --json --max-age=300s

Exit Codes

• 0 = Healthy (all checks passed)
• 1 = Unhealthy (one or more checks failed)
• 2 = Error (file not found, parse error, etc.)

Kubernetes Integration

Perfect for Kubernetes liveness and readiness probes:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: metrics-collector
    image: myapp:latest
    livenessProbe:
      exec:
        command: ["/usr/local/bin/omet-healthcheck", "/shared/metrics.prom", "--max-consecutive-errors=10"]
      initialDelaySeconds: 60
      periodSeconds: 60
      timeoutSeconds: 10
      failureThreshold: 3
    readinessProbe:
      exec:
        command: ["/usr/local/bin/omet-healthcheck", "/shared/metrics.prom", "--max-age=300s"]
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 5
      failureThreshold: 2

Health Check Options

Flag	Description	Example
--max-age	Maximum age since last write	--max-age=300s
--max-consecutive-errors	Maximum consecutive errors allowed	--max-consecutive-errors=10
--metric-exists	Check that specific metric exists	--metric-exists=omet_last_write
--json	Output results in JSON format	--json
--verbose	Enable verbose output	--verbose

JSON Output Format

{
  "healthy": false,
  "checks": {
    "max_age": {
      "passed": false,
      "message": "Last write too old: 10m0s (max: 5m0s)",
      "value": "10m0s"
    },
    "consecutive_errors": {
      "passed": true,
      "message": "Consecutive errors OK: 0 (max: 5)",
      "value": "0"
    }
  },
  "last_write_timestamp": 1234567890,
  "consecutive_errors": 0,
  "metrics_found": ["omet_last_write", "test_counter"]
}

Troubleshooting

Common Issues

Q: Getting "unknown operation" error

Error: unknown operation: increment (supported: inc, set, observe)

A: Use inc instead of increment. Run omet --help for valid operations.

Q: Metric type mismatch

Error: metric requests_total is not a gauge (type: COUNTER)

A: You're trying to set a counter. Use inc for counters, set for gauges.

Q: Label format error

Error: invalid label format: env:prod (expected KEY=VALUE)

A: Use = instead of :. Correct format: -l env=prod

Q: Health check failing

UNHEALTHY - max_age: Last write too old: 10m0s (max: 5m0s)

A: OMET hasn't written metrics recently. Check if OMET process is running and has write permissions.

Performance Issues

# Check processing time
omet -v -f large_metrics.txt test_metric inc 1
# Look for: "Processing took: XXXms"

# Monitor self-metrics
grep "omet_process_duration" metrics.txt

Health Check Debugging

# Verbose health checking
omet-healthcheck -v -f metrics.txt --max-age=300s

# Check what metrics exist
omet-healthcheck -v -f metrics.txt --metric-exists=nonexistent
# Shows: "Available metrics: [list]"

Debug Mode

Use -v flag for verbose output:

omet -v -f metrics.txt -l env=prod request_count inc 1
omet-healthcheck -v /shared/metrics.prom --max-age=300s

License

MIT License - see LICENSE file for details.

Acknowledgments

• Built with Prometheus Go libraries
• CLI powered by urfave/cli
• Inspired by the Prometheus ecosystem

---

Made with ❤️ for the observability community