Skip to content

MASTERY.md

Latest commit

 

History

History
62 lines (47 loc) · 7.38 KB

MASTERY.md

File metadata and controls

62 lines (47 loc) · 7.38 KB
Raw

Mastery Engineering Audit — Cloud Calibration Platform

"Simplicity under pressure = mastery"

Project Summary

A production-quality ISO 17025 calibration management platform for Beamex Oy Ab, written in Go with PostgreSQL. Provides full calibration lifecycle management (draft -> completed -> certified -> expired), an ISO compliance engine that validates ambient conditions and measurement uncertainty, sequential certificate numbering, and an append-only event-sourced audit log.

Failure Mode Analysis

What breaks when the main service goes down?

The API and compliance engine are a single Go binary. Active calibrations in progress (status: draft or running) are not affected in the database — state is in Postgres, not in-process memory. However, any HTTP requests in-flight are dropped without a response. The auto-refreshing web dashboard stops updating. Because calibration is an offline/batch workflow (not a real-time stream), a brief outage is recoverable: operators can resume after restart without data loss.

What breaks when the database/storage slows down?

All reads and writes go through pgxpool. Slow Postgres blocks every API request (no async path exists). The connection pool will exhaust under sustained latency, causing cascading 500 errors to all clients. There is no circuit breaker or degraded-mode response (e.g., serving cached dashboard stats). The append-only event log (calibration_events) is also written synchronously in the same request — a slow event insert blocks the primary write.

What breaks when the network is partitioned?

The platform is a single-region, single-node architecture (Docker Compose). Network partition between the API container and Postgres drops all operations. There is no read replica, cache, or offline capability. Certificate generation and compliance checks fail immediately. The dashboard shows a connection error. This is acceptable for the stated single-deployment target but must be addressed before multi-region deployment.

What breaks under duplicate execution?

The Certify method is explicitly idempotent: it checks rec.Status == StatusCertified and returns the existing certificate rather than creating a new one. Certificate numbers use sequential DB-generated IDs, preventing gaps or duplicates. However, the CompleteRecord + CreateCertificate sequence is not wrapped in a single database transaction — a crash between completion and certificate creation leaves the record as completed without a certificate, requiring manual intervention.

Checklist Status

Check Status Notes
Invariants defined ✅ ISO 17025 rules are explicitly coded: temperature 18-28C, humidity 30-75%RH, deviation <= 2u per measurement point
Idempotency ✅ Certify() checks for existing certificate and returns it — correctly idempotent
Race conditions handled ⚠️ No row-level locking on calibration records; concurrent certification attempts could both pass the status check before either commits
State consistency ⚠️ CompleteRecord + CreateCertificate are separate DB calls without a transaction; partial write possible
Structured logging ⚠️ Not visible from code review — no zap or slog usage found in calibration service; logging strategy unclear
Metrics (not just logs) ❌ No Prometheus metrics endpoint; no counters for certifications issued, compliance failures, or request rates
Distributed tracing ❌ No OpenTelemetry instrumentation; cannot trace a calibration request end-to-end through API -> service -> store
Rollback strategy ⚠️ Migrations exist but no down-migration strategy; rollback requires manual SQL
Safe migrations ⚠️ Migration execution strategy not confirmed from README — no golang-migrate version table visible
10x traffic plan ⚠️ Single Postgres instance; at 10x concurrent calibration events the append-only event log becomes a write hotspot
Bottleneck identified ⚠️ Synchronous event append on every write operation adds latency to every mutation; could be async
Simplicity test passed ✅ Removing the event log, the system still manages calibrations correctly — event log is additive, not load-bearing for core flow

Critical Gaps (Must Fix)

  1. Missing transaction for CompleteRecord + CreateCertificate: these two operations must be atomic. A crash between them leaves a completed record without a certificate — an orphaned state that the compliance engine and dashboard will handle inconsistently.
  2. Race condition on concurrent certification: two simultaneous Certify calls for the same record can both read status != certified, both pass the compliance check, and both call CreateCertificate. Use SELECT ... FOR UPDATE or a DB-level unique constraint on (record_id) in the certificates table.
  3. No metrics: ISO 17025 audit trails require operational traceability. At minimum expose: calibrations_certified_total, compliance_failures_total{reason}, certificates_expired_total, and API request latency.
  4. No structured logging in service layer: the calibration service (service.go) has no logging — compliance failures and certifications are silent at the application log level, making audit reviews impossible without DB queries.
  5. Certificate expiry not enforced: the data model tracks expires_at but there is no background job or API check that transitions certificates from certified to expired — operators must manually track expiry.

What is Already Mastery-Level

  • ISO 17025 compliance engine: the compliance checks are specific, correct, and code-documented. Temperature range (18-28C), humidity range (30-75%RH), and per-point deviation validation (|dev| <= 2u) are all explicitly enforced with descriptive violation messages.
  • Idempotent certification: Certify() correctly returns the existing certificate on re-call instead of creating a duplicate — a rare but important case for ISO audit workflows.
  • Append-only event sourcing: every state transition is recorded in calibration_events as an immutable append — this is the correct pattern for audit-regulated systems.
  • Sequential certificate numbering: CAL-YYYY-NNNNNN format with DB-generated sequential IDs prevents gaps or duplicates in the certificate series.
  • Distroless runtime image: the multi-stage Docker build produces an ~8MB image, reducing attack surface significantly.
  • pgx/v5 connection pooling: pgxpool is used throughout — connections are reused correctly, not opened per-request.

Recommended Next Steps

  1. Wrap CompleteRecord + CreateCertificate in a DB transaction: use pgxpool.BeginTx so the state machine transition is atomic.
  2. Add SELECT FOR UPDATE on certification: prevent race conditions by locking the calibration record row before reading status in Certify().
  3. Certificate expiry job: add a background goroutine that runs daily and transitions certified records past their due_date to expired, emitting a certificate.expired event.
  4. Add structured logging: instrument service.go with zap or slog, logging compliance check outcomes (pass/fail + violations) and certification events with record ID and technician.
  5. Prometheus /metrics endpoint: expose calibrations_total{status}, compliance_checks_total{result}, and request latency histograms — required for any production deployment in a regulated context.