Skip to content

Fix long-running memory retention and stalled delivery - #29

Merged
jakthom merged 1 commit into
mainfrom
fix/runtime-resource-lifecycles
Sep 5, 2026
Merged

jakthom merged 1 commit into
mainfrom
fix/runtime-resource-lifecycles

Conversation

@jakthom

@jakthom jakthom commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Long-lived Beacon processes could retain historical device identities and metric labels, slow down once durable queues filled, or stall during connector shutdown and MQTT acknowledgments. This change bounds those lifetimes and keeps unconfirmed MQTT delivery pending for retry.

  • Unsubscribe before draining connector input, flush in batches of at most 64 within one five-second shutdown window, and account for every unpersisted message.
  • Reject oversized MQTT packet lengths before Paho allocates the body, across TCP, TLS, and WebSocket transports. Close retired connections and prevent source callbacks from outliving their owner.
  • Bound CEL evaluation with a 10,000-unit per-expression cost limit, a 50 ms chain deadline, and connector/request cancellation.
  • Limit MQTT CONNECT/SUBSCRIBE acknowledgment waits to 15 seconds and PUBACK waits to 30 seconds. A timeout retires the client generation before durable delivery retries.
  • Cap bus discovery bookkeeping at 1,024 Device NAMEs per endpoint, expire stale entries, and clear discovery state on reconnect.
  • Bound uncommissioned inventory observations even during persistent database write failures; preserve commissioned devices.
  • Find count-retention cutoffs from the oldest excess rows, avoiding a scan across retained history on each saturated append.
  • Delete cumulative metric series with their configured source, sink, or connector. Preserve existing metric names, labels, histogram buckets, and totals across restarts/disables.

Regression coverage exercises interrupted shutdown, stalled brokers, guarded TLS/WebSocket transports, costly filters, 10,000 device identities, persistent inventory failures, and 2,100 deleted metric identities. The vessel gate now uses a saturated retention window and requires evidence of pruning. Operator docs describe the new limits and metric lifecycle.

Validation:

  • go test ./... -timeout 300s
  • go test -race ./... -timeout 300s, plus focused race tests after the final MQTT callback cleanup change
  • go vet ./..., go build ./..., and golangci-lint run ./... (zero issues)
  • npm test (all 7 browser tests)
  • govulncheck ./... (no affected vulnerabilities) and gosec -quiet -exclude-generated -exclude-dir=.claude ./...
  • bash -n scripts/vessel-release-gate.sh and git diff --check
  • Saturated-queue benchmark: approximately 0.37 ms per append/prune with 300,000 retained messages on an Apple M1 Pro, versus approximately 18 ms in the review probe. These are short local measurements, not a multi-day soak.

The Linux CGO-disabled build and vessel RSS/FD/recovery gate run in CI; the local workspace is macOS. MQTT remains at least once: a timed-out acknowledgment can cause a duplicate after broker acceptance. Filters that exceed the new budgets report evaluation errors, and deleting/recreating an entity resets its metric totals.

@jakthom
jakthom merged commit 5708fa5 into main Sep 5, 2026
9 checks passed
@jakthom
jakthom deleted the fix/runtime-resource-lifecycles branch September 5, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant