feat(backend): implement standardized errors, redis retry strategy, memory stats, and worker shutdown

[Dopezapha]

Closes #731
Closes #732
Closes #733
Closes #734

PR Description

I resolved four backend issues on the Crucible smart contract testing platform: implementing worker graceful shutdown handlers, standardizing API error response formats, setting up a Redis connection retry strategy, and adding memory metrics tracking.

For issue #731, I exposed the executor and scheduler modules under the workers directory. I built a shutdown coordinator struct using watch channels and join handles to track background worker tasks. I registered SIGINT and SIGTERM signal handles to monitor OS calls and trigger graceful worker shutdowns before the process exits. I also added unit tests verifying that running worker tasks are given time to drain and complete within their grace period.

For issue #732, I created a new errors handler containing a standardized JSON schema with code, message, and timestamp. I routed fallback 404 paths to this schema and updated Authentication, Permission, and Rate Limiting middlewares to return this format instead of plain text. I also updated the main application error converter and the API result contract responses to include chrono Utc timestamps.

For issue #733, I implemented a connection manager retry helper using exponential backoff delay starting at 100ms up to 5s. I configured connection timeouts based on existing Redis parameters. I replaced the standard connection establishment calls at startup in main to prevent service crashes on transient failures.

For issue #734, I updated the system metrics tracker to extract the real Resident Set Size and Heap Allocation segment bytes from the local Linux proc self status file. I exposed the resident memory size to the Prometheus exposition handler endpoints.

Changes Made

• Modified backend/Cargo.toml to configure utoipa swagger ui to compile using the vendored feature for offline environments (resolving build blockers).
• Added a new module backend/src/api/errors.rs defining the standardized ErrorResponse structure and fallback handler containing code, message, and timestamp fields (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/api/mod.rs to expose the new errors module (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/router.rs to wire the standard JSON fallback response to catch all invalid route requests (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/api/middleware/auth.rs to reject unauthenticated calls using the standardized JSON error format (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/api/middleware/permissions.rs to handle unauthorized access and permission failures with the standardized JSON structure (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/api/middleware/rate_limit.rs to reject requests exceeding limits with standard JSON while retaining limit headers (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/error.rs to add standard UTC timestamp fields to the core application error response struct and updated serialization tests (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Modified backend/src/api/contracts.rs to unify ApiErrorResponse and its axum response converter with the standard schema (resolving issue [Backend] Standardize Unified API Error Response Structures #732).
• Added a new module backend/src/db/redis.rs containing the exponential backoff retry connection establishment strategy (resolving issue [Backend] Add Connection Pooling Timeout and Retry Strategy for Redis #733).
• Modified backend/src/db/mod.rs to declare the new redis database module (resolving issue [Backend] Add Connection Pooling Timeout and Retry Strategy for Redis #733).
• Modified backend/src/main.rs to wrap Redis client connection pool startup connections in the retry helper (resolving issue [Backend] Add Connection Pooling Timeout and Retry Strategy for Redis #733).
• Modified backend/src/services/sys_metrics.rs to parse proc self status files for resident memory size and heap allocations, and updated metrics exporter workers and tests (resolving issue [Backend] Enhance System Metrics Collector with Memory Fragmentation Stats #734).
• Modified backend/src/services/metrics.rs to report process resident memory bytes in the Prometheus exposition registry (resolving issue [Backend] Enhance System Metrics Collector with Memory Fragmentation Stats #734).
• Modified backend/src/api/handlers/profiling.rs to extract and output process resident memory bytes in the prometheus handler (resolving issue [Backend] Enhance System Metrics Collector with Memory Fragmentation Stats #734).
• Modified backend/src/workers/mod.rs to declare custom worker scheduler and executor modules, implement a graceful shutdown coordinator with SIGINT and SIGTERM handlers, and added test coverage (resolving issue [Backend] Implement Graceful Shutdown Handlers for Async Workers #731).

Testing

• Checked compilation via cargo check. The custom implementations compile correctly.
• Checked unit tests using cargo test.
  Note: The local test run encountered a pre-existing system linkage error in backend/src/services/audit.rs due to missing pkg config / OpenSSL headers on the host environment, which was left out of scope per instruction.

[drips-wave]

@Dopezapha Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[adams813]

Nice implementation, LGTM!

[github-actions]

🎉 This PR is included in version 1.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀