feature: Async jobs, real-time web socket gateway

[anonfedora]

Description

Please provide a brief description of the changes in this pull request. Explain what the PR
accomplishes and why it is needed.

Reference Issues

Fixes #214
Fixes #212

Issues Fixed

• Fixes #214: [Backend/Frontend] Implement Asynchronous Background Jobs for Large Frontend Exports
• Fixes #212: [Backend/Frontend] Architect Real-Time WebSocket Gateway for Frontend Data Streaming

---

Changes Made

1. Infrastructure & Dependencies

• Added Redis to docker-compose.yml as the primary message broker and job store.
• Installed bullmq, ioredis, socket.io, and json2csv.
• Created a centralized Redis utility in redis.ts supporting standard connections and Pub/Sub clients.

2. Asynchronous Background Jobs (#214)

• Queue System: Established a export-queue in export.queue.ts.
• Worker Process: Implemented export.worker.ts to handle heavy CSV/JSON generation without blocking the main event loop.
• REST API: Added new endpoints in export.routes.ts for job submission, status tracking, and secure file downloads.
• Auto-Cleanup: Included a recurring cleanup job that purges exported files older than 24 hours.

3. Real-Time WebSocket Gateway (#212)

• Socket.io Integration: Built a secure WebSocket gateway in gateway.ts featuring JWT authentication.
• Redis Pub/Sub Layer: Enabled cross-instance event broadcasting, allowing any part of the backend to push events to the frontend via broadcastEvent.
• Live Updates:
  • Integrated real-time notifications for STUDENT_CREATED events in students.ts.
  • Added automated WebSocket alerts for completed export jobs.

4. Documentation & Maintenance

• Created DOCS_WS_EXPORT.md to guide frontend developers on connecting to the gateway and consuming the export API.
• Resolved various TypeScript/Linter errors in audit.ts to ensure a clean build.

---

How to Test

1. Infrastructure Setup

docker-compose up -d redis
cd backend
npm install
npm run dev

2. Test WebSocket Connection

• Connect a Socket.io client to ws://localhost:8080.
• Provide a valid JWT in the auth.token field.
• Listen for dashboard_updated or user_metrics_updated events.

3. Test Asynchronous Export

• POST /api/v1/export with { "type": "students", "format": "csv" }.
• Capture the jobId.
• GET /api/v1/export/:jobId/status until state is completed.
• Download via the provided downloadUrl.

---

Checklist

• Redis integrated into Docker environment.
• BullMQ workers handling background processing.
• Socket.io gateway with JWT authentication implemented.
• Redis Pub/Sub layer for event broadcasting.
• Documentation provided for frontend integration.
• TypeScript build passes without errors.

[drips-wave]

@anonfedora Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[ayomideadeniran]

Pr under review

[ayomideadeniran]

Pr under review, i will get back to you if i find any wrong implementations.

[ayomideadeniran]

pr under review, if i find any wrong implementation i will notify you.