feat: event-driven lighthouse implementation

[otsybizov]

🤖 Linear

Closes CONG-XXX

[Copilot]

Pull request overview

This PR introduces an event-driven “lighthouse handler” path by pushing Cartographer webhook processing events into BullMQ queues (Redis), plus updates subgraph/webhook deployment tooling and infrastructure to support the new pipelines and networks.

Changes:

• Add a shared BullMQ adapter (@chimera-monorepo/mqclient) and integrate it into Cartographer handler (producer) + Lighthouse handler (workers + repeatable jobs).
• Extend subgraph deployment scripts to auto-detect Goldsky labels and optionally deploy webhook pipelines; add Solana/Tron webhook pipeline templates/configs.
• Provision a dedicated TLS/auth Redis for the lighthouse queue and deploy a new ECS lighthouse-handler service; update CI to build/push its image.

Reviewed changes

Copilot reviewed 62 out of 63 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
yarn.lock	Adds BullMQ/ioredis/msgpack dependencies and workspace wiring for mqclient.
packages/subgraph/src/common/network.ts	Adds chainId mappings for megaeth-mainnet and plasma-mainnet.
packages/subgraph/src/cartographer-webhooks/tron-webhook.template.yaml	New Goldsky pipeline template for Tron dataset → webhook sink.
packages/subgraph/src/cartographer-webhooks/solana-webhook.template.yaml	New Goldsky pipeline template for Solana dataset → webhook sink.
packages/subgraph/scripts/utils.ts	New helper to query Goldsky for latest deployed label and increment patch.
packages/subgraph/scripts/deploy.ts	Adds label auto-detection and optional webhook pipeline deployment (--with-webhooks).
packages/subgraph/scripts/deploy-webhooks.ts	Refactors webhook deploy into an exported function; adds Solana/Tron pipeline support and auto-labeling.
packages/subgraph/package.json	Updates deploy scripts to use --with-webhooks and removes hardcoded labels in many commands.
packages/subgraph/config/everclear-tron-webhooks-staging.json	New Tron webhook pipeline config (staging).
packages/subgraph/config/everclear-tron-webhooks-production.json	New Tron webhook pipeline config (production).
packages/subgraph/config/everclear-spoke-webhooks-staging.json	Updates network keys to megaeth-mainnet / plasma-mainnet.
packages/subgraph/config/everclear-spoke-staging.json	Adds spoke subgraph configs for Plasma and MegaETH (staging).
packages/subgraph/config/everclear-spoke-production.json	Adds spoke subgraph configs for Plasma and MegaETH (production).
packages/subgraph/config/everclear-solana-webhooks-staging.json	New Solana webhook pipeline config (staging).
packages/subgraph/config/everclear-solana-webhooks-production.json	New Solana webhook pipeline config (production).
packages/subgraph/README.md	Documents combined deploy flow, webhook-only deploy flow, and CLI options.
packages/contracts/deployments/staging/9745/SpokeGateway.json	New deployment artifact for Plasma staging.
packages/contracts/deployments/staging/9745/FeeAdapter.json	New deployment artifact for Plasma staging.
packages/contracts/deployments/staging/4326/SpokeGateway.json	New deployment artifact for MegaETH staging.
packages/contracts/deployments/staging/4326/FeeAdapter.json	New deployment artifact for MegaETH staging.
packages/contracts/deployments/production/9745/SpokeGateway.json	New deployment artifact for Plasma production.
packages/contracts/deployments/production/324/FeeAdapter.json	Updates FeeAdapter address for domain 324 in production.
packages/agents/lighthouse/tsconfig.build.json	Adds TS project reference to mqclient.
packages/agents/lighthouse/src/tasks/invoice/processDepositsAndInvoices.ts	Fixes typo/variable naming for asset lookup.
packages/agents/lighthouse/src/tasks/clearing/processExpiredIntents.ts	Fixes docstring typo and adjusts expired intent processing flow.
packages/agents/lighthouse/src/server.ts	New Fastify-based “handler” that runs BullMQ workers + periodic schedulers + admin-trigger endpoints.
packages/agents/lighthouse/src/context.ts	Refactors context init for reuse by handler mode; simplifies logger wiring and solana init.
packages/agents/lighthouse/src/config.ts	Adds handler service type + redisUrl and adminToken fields.
packages/agents/lighthouse/package.json	Adds mqclient + fastify deps and a start:handler script.
packages/agents/cartographer/handler/tsconfig.build.json	Adds TS project reference to mqclient.
packages/agents/cartographer/handler/test/processors/tronLogProcessor.spec.ts	New tests for Tron log topic → queue routing.
packages/agents/cartographer/handler/test/processors/solanaInstructionProcessor.spec.ts	New tests for Solana CPI/event discriminator → queue routing.
packages/agents/cartographer/handler/src/webhooks/webhookHandler.ts	Adds Solana/Tron webhook routing and renames secret verification function arg.
packages/agents/cartographer/handler/src/processors/tronLogProcessor.ts	New processor mapping Tron topics to lighthouse queues.
packages/agents/cartographer/handler/src/processors/solanaInstructionProcessor.ts	New processor decoding base58 instruction data to route events to lighthouse queues.
packages/agents/cartographer/handler/src/processors/monitorProcessor.ts	Notifies lighthouse queues for settlement/fill messages after persistence.
packages/agents/cartographer/handler/src/processors/invoiceProcessor.ts	Notifies lighthouse invoice queue after invoice/deposit processing.
packages/agents/cartographer/handler/src/processors/intentProcessor.ts	Notifies lighthouse intent/fill queues after intent processing.
packages/agents/cartographer/handler/src/notify.ts	New BullMQ producer initialization + enqueue helper with time-bucket job IDs.
packages/agents/cartographer/handler/src/init.ts	Adds optional redisUrl to handler config sourced from REDIS_URL.
packages/agents/cartographer/handler/src/index.ts	Initializes/closes BullMQ notification queues on start/shutdown; fixes timeout naming.
packages/agents/cartographer/handler/package.json	Adds mqclient workspace dep and bs58 dependency.
packages/adapters/mqclient/tsconfig.json	New TS config for mqclient package.
packages/adapters/mqclient/tsconfig.build.json	New composite build config for mqclient package.
packages/adapters/mqclient/src/index.ts	New BullMQ wrapper: queue constants, redis URL parsing, producer/worker factories.
packages/adapters/mqclient/package.json	New workspace package definition for mqclient (BullMQ dependency).
package.json	Adds mqclient workspace to monorepo workspaces.
ops/modules/redis/variables.tf	Adds auth_token and transit_encryption_enabled inputs.
ops/modules/redis/outputs.tf	Updates outputs to support either cluster or replication group.
ops/modules/redis/main.tf	Adds replication-group path for TLS/auth; uses cluster only when TLS disabled.
ops/mainnet/staging/core/variables.tf	Adds lighthouse handler image tag + redis auth token variable.
ops/mainnet/staging/core/outputs.tf	Exposes a rediss:// lighthouse queue redis URL output.
ops/mainnet/staging/core/main.tf	Adds lighthouse-handler ECS service and lighthouse queue Redis module; comments out old lighthouse lambda cron jobs.
ops/mainnet/staging/core/config.tf	Adds env vars for lighthouse-handler, including REDIS_URL.
ops/mainnet/staging/backend/main.tf	Adds remote-state read of core outputs (for REDIS_URL injection).
ops/mainnet/staging/backend/config.tf	Passes core’s lighthouse queue REDIS_URL into cartographer handler env.
ops/env/mainnet/core/secrets.staging.json	Adds encrypted lighthouse_queue_redis_auth_token secret.
docker/lighthouse-handler/Dockerfile	New Docker build for lighthouse handler runtime.
.github/workflows/build-test-deploy.yml	Builds/pushes lighthouse-handler image and deploys staging core with new image tag.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.