fix(#610): remove orphaned EventProcessor; add DuplicateStats to idempotency#682
Open
playmaker410 wants to merge 478 commits into
Open
fix(#610): remove orphaned EventProcessor; add DuplicateStats to idempotency#682playmaker410 wants to merge 478 commits into
playmaker410 wants to merge 478 commits into
Conversation
…ons-route-242 chore(api): add env vars for positions route to .env.example
…sma-schema-243 test(api): add Vitest test for Prisma schema
…validator-type-244 feat(api): add TypeScript type for migration validator
…e/onboarding-tasks Feature/onboarding tasks
…-env-example chore: add Redis env vars to .env.example
…ve-error-handler-log fix: improve error handler log message with structured fields
…pi-input-validation feat: add input validation to OpenAPI stub
…ent-rate-limiter docs: document rate limiter in docs/rate-limiting.md
…sor-env-example docs: add indexer cursor env example
…atix-Protocol#349 - Vatix-Protocol#346 [api]: Add Vitest test for docker-compose.yml Added tests/docker-compose.test.ts that validates the compose file exists, defines postgres and redis services, exposes the expected ports, and declares named volumes for persistence. - Vatix-Protocol#347 [indexer]: Add TypeScript type for metrics log Added IndexerMetricsLog interface to apps/indexer/src/metrics.ts and used it with a 'satisfies' check in main.ts when logging the startup metrics snapshot. No 'any' in new code; exported from metrics module. - Vatix-Protocol#348 [oracle]: Improve log messages in submission queue Replaced all console.log/warn/error calls in oracle-service.ts with structured Logger calls (info/warn/error). Logger is injected via OracleServiceConfig.logger (optional, defaults to no-op) so existing tests require no changes. Each log entry carries structured fields (marketId, source, error, attempt, delayMs) at the appropriate level. - Vatix-Protocol#349 [workers]: Add input validation to finalization job Added FinalizationValidationError (statusCode=400) and a guard at the top of FinalizationJob.run() that throws it when challengeWindowSeconds is negative, NaN, or Infinity. Added job.test.ts covering the invalid and valid input cases.
…-api-indexer-oracle-workers-improvements feat(api,indexer,oracle,workers): address issues Vatix-Protocol#346-Vatix-Protocol#349
…env-error-handler-comment [topup] [api] Add env var to .env.example for error handler (2)
…atix-Protocol#299 Vatix-Protocol#344 - Vatix-Protocol#297: Add input validation to test utilities - Add validateDatabaseUrl() function with comprehensive input validation - Return false for invalid inputs (non-strings, empty, invalid URLs, wrong schemes) - Validate parameters in cleanDatabase() - Add 13 test cases covering all validation scenarios - Vatix-Protocol#298: Document logger in docs/ - Create logger.md with complete Logger API documentation - Include usage examples for routes, workers, and databases - Document log levels, environment configuration, and best practices - Add link from README - Vatix-Protocol#299: Add env var for config types to .env.example - Add Configuration Types Reference section - Document REQUIRED vs OPTIONAL distinction - Explain Enum, Integer, URL, and String type validation - Add missing env vars: ORACLE_CHALLENGE_WINDOW_SECONDS, ORACLE_LOG_LEVEL, LOG_LEVEL - Vatix-Protocol#344: Document graceful shutdown in docs/ - Create graceful-shutdown.md with comprehensive guide - Include signal handling, worker pattern, and resource cleanup - Add timeout protection and testing strategies - Cover environment-specific considerations (Docker, Kubernetes) - Add link from README
…xample chore: add comprehensive env var documentation to .env.example (Vatix-Protocol#230)
…-queue-migration feat(workers): migrate oracle and settlement consumers to BullMQ (Vatix-Protocol#551)
…idate-position-routes feat: add GET /v1/wallets/:wallet/positions/:marketId sub-route (Vatix-Protocol#556)
…openapi-docs feat: mount interactive OpenAPI docs at /docs (Vatix-Protocol#557)
The /v1/ready endpoint was missing response body schemas for both 200 and 503 responses. Added DependencyResult component schema and wired it into the readiness endpoint responses.
redisConnectionFromEnv() returned { url } which BullMQ does not
accept as a connection option. Parse the URL into { host, port,
password } so BullMQ workers can connect properly.
…ring Trade.outcome was declared as String @db.VarChar(8) which bypasses Prisma's enum validation, unlike Order.outcome which correctly uses the Outcome enum. Align both models to use the same type.
Audit stream keys were hardcoded without REDIS_KEY_PREFIX, unlike the settlement queue. This caused cross-environment contamination in shared Redis deployments. Read prefix from env at construction.
…arison The resolution parser in the workers scope was using `outcome === true` which is redundant for boolean values and fragile if outcome type changes.
…orker" The oracle queue nack() was using a hardcoded "nack-worker" consumer name in xclaim, causing retried messages to be assigned to a phantom consumer that never processes them. Messages would get stuck instead of being retried.
…Failure
Both methods had empty data: {} in their Prisma calls, making them no-ops.
updateAttempt now degrades confidence on retry. updateOnFailure now clears
candidateResolution and marks the ResolutionCandidate as REJECTED.
…-issues Fix/four minimal issues
…-routes Feat/add scope routes
…itions Fix/scope additions
…pe-issues Fix/multi scope issues
…-resolution-oracle-submission Fix/minimal tasks resolution oracle submission
…eStats to idempotency The in-memory EventProcessor in src/services/event-processor.ts was never imported or wired into any production code path. Deduplication is already enforced at the DB boundary via IndexerProcessedEvent and the idempotency helpers in apps/indexer/src/idempotency.ts. Changes: - Delete src/services/event-processor.ts and its companion test. The class held an unbounded in-memory Set<string> of seen event IDs (memory leak), used console.warn/error instead of structured logging, and wrapped the legacy matching-engine Trade type rather than the chain-indexer RawChainEvent type. - Add DuplicateStats class to idempotency.ts. Provides the same observability surface (totalInserted, totalDuplicates, toLogFields, reset) that EventProcessor offered, but without holding any IDs in memory — the DB is the source of truth. - Add 12 new unit tests for DuplicateStats covering: zero state, record(inserted), record(duplicate), accumulated counts, recordBatch, toLogFields snapshot, reset, and integration with insertIfNew / insertAllIfNew. Closes Vatix-Protocol#610
Contributor
|
@playmaker410 fix pr and link issues |
- Add private duplicateStats field to PollingIngestionLoop - Accumulate written/skipped counts from each BatchWriteResult via recordBatch() after every batchWriter.write() call - Emit toLogFields() spread in heartbeat log (indexer.heartbeat) - Emit toLogFields() spread in stop() log - Add 3 new ingestion tests: accumulation across batches, heartbeat log includes totalInserted/totalDuplicates, stop() log includes totalInserted/totalDuplicates Closes Vatix-Protocol#610
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #610
Problem
src/services/event-processor.tscontained anEventProcessorclass that was never imported or used anywhere in the production code path. It had three design issues:Set<string>of every seen event ID, growing forever with the process lifetime.console.warn/console.errorinstead of the structuredILoggerused everywhere else in the codebase.IndexerEventinterface wrapped the legacy matching-engineTradetype, notRawChainEventused by the real indexer.Deduplication is already fully enforced at the DB boundary via
IndexerProcessedEvent+apps/indexer/src/idempotency.ts.Changes
src/services/event-processor.tssrc/services/event-processor.test.tsapps/indexer/src/idempotency.tsDuplicateStatsclassapps/indexer/src/idempotency.test.tsDuplicateStatsapps/indexer/src/ingestion.tsDuplicateStatsintoPollingIngestionLoopapps/indexer/src/ingestion.test.tsCHANGELOG.mdDuplicateStats — integrated into the ingestion loop
DuplicateStatsreplaces the observability surfaceEventProcessoroffered (totalInserted,totalDuplicates,toLogFields(),reset()) but:PollingIngestionLoopas a process-lifetime counter:written/skippedcounts from eachBatchWriteResultafter everybatchWriter.write()call.totalInsertedandtotalDuplicatesin everyindexer.heartbeatlog.totalInsertedandtotalDuplicatesin the finalstop()log.Tests
15 new unit tests total:
idempotency.test.ts: zero state,record(inserted),record(duplicate), accumulated counts,recordBatch(),toLogFields()snapshot,reset(), integration withinsertIfNew/insertAllIfNew.ingestion.test.ts: accumulation across batches, heartbeat log includes stats, stop() log includes stats.No regressions
EventProcessorexisted — this is a pure dead-code removal plus stats wiring.idempotency.test.ts,batchWriter.test.ts, andingestion.test.tssuites are unaffected.