feat(ai-208): run new AI review logic in shadow mode before broad rollout - #2
Open
khaadish wants to merge 13 commits into
Open
feat(ai-208): run new AI review logic in shadow mode before broad rollout#2khaadish wants to merge 13 commits into
khaadish wants to merge 13 commits into
Conversation
- ScoreCalibrationService decouples raw model confidence from policy thresholds - CalibrationPolicy exposes approveThreshold, rejectThreshold, humanReviewThreshold, biasCorrectionFactor - needsHumanReview=true whenever confidence < humanReviewThreshold (explicit review boundary) - appliedPolicy + rawScore preserved in every output for audit/replay - Corresponding types exported from @devconsole/api-contracts - Spec covers approve/escalate/reject bands, bias correction, clamping, and traceability Closes Ibinola#423
- ShadowModeService runs candidate model without any side-effects - Catches scorer errors so live path is never disrupted - diverged=true emitted as WARN log when candidate and live bands differ - Structured shadow_score log lines for offline metric pipelines - Spec covers divergence detection, error isolation, clamping, and latency fields Closes Ibinola#425
… automation - ModelRolloutService supports pinned / canary / full rollout modes - rollback() restores previous RolloutConfig instantly (single-level undo) - resolveModel() is deterministic per requestId for consistent canary bucketing - setRollout / rollback emit structured log events for operator audit - RolloutConfig/RolloutResolution/ModelRolloutState types in @devconsole/api-contracts - Spec covers all three modes, determinism, rollback, and no-op safety Closes Ibinola#426
…issues, and appeals - CoordinatedAbuseDetectionService analyses batches of AbuseEvents for 4 pattern kinds - VELOCITY_CLUSTER, APPEAL_FLOODING, ISSUE_FARMING, SHARED_METADATA detectors - overallRisk and requiresHumanReview computed from _signalStrength (never automated punishment) - All thresholds in AbuseDetectionPolicy — tunable without code changes - _signalStrength internal; stripped from public API contract types - Structured coordinated_abuse_flagged log for operator pipelines - Spec covers all detectors, clean-events baseline, and immutability Closes Ibinola#427
…, INFRA-214) AI-201 — Versioned prompt and policy registry - Add PromptPolicyEntry schema model (key, version, kind, content, isActive) - NestJS PromptPolicyService: create versions, activate, list by key/kind - PromptPolicyController: POST /prompt-policy, GET active/versions, PATCH activate - api-contracts: PromptPolicyEntrySummary, CreatePromptPolicyPayload, PromptPolicyKind AI-202 — Review context AI preprocessor - ReviewContextPreprocessorService: explicit signal derivation (strong_approval → approval → neutral → changes_requested → rejected), 0–1 confidence score, humanOverrideRecommended flag for low-confidence or ambiguous cases - New endpoint: GET /review-context/pull-requests/:id/ai-ready - All heuristics are explicit, measurable, and described in code comments INFRA-213 — Data-retention lifecycle jobs - RetentionService with three idempotent policies: notifications (30d), dead background jobs (14d), appeal evidenceJson (90d — nulled, record kept) - Dry-run support: ?dryRun=true previews impact without deleting - Audit log on every real run for operator traceability - GET /retention/policies, POST /retention/run INFRA-214 — Feature-flag and config distribution - WaveConfigService: resolves all 7 Wave 5 flags from env vars (WAVE_FLAG_*, WAVE_ROLLOUT_*) - Runtime operator overrides via PATCH /wave-config/flags/:key (in-memory, audit-logged) - Deterministic per-contributor rollout bucketing via isEnabled(key, contributorId) - GET /wave-config/flags, GET /wave-config/flags/:key/check Also: - Register BudgetModule, PromptPolicyModule, RetentionModule, WaveConfigModule in AppModule - Add types for all 4 issues to packages/api-contracts/src/index.ts - Build passes (tsc -p tsconfig.json exits 0) Closes Ibinola#416, Ibinola#417, Ibinola#418, Ibinola#419
…ation feat(ai-206): add policy-aware score calibration for AI appeal outputs
…rols feat(ai-209): add model rollback and safe rollout controls for appeal automation
…abuse feat(ai-210): detect coordinated abuse patterns across contributors, issues, and appeals
…5-issues feat: Wave 5 AI & Infra — prompt registry, review preprocessor, data retention, feature flags
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Creates a shadow-mode path that scores live-like appeal traffic without affecting any outcomes, so model changes can be evaluated safely before promotion.
Changes
ShadowModeService— fire-and-observe shadow scorer inappeal-decisionsmodulediverged=truewhen candidate and live model fall in different policy bandsshadow_scorelog lines for offline metric pipelinesdocs/ai-208-shadow-mode.mdAcceptance Criteria
liveScore,features,candidateScorer) and outputs (diverged,delta,latencyMs) — no hidden heuristicsdivergedandlatencyMsare directly measurable signals for promotion decisionsdiverged=truecases before promotionCloses Ibinola#425