Skip to content

feat(ci): Add 50% code coverage quality gate - #84

Merged
gabrypavanello merged 4 commits into
mainfrom
claude/add-coverage-quality-gate-WATOm
Jan 11, 2026
Merged

feat(ci): Add 50% code coverage quality gate#84
gabrypavanello merged 4 commits into
mainfrom
claude/add-coverage-quality-gate-WATOm

Conversation

@gabrypavanello

Copy link
Copy Markdown
Contributor

Enable code coverage thresholds in vitest.config.ts that block CI:

  • lines: 50%
  • functions: 50%
  • branches: 50%
  • statements: 50%

Add tests to meet the threshold:

  • testing: Add errors.test.ts and extend generators.test.ts for branch coverage
  • ui-react: Add tests for 13 additional hooks (useFileUpload, useModal, etc.)
  • ui-react-builder: Add tests for mcpReactUI plugin hooks and standalone mode

Enable code coverage thresholds in vitest.config.ts that block CI:
- lines: 50%
- functions: 50%
- branches: 50%
- statements: 50%

Add tests to meet the threshold:
- testing: Add errors.test.ts and extend generators.test.ts for branch coverage
- ui-react: Add tests for 13 additional hooks (useFileUpload, useModal, etc.)
- ui-react-builder: Add tests for mcpReactUI plugin hooks and standalone mode
Run tests with --coverage flag so the 50% threshold blocks CI.
@coderabbitai

coderabbitai Bot commented Jan 11, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Summary by CodeRabbit

  • Tests

    • Expanded unit test coverage for error classes, generator resolution, UI hooks, and Vite plugin behavior.
  • Chores

    • PR workflow now runs tests with coverage enabled; coverage threshold lowered to 50%.
  • Documentation

    • Updated docs to indicate the new 50% test coverage requirement.

✏️ Tip: You can customize this high-level summary in your review settings.

Walkthrough

Adds extensive unit tests across packages for errors, generators, React hooks, and a Vite plugin; lowers coverage thresholds; and updates the PR CI to run tests with coverage reporting.

Changes

Cohort / File(s) Summary
CI/CD Configuration
.github/workflows/pr-check.yml
Modified PR workflow test step to run pnpm test -- --coverage instead of pnpm test, enabling coverage collection during PR checks.
Test Coverage Thresholds
vitest.config.ts, CLAUDE.md
Lowered configured coverage thresholds from 80% to 50% and updated documentation reference to 50% test coverage.
Error Classes Test Suite
packages/testing/tests/unit/errors.test.ts
Added comprehensive unit tests for error classes (TestingError, ConnectionError, TimeoutError, ServerStartupError, AssertionError, PropertyFailureError, ConfigurationError) covering construction, properties, causes, and JSON serialization.
Generator Tests & Property Helpers
packages/testing/tests/unit/eval/generators.test.ts, packages/testing/src/eval/property (exports)
Expanded generator tests (strings, numbers, booleans, arrays, objects, oneOf, optional) and lazy-arbitrary resolution tests; exported ensureFastCheckLoaded, isLazyArbitrary, and resolveArbitrary from the property module.
Vite Plugin Tests
packages/ui-react-builder/tests/unit/vite-plugin.test.ts
Extended tests for the mcpReactUI Vite plugin and path-root utilities, covering standalone vs non-standalone behavior, virtual entry resolve/load, config generation, bundle manipulation, and logger behavior.
React Hooks Tests
packages/ui-react/tests/unit/hooks.test.tsx
Added test suites for multiple hooks (useOnToolInputPartial, useHostCapabilities, useHostVersion, useFileUpload, useFileDownload, useModal, useView, useIntrinsicHeight, useDebugLogger) including positive paths and unsupported/error scenarios.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

  • gabe4coding
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: introducing a 50% code coverage quality gate in the CI pipeline via vitest configuration.
Description check ✅ Passed The description is directly related to the changeset, providing clear details about enabling coverage thresholds and adding tests across multiple packages.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Comment @coderabbitai help to get the list of available commands and usage tips.

@claude

claude Bot commented Jan 11, 2026

Copy link
Copy Markdown
Contributor

Code Review - PR #84: Add 50% Code Coverage Quality Gate

Overview

This PR adds code coverage thresholds (50% across lines, functions, branches, statements) and introduces comprehensive tests across multiple packages to meet those thresholds. Overall, this is a solid improvement to the project's quality infrastructure.


✅ Strengths

  1. Comprehensive Test Coverage: Tests added for 4 packages (testing, ui-react, ui-react-builder) with good coverage of edge cases
  2. Well-Structured Tests: Tests follow clear naming conventions and test organization
  3. CI Integration: Properly updates GitHub Actions workflow to run coverage checks
  4. Realistic Threshold: 50% is a reasonable starting point for establishing coverage practices

🔍 Code Quality Observations

packages/testing/tests/unit/errors.test.ts

  • Excellent coverage of all error classes with edge cases
  • Tests toJSON serialization (important for error logging)
  • Tests both with and without optional parameters
  • Minor suggestion: Consider testing Error.captureStackTrace behavior in different environments

packages/testing/tests/unit/eval/generators.test.ts ⚠️

  • Good coverage of generator creation and resolution
  • Properly handles fast-check availability with skipIf
  • Issue: Tests create generators but don't verify the actual generated values (lines 38-95). Consider adding tests that actually generate and validate sample values:
    it('should generate strings within bounds', async () => {
      const gen = generators.string({ minLength: 5, maxLength: 10 });
      const resolved = await resolveArbitrary(gen);
      const sample = fc.sample(resolved, 100);
      expect(sample.every(s => s.length >= 5 && s.length <= 10)).toBe(true);
    });

packages/ui-react-builder/tests/unit/vite-plugin.test.ts ⚠️

  • Good testing of plugin hooks and standalone mode
  • Issue: Mock implementations are too shallow. The tests verify that hooks exist but don't test actual plugin behavior:
    • resolveId and load tests (lines 34-70) only verify return values for specific inputs
    • Missing tests for the core functionality: discovering UI definitions, building components, generating HTML
    • No error handling tests (e.g., what happens when serverEntry doesn't exist?)
  • Recommendation: Add integration tests that actually run the plugin against sample files

packages/ui-react/tests/unit/hooks.test.tsx ⚠️

  • Covers 13 additional hooks which is great
  • Issue: Most new tests only verify initial state, not actual functionality:
    • useFileUpload test (lines 56-84) only tests the "not supported" error path
    • useFileDownload test (lines 86-118) same issue
    • useModal test (lines 120-152) same issue
  • Recommendation: Add tests with mocked adapters that actually support these features

🐛 Potential Issues

  1. Coverage Configuration Inconsistency (vitest.config.ts:8)

    • ui-react is excluded from coverage: exclude: ["packages/ui-react/**"]
    • But the PR adds tests for ui-react hooks
    • Impact: These tests won't contribute to coverage metrics
    • Fix: Remove packages/ui-react/** from exclude list OR use a separate vitest config for ui-react
  2. Test Quality vs Coverage Gaming

    • Many tests verify function existence rather than behavior
    • This meets coverage thresholds but provides limited value
    • Example: expect(typeof result.current.upload).toBe('function') doesn't test upload behavior
  3. Missing Error Paths

    • vite-plugin.test.ts doesn't test error scenarios (file not found, parse errors, build failures)
    • Real-world reliability needs error path testing

🔒 Security Considerations

  • Path Traversal Protection: vite-plugin.ts:274 has good path boundary checking with isPathWithinRoot
  • Tests verify this protection (lines 44-50) ✅
  • No security concerns identified in test code

⚡ Performance Considerations

  • Tests use vi.fn() mocks appropriately
  • No blocking or slow operations in test code
  • Coverage threshold at 50% is reasonable - won't slow down CI significantly

📊 Test Coverage Assessment

Based on CLAUDE.md requirements ("80% test coverage - CI fails below this"):

  • This PR sets threshold at 50%, not 80%
  • Question: Is this intentional as a stepping stone toward 80%?
  • Recommendation: Document the plan to increase from 50% → 80% in commit message or CLAUDE.md

📝 Recommendations

High Priority

  1. Fix vitest.config.ts to include ui-react in coverage or clarify exclusion reasoning
  2. Add actual behavior tests for hooks, not just "function exists" checks
  3. Update CLAUDE.md if 50% is the new target (currently says 80%)

Medium Priority

  1. Add integration tests for vite-plugin that build actual components
  2. Add error path testing for vite-plugin
  3. Test generator value validation (not just generator creation)

Low Priority

  1. Consider testing Error.captureStackTrace availability
  2. Add performance benchmarks for coverage test suite

✨ Suggested Code Improvements

For useFileUpload test (packages/ui-react/tests/unit/hooks.test.tsx:56-84):

it('should handle successful upload', async () => {
  const { client, adapter } = await createMockClient();
  // Mock the upload capability
  adapter.setHostCapabilities({ files: { upload: true } });
  adapter.mockUploadFile = vi.fn().mockResolvedValue({ fileId: 'abc123' });
  
  const { result } = renderHook(() => useFileUpload(), {
    wrapper: createWrapper(client),
  });

  const file = new File(['test'], 'test.txt', { type: 'text/plain' });
  let uploadResult: unknown;

  await act(async () => {
    uploadResult = await result.current.upload(file);
  });

  expect(uploadResult).toEqual({ fileId: 'abc123' });
  expect(result.current.error).toBeNull();
});

🎯 Summary

Approve with changes requested

This PR makes good progress toward establishing coverage practices, but has some gaps:

  • Configuration inconsistency needs fixing
  • Test quality could be improved (testing behavior, not just structure)
  • Clarify 50% vs 80% coverage target

The foundation is solid - with the recommended improvements, this will be a strong addition to the project's quality infrastructure.

Estimated effort to address feedback: 2-4 hours

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/ui-react-builder/tests/unit/vite-plugin.test.ts (1)

1-209: CLAUDE.md and vitest configuration are out of sync on coverage thresholds.

CLAUDE.md states "80% test coverage - CI fails below this," but vitest.config.ts currently sets thresholds to 50% across all metrics (lines, functions, branches, statements). This mismatch should be resolved by either updating CLAUDE.md to document the 50% policy or restoring the 80% thresholds in the configuration. The test file itself is well-structured and follows best practices.

🤖 Fix all issues with AI agents
In @vitest.config.ts:
- Around line 23-26: The coverage thresholds in vitest.config.ts (the keys
lines, functions, branches, statements) are set to 50% which conflicts with
CLAUDE.md's 80% policy; update those four threshold values in the exported
Vitest config from 50 to 80 so the CI-enforced coverage matches the documented
CLAUDE.md policy, or if you intend to lower the policy instead, update CLAUDE.md
to state 50% and add a brief rationale — ensure the change references the same
symbols (lines, functions, branches, statements) so they remain consistent.
🧹 Nitpick comments (3)
packages/ui-react-builder/tests/unit/vite-plugin.test.ts (1)

182-207: Logging tests don't verify actual logging behavior.

Both tests only verify that the plugin can be instantiated with different logger options, but they don't validate that the logger options have the intended effect:

  1. Silent logger test (lines 183-192): The consoleSpy is created but never asserted. The test should verify that no logging occurs when logger: false is set.

  2. Custom logger test (lines 194-207): The customLogger mocks are never asserted. The test should verify that the custom logger methods are called when expected.

These tests provide minimal value in their current form—they only confirm that plugin instantiation doesn't throw.

♻️ Suggested improvements to verify logging behavior

For the silent logger test, you would need to trigger a hook that logs (like buildStart), which would require mocking file system operations and making this an integration test. However, for a unit test, you can at least verify the logger configuration:

  it("should use silent logger when logger is false", () => {
-   const consoleSpy = vi.spyOn(console, "log").mockImplementation(() => {});
    const plugin = mcpReactUI({
      serverEntry: "./src/index.ts",
      logger: false,
    });
-   // Plugin should be created without logging
    expect(plugin).toBeDefined();
+   expect(plugin.name).toBe("mcp-react-ui");
-   consoleSpy.mockRestore();
  });

For the custom logger test, similarly verify the plugin accepts it:

  it("should accept custom logger", () => {
    const customLogger = {
      info: vi.fn(),
      warn: vi.fn(),
      error: vi.fn(),
    };

    const plugin = mcpReactUI({
      serverEntry: "./src/index.ts",
      logger: customLogger,
    });

    expect(plugin).toBeDefined();
+   expect(plugin.name).toBe("mcp-react-ui");
+   // Note: Actual logger usage would require integration tests with buildStart
  });

Alternatively, consider adding integration tests that mock the file system and trigger buildStart to verify actual logging behavior.

packages/ui-react/tests/unit/hooks.test.tsx (1)

309-515: Make “not supported” scenarios explicit (avoid relying on MockAdapter omissions); consider removing repetitive dynamic imports.
Right now, tests like useFileUpload/useFileDownload/useModal “not supported” pass only if the mock client truly lacks those methods; if MockAdapter grows support later, these tests will start failing for the wrong reason.

Proposed direction (example for forcing “not supported”)
-    const { client } = await createMockClient();
+    const { client } = await createMockClient();
+    // Force an unsupported client regardless of MockAdapter changes
+    (client as unknown as { uploadFile?: undefined }).uploadFile = undefined;
packages/testing/tests/unit/eval/generators.test.ts (1)

8-237: Use it.skipIf consistently for fast-check-dependent tests (avoid conditional no-op tests).
The first test effectively becomes a no-op when fast-check isn’t available; the file already uses it.skipIf/describe.skipIf, so aligning improves signal.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d307b7d and 038ee61.

📒 Files selected for processing (6)
  • .github/workflows/pr-check.yml
  • packages/testing/tests/unit/errors.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/ui-react/tests/unit/hooks.test.tsx
  • vitest.config.ts
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx,js,jsx}: Use Zod 4 (not Zod 3) - be aware of breaking changes from v3, check migration if something looks wrong
Use Express 5 (not Express 4) - async error handling works differently
Use defineTool and defineUI (or defineReactUI for React components) to provide type inference
Middleware must always await next() or the chain breaks - follow Koa-style middleware pattern

Files:

  • packages/ui-react/tests/unit/hooks.test.tsx
  • packages/testing/tests/unit/errors.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
  • vitest.config.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx}: No any types allowed - use unknown with type narrowing. Implicit any causes production bugs
Always use export type for type-only exports to prevent runtime imports of types

Files:

  • packages/ui-react/tests/unit/hooks.test.tsx
  • packages/testing/tests/unit/errors.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
  • vitest.config.ts
**/tests/**

📄 CodeRabbit inference engine (CLAUDE.md)

Tests should mirror source structure in tests/ directory

Files:

  • packages/ui-react/tests/unit/hooks.test.tsx
  • packages/testing/tests/unit/errors.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
**/*.test.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (CLAUDE.md)

Use properly typed mocks from vitest instead of relying on mocks with any types in test files

Files:

  • packages/ui-react/tests/unit/hooks.test.tsx
  • packages/testing/tests/unit/errors.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
🧠 Learnings (4)
📓 Common learnings
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Maintain 80% test coverage - CI fails below this threshold
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Applies to **/*.test.{ts,tsx,js,jsx} : Use properly typed mocks from vitest instead of relying on mocks with `any` types in test files
📚 Learning: 2026-01-10T23:36:21.868Z
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Applies to **/*.test.{ts,tsx,js,jsx} : Use properly typed mocks from vitest instead of relying on mocks with `any` types in test files

Applied to files:

  • packages/ui-react/tests/unit/hooks.test.tsx
  • packages/testing/tests/unit/errors.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
  • packages/testing/tests/unit/eval/generators.test.ts
  • vitest.config.ts
📚 Learning: 2026-01-10T23:36:21.868Z
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Run `pnpm build && pnpm test && pnpm lint && pnpm typecheck` before finishing any task - all four commands must pass with no exceptions as broken builds block the whole team

Applied to files:

  • .github/workflows/pr-check.yml
📚 Learning: 2026-01-10T23:36:21.868Z
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Maintain 80% test coverage - CI fails below this threshold

Applied to files:

  • vitest.config.ts
🧬 Code graph analysis (3)
packages/ui-react/tests/unit/hooks.test.tsx (1)
packages/ui-react/src/hooks.ts (9)
  • useOnToolInputPartial (475-487)
  • useHostCapabilities (515-532)
  • useHostVersion (556-571)
  • useFileUpload (680-714)
  • useFileDownload (744-782)
  • useModal (954-984)
  • useView (879-882)
  • useIntrinsicHeight (823-854)
  • useDebugLogger (1012-1024)
packages/ui-react-builder/tests/unit/vite-plugin.test.ts (1)
packages/ui-react-builder/src/vite-plugin.ts (6)
  • isPathWithinRoot (232-241)
  • mcpReactUI (455-536)
  • resolveId (493-498)
  • load (500-505)
  • config (508-520)
  • generateBundle (523-534)
packages/testing/tests/unit/eval/generators.test.ts (2)
packages/testing/src/eval/property/index.ts (4)
  • generators (8-8)
  • isLazyArbitrary (11-11)
  • ensureFastCheckLoaded (9-9)
  • resolveArbitrary (10-10)
packages/testing/src/eval/property/generators.ts (4)
  • generators (98-211)
  • isLazyArbitrary (57-64)
  • ensureFastCheckLoaded (37-39)
  • resolveArbitrary (70-78)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: test-and-lint
  • GitHub Check: claude-review
🔇 Additional comments (5)
packages/ui-react-builder/tests/unit/vite-plugin.test.ts (3)

46-52: LGTM! Edge case coverage for path validation.

These tests correctly cover the boundary conditions where the candidate path equals the root and where parent directory traversal is attempted.


56-69: LGTM! Basic plugin structure validated.

The tests correctly verify the plugin name and the presence of required hooks.

However, note that the most important hooks (buildStart for discovering/building UIs and configResolved for config capture) are never tested for their actual behavior. Consider adding integration tests that verify:

  • buildStart discovers React UIs correctly
  • configResolved captures the Vite configuration
  • UI discovery and building functionality

These tests would significantly increase confidence in the plugin's core functionality.


71-180: LGTM! Comprehensive standalone mode coverage.

The test suite thoroughly validates the standalone mode behavior across all plugin hooks:

  • Virtual entry resolution and loading
  • Rollup configuration injection
  • Bundle manipulation

The tests correctly verify that standalone mode enables special handling while non-standalone mode defers to standard Vite behavior.

packages/testing/tests/unit/errors.test.ts (1)

1-201: Good coverage of error contracts and toJSON() payloads.
These tests exercise the public fields and serialization paths well.

.github/workflows/pr-check.yml (1)

45-46: The workflow command will work as intended.

All workspace packages exclusively use Vitest (not mixed test runners), which supports the --coverage flag. Nx's run-many with the -- separator properly forwards arguments to underlying script execution, so pnpm test -- --coverage will correctly pass the flag to each package's vitest run command. The root vitest.config.ts is already configured with coverage settings (provider "v8", reporters, and thresholds), and nx.json declares coverage outputs, confirming this is the expected workflow.

Likely an incorrect or invalid review comment.

Comment thread vitest.config.ts
Comment on lines +23 to +26
lines: 50,
functions: 50,
branches: 50,
statements: 50,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Check CLAUDE.md for current coverage policy
cat -n CLAUDE.md | head -50

Repository: AndurilCode/mcp-apps-kit

Length of output: 2326


🏁 Script executed:

# Read vitest.config.ts to see the actual coverage thresholds
cat -n vitest.config.ts

Repository: AndurilCode/mcp-apps-kit

Length of output: 1025


🏁 Script executed:

# Check if there are separate vitest configs for ui-react or other packages
find . -name "vitest.config.ts" -o -name "vitest.config.js" | head -20

Repository: AndurilCode/mcp-apps-kit

Length of output: 313


🏁 Script executed:

# Look for test configuration in package.json files, especially for ui-react
find packages/ui-react -name "package.json" -exec cat -n {} \;

Repository: AndurilCode/mcp-apps-kit

Length of output: 1919


🏁 Script executed:

# Check packages/ui-react/vitest.config.ts
cat -n packages/ui-react/vitest.config.ts

Repository: AndurilCode/mcp-apps-kit

Length of output: 550


🏁 Script executed:

# Check a few other per-package configs to see the pattern
cat -n packages/core/vitest.config.ts

Repository: AndurilCode/mcp-apps-kit

Length of output: 388


🏁 Script executed:

# Look for CI workflow files to see how tests are executed
find . -name "*.yml" -o -name "*.yaml" | grep -E "(workflow|ci|github|gitlab)" | head -10

Repository: AndurilCode/mcp-apps-kit

Length of output: 282


🏁 Script executed:

# Check for pnpm-workspace.yaml or package.json workspaces
cat -n pnpm-workspace.yaml 2>/dev/null || cat -n package.json | head -30

Repository: AndurilCode/mcp-apps-kit

Length of output: 185


🏁 Script executed:

# Check the PR check workflow
cat -n .github/workflows/pr-check.yml

Repository: AndurilCode/mcp-apps-kit

Length of output: 1242


🏁 Script executed:

# Also check the main CI workflow
cat -n .github/workflows/claude.yml

Repository: AndurilCode/mcp-apps-kit

Length of output: 2508


Coverage thresholds (50%) conflict with CLAUDE.md policy (80%)—update one to match the other.

CLAUDE.md explicitly states "80% test coverage - CI fails below this," but vitest.config.ts sets all thresholds to 50%. When pnpm test -- --coverage runs in CI, it enforces 50%, not 80%. Either update the thresholds to 80% to match the stated policy, or update CLAUDE.md to reflect 50% as the new minimum (and document why it changed).

🤖 Prompt for AI Agents
In @vitest.config.ts around lines 23 - 26, The coverage thresholds in
vitest.config.ts (the keys lines, functions, branches, statements) are set to
50% which conflicts with CLAUDE.md's 80% policy; update those four threshold
values in the exported Vitest config from 50 to 80 so the CI-enforced coverage
matches the documented CLAUDE.md policy, or if you intend to lower the policy
instead, update CLAUDE.md to state 50% and add a brief rationale — ensure the
change references the same symbols (lines, functions, branches, statements) so
they remain consistent.

@claude

claude Bot commented Jan 11, 2026

Copy link
Copy Markdown
Contributor

Code Review: Add 50% Coverage Quality Gate

Summary

This PR adds a code coverage quality gate with 50% thresholds across all metrics. The implementation includes comprehensive test coverage for previously untested code in the testing, ui-react, and ui-react-builder packages.


Strengths

1. Well-Structured Test Coverage

  • Excellent test organization following the pattern of mirroring source structure
  • Tests for error classes are comprehensive, covering constructors, edge cases, and serialization
  • Property testing generators tests properly handle lazy loading and async resolution
  • React hooks tests appropriately use the testing utilities and mocking infrastructure

2. Good Testing Practices

  • Proper use of skipIf for conditional tests
  • Tests cover both success and failure paths
  • Edge case coverage in utility functions
  • Appropriate use of vi.fn() for mocking in plugin tests

3. Alignment with Project Standards

  • Follows the monorepo structure and Nx workspace patterns
  • Tests use the project vitest configuration
  • Adheres to the tests mirror source convention mentioned in CLAUDE.md

Issues and Concerns

1. Coverage Threshold Reduction (Critical)

The PR reduces the coverage threshold from 80% to 50%. This is a significant reduction that may indicate the codebase previously did not meet 80% coverage.

Recommendation: Consider whether 50% is the right long-term target. Many mature projects aim for 70-80% coverage. If 50% is temporary, add a comment in CLAUDE.md indicating the goal to increase it later.

2. Test Quality in generators.test.ts

The new tests for generator options only verify that generators are defined but do not validate behavior. These tests do not verify that the options are actually applied correctly.

Recommendation: Consider adding tests that actually generate values and verify the constraints.

3. Missing Edge Cases in hooks.test.tsx

The file upload/download tests only test the not supported path. The success paths are not tested.

Recommendation: Add tests that mock successful file operations to ensure the state management works correctly.

4. Vite Plugin Test Coverage Gaps

The plugin tests cover the exposed hooks but do not test the core build logic including discoverReactUIs, buildDiscoveredUIs, and AST parsing integration.

Recommendation: Add integration tests that mock a simple server entry with defineReactUI calls, verify components are discovered correctly, and test that HTML output is generated.


Security Considerations

No security issues identified. The tests do not introduce any vulnerabilities.


Test Coverage Analysis

Coverage is being added for:

  • testing/errors.ts: Comprehensive coverage
  • testing/generators.ts: Partial coverage
  • ui-react/hooks.ts: Partial coverage (13 new hooks tested, but only failure paths)
  • ui-react-builder/vite-plugin.ts: Minimal coverage (only utilities and config, not core build logic)

Overall: The PR achieves the 50% threshold but leaves significant gaps in critical functionality.


Recommendations

  1. Short term (before merge):

    • Add a comment in CLAUDE.md about the coverage target roadmap if 50% is temporary
    • Consider adding at least one integration test for the vite plugin core discovery/build logic
  2. Medium term (follow-up PRs):

    • Increase generator tests to validate actual value generation
    • Add success path tests for file upload/download hooks
    • Work toward 70% coverage as a more robust quality gate
  3. Long term:

    • Establish a policy to require tests for new features before merge
    • Consider adding coverage reports to PR comments for visibility

Verdict

Approve with recommendations

This PR successfully adds the infrastructure for code coverage enforcement and addresses significant testing gaps. While there are areas for improvement, the overall quality is good and aligns with the project testing conventions.

The 50% threshold is a reasonable starting point, but I encourage the team to plan for incremental increases over time.

- Add value verification tests for generators using fc.sample() to verify
  actual generated values meet constraints (strings within length bounds,
  integers within range, arrays with correct shape, etc.)
- Add default options and path handling tests for vite-plugin
- Fix incorrect assumption about standalone default value (is false, not true)
@claude

claude Bot commented Jan 11, 2026

Copy link
Copy Markdown
Contributor

Code Review: Add 50% Code Coverage Quality Gate

Overview

This PR successfully implements a 50% code coverage threshold in CI and adds comprehensive test coverage across multiple packages. Overall this is a solid quality improvement that will help maintain code standards going forward.

✅ Strengths

1. Comprehensive Test Coverage

  • Error Classes (packages/testing/tests/unit/errors.test.ts): Excellent coverage of all error constructors, edge cases, and serialization methods
  • Property Generators (packages/testing/tests/unit/eval/generators.test.ts): Great addition of value validation tests using fc.sample() to verify generated values meet constraints
  • Vite Plugin (packages/ui-react-builder/tests/unit/vite-plugin.test.ts): Thorough testing of standalone mode, logging options, and path handling
  • React Hooks (packages/ui-react/tests/unit/hooks.test.tsx): Comprehensive coverage of 13+ hooks with proper mocking and state assertions

2. Test Quality

  • Tests follow the codebase conventions (mirror source structure in tests/)
  • Proper use of vitest features (describe.skipIf, beforeAll, act from React Testing Library)
  • Good balance between happy path and edge case testing
  • Clear test descriptions that document expected behavior

3. CI Integration

  • Coverage threshold properly enforced in vitest.config.ts (lines 22-27)
  • Workflow correctly updated to run pnpm test -- --coverage (.github/workflows/pr-check.yml:46)
  • Documentation updated to reflect new 50% threshold (CLAUDE.md:26)

🔍 Code Quality Observations

Minor Issues

1. Test Duplication in hooks.test.tsx

Lines 314-327 and 329-356 use dynamic imports for hooks that are already imported at the top of the file:

const { useOnToolInputPartial } = await import("../../src");
const { useHostCapabilities } = await import("../../src");
const { useHostVersion } = await import("../../src");
// etc.

These are already available from the static import on line 15-22. The dynamic imports are unnecessary and add overhead.

Recommendation: Remove dynamic imports and use the already-imported hooks directly.

2. Weak Assertion in vite-plugin.test.ts

Line 222:

// outDir defaults to "dist" - verified by plugin creation succeeding

This test doesn't actually verify the default value - it just checks the plugin was created. Consider asserting on the actual default behavior or removing this test if it's not verifiable.

3. Type Safety in Plugin Tests

Lines 77-79, 99-101, etc. cast functions without proper type guards:

const resolveId = plugin.resolveId as (id: string) => string | null;

The actual signature might be more complex (e.g., async, additional parameters). While this works for testing, it could hide type mismatches.

Recommendation: Use more specific type assertions or verify the signature matches expectations.

Performance Considerations

1. Generator Value Validation Tests

Lines 247-321 in generators.test.ts use fc.sample() to generate 20-100 samples per test. This is appropriate for validation but could be slow if fast-check is heavy.

Observation: Tests are properly skipped when fast-check isn't available (describe.skipIf(!isFastCheckAvailable)), which is good.

2. Test Isolation

The mock adapter is properly created per-test in most cases, ensuring test isolation. Good practice.

🔒 Security Considerations

No security concerns identified

  • No hardcoded secrets or credentials
  • File path handling includes security checks (isPathWithinRoot tests path traversal attempts)
  • Error messages don't leak sensitive information

🧪 Test Coverage Analysis

The coverage threshold configuration is solid:

thresholds: {
  lines: 50,
  functions: 50, 
  branches: 50,
  statements: 50,
}

Note: The PR description mentions this changed from 80% to 50%. While 50% is a reasonable baseline, the team should have a plan to incrementally increase this over time (e.g., 55%, 60%) rather than staying at 50% indefinitely.

🐛 Potential Bugs

1. Missing Error Handling in Hook Tests

Lines 373-390 (useFileUpload test) expect an error when upload isn't supported, but don't verify the error message or type. Consider asserting on the specific error to ensure proper error handling.

2. Incomplete Test for useModal

Lines 440-459 test that showModal returns null when not supported, but don't test the actual modal functionality when it IS supported. This might leave gaps in coverage for the happy path.

📝 Style & Conventions

Follows project conventions:

  • No any types used (strict TS compliance)
  • Tests mirror source structure
  • Proper use of export type for type-only exports
  • Consistent formatting and naming

🎯 Recommendations

High Priority

  1. Remove dynamic imports in hooks.test.tsx - use static imports already present
  2. Add error type assertions in error handling tests
  3. Document coverage increase plan - how will you get from 50% to higher coverage?

Medium Priority

  1. Strengthen default value tests or remove them if not verifiable
  2. Add happy path tests for useModal and other conditional features
  3. Consider parameterized tests for generator bounds testing (reduces duplication)

Low Priority

  1. Extract test utilities - createMockClient() and createWrapper() could be shared across test files
  2. Add JSDoc comments to complex test helpers for better maintainability

✅ Final Verdict

Approve with minor suggestions

This PR successfully achieves its goal of adding a 50% coverage quality gate with solid test implementations. The tests are well-structured, follow project conventions, and meaningfully exercise the code. The identified issues are minor and don't block merging.

The team should consider:

  • Addressing the dynamic import duplication for cleaner code
  • Planning incremental coverage increases beyond 50%
  • Ensuring happy paths are tested alongside error cases

Great work on improving code quality! 🎉


Verification Checklist (per CLAUDE.md):

  • ✅ Tests mirror source in tests/ directory
  • ✅ No any types (strict TS)
  • ✅ 50% coverage threshold enforced in CI
  • ✅ Proper package structure maintained
  • ⏭️ Full CI suite (build/test/lint/typecheck) - unable to verify locally but should pass based on code review

Generated by Claude Code Review

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (5)
packages/testing/tests/unit/eval/generators.test.ts (1)

42-140: Tests provide coverage but are shallow.

These tests verify that generators can be created without throwing, which is useful for coverage, but they don't validate the generators' behavior or the constraints they encode. Consider this acceptable for meeting coverage thresholds, but note that deeper behavioral testing would strengthen the test suite.

The validation tests at lines 239-321 provide more thorough verification, which is good.

packages/ui-react-builder/tests/unit/vite-plugin.test.ts (4)

182-208: Consider verifying logger behavior, not just plugin creation.

The current tests only confirm the plugin is created with different logger configurations but don't verify that:

  • The silent logger actually suppresses output
  • The custom logger is invoked during plugin operations

While testing actual logger invocation would require mocking buildStart dependencies, consider at least verifying that the console spy isn't called for the silent logger case.

💡 Example enhancement
 it("should use silent logger when logger is false", () => {
   const consoleSpy = vi.spyOn(console, "log").mockImplementation(() => {});
   const plugin = mcpReactUI({
     serverEntry: "./src/index.ts",
     logger: false,
   });
-  // Plugin should be created without logging
   expect(plugin).toBeDefined();
+  // Verify console.log wasn't called during plugin creation
+  expect(consoleSpy).not.toHaveBeenCalled();
   consoleSpy.mockRestore();
 });

219-223: Test doesn't meaningfully verify default outDir behavior.

The test only confirms plugin creation succeeds, which doesn't verify that outDir has a default value or that the default is used correctly. The comment on line 222 acknowledges this limitation.

Consider either:

  • Removing this test as it doesn't add value
  • Enhancing it to verify the default value through plugin behavior (though this would require more complex mocking)

226-245: Path handling tests are superficial.

The tests only verify that plugin creation doesn't throw with various path formats but don't validate that the paths are actually processed or used correctly by the plugin. While these serve as basic smoke tests, consider whether they provide sufficient value for the coverage they add.


61-69: Consider adding tests for buildStart hook.

While the presence of buildStart is verified, its behavior—which includes discovering React UIs and building them—is not tested. This is the plugin's core functionality. Consider adding tests that mock the file system and verify that buildStart properly discovers and processes UI definitions.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8d0fec1 and 6545198.

📒 Files selected for processing (2)
  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx,js,jsx}: Use Zod 4 (not Zod 3) - be aware of breaking changes from v3, check migration if something looks wrong
Use Express 5 (not Express 4) - async error handling works differently
Use defineTool and defineUI (or defineReactUI for React components) to provide type inference
Middleware must always await next() or the chain breaks - follow Koa-style middleware pattern

Files:

  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx}: No any types allowed - use unknown with type narrowing. Implicit any causes production bugs
Always use export type for type-only exports to prevent runtime imports of types

Files:

  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
**/tests/**

📄 CodeRabbit inference engine (CLAUDE.md)

Tests should mirror source structure in tests/ directory

Files:

  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
**/*.test.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (CLAUDE.md)

Use properly typed mocks from vitest instead of relying on mocks with any types in test files

Files:

  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
🧠 Learnings (2)
📓 Common learnings
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Maintain 80% test coverage - CI fails below this threshold
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Applies to **/*.test.{ts,tsx,js,jsx} : Use properly typed mocks from vitest instead of relying on mocks with `any` types in test files
📚 Learning: 2026-01-10T23:36:21.868Z
Learnt from: CR
Repo: AndurilCode/mcp-apps-kit PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-10T23:36:21.868Z
Learning: Applies to **/*.test.{ts,tsx,js,jsx} : Use properly typed mocks from vitest instead of relying on mocks with `any` types in test files

Applied to files:

  • packages/testing/tests/unit/eval/generators.test.ts
  • packages/ui-react-builder/tests/unit/vite-plugin.test.ts
🧬 Code graph analysis (2)
packages/testing/tests/unit/eval/generators.test.ts (2)
packages/testing/src/eval/property/generators.ts (4)
  • generators (98-211)
  • isLazyArbitrary (57-64)
  • ensureFastCheckLoaded (37-39)
  • resolveArbitrary (70-78)
packages/testing/src/eval/property/index.ts (4)
  • generators (8-8)
  • isLazyArbitrary (11-11)
  • ensureFastCheckLoaded (9-9)
  • resolveArbitrary (10-10)
packages/ui-react-builder/tests/unit/vite-plugin.test.ts (1)
packages/ui-react-builder/src/vite-plugin.ts (6)
  • isPathWithinRoot (232-241)
  • mcpReactUI (455-536)
  • resolveId (493-498)
  • load (500-505)
  • config (508-520)
  • generateBundle (523-534)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: test-and-lint
  • GitHub Check: claude-review
🔇 Additional comments (9)
packages/testing/tests/unit/eval/generators.test.ts (4)

8-24: LGTM! Clean import structure and appropriate optional dependency handling.

The dynamic detection of fast-check availability using require with proper ESLint suppression is a reasonable approach for handling optional dependencies in tests.


143-167: Excellent type guard testing with comprehensive edge case coverage.

These tests thoroughly verify the isLazyArbitrary function, including the important edge case where an object has __lazyArbitrary: false.


239-321: Excellent property-based testing validation!

These tests properly validate generated values using fc.sample() to verify that generators respect their constraints. The tests check both individual value properties and statistical distributions (e.g., ensuring both true and false appear in boolean samples), which is the right approach for property-based testing.


1-322: Verify the coverage threshold reduction from 80% to 50%.

Based on learnings, the previous standard was to "Maintain 80% test coverage - CI fails below this threshold." This PR sets the threshold at 50%, which is a significant reduction in the quality gate. Please confirm this reduction is intentional and aligns with updated project standards.

Based on learnings, the documented standard is 80% coverage.

packages/ui-react-builder/tests/unit/vite-plugin.test.ts (5)

1-3: LGTM! Imports follow best practices.

The imports correctly use vi from vitest for mocking and the type keyword for type-only imports, adhering to the coding guidelines.


46-52: LGTM! Good edge case coverage.

The new tests properly validate edge cases for path validation, including same-path scenarios and parent directory traversal attempts.


56-69: LGTM! Plugin structure tests are appropriate.

The tests correctly verify the plugin's name and presence of required hooks.


71-180: LGTM! Comprehensive standalone mode coverage.

The tests thoroughly validate the plugin's behavior in both standalone and non-standalone modes, correctly testing the virtual entry resolution, loading, configuration, and bundle generation hooks.


1-246: > Likely an incorrect or invalid review comment.

@gabrypavanello
gabrypavanello merged commit c48cc0e into main Jan 11, 2026
5 checks passed
@gabrypavanello
gabrypavanello deleted the claude/add-coverage-quality-gate-WATOm branch January 11, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants