Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
name: Bug report
about: Report a reproducible problem
labels: bug, triage
---

## Summary

## Steps to reproduce

1.
2.
3.

## Expected behavior

## Actual behavior

```text
paste errors/logs
```

## Scope

- [ ] parity runner (`cmd/parity-test`)
- [ ] parity fixtures (`test/fixtures/parity`)
- [ ] benchmark scripts (`scripts/`)
- [ ] docs

## Environment

- Go version:
- OS:
- Command used:
8 changes: 8 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Questions and Discussions
url: https://github.com/go-modkit/benchmarks/discussions
about: Use Discussions for usage questions and open-ended design discussion.
- name: Security vulnerabilities
url: https://github.com/go-modkit/benchmarks/security/policy
about: Report vulnerabilities privately.
20 changes: 20 additions & 0 deletions .github/ISSUE_TEMPLATE/documentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: Documentation issue
about: Report missing, outdated, or unclear docs
labels: docs, triage
---

## Location

<!-- file path or link -->

## Issue type

- [ ] unclear explanation
- [ ] missing information
- [ ] outdated content
- [ ] broken command/example

## Description

## Suggested update
23 changes: 23 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
name: Feature request
about: Propose an enhancement
labels: enhancement, triage
---

## Problem statement

## Proposed solution

## Alternatives considered

## Acceptance criteria

- [ ]
- [ ]

## Affected areas

- [ ] parity runner
- [ ] fixtures
- [ ] benchmark orchestration
- [ ] reporting
19 changes: 19 additions & 0 deletions .github/ISSUE_TEMPLATE/rfc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
name: RFC / design proposal
about: Propose a significant design change
labels: rfc, discussion
---

## Summary

## Motivation

## Detailed design

## Compatibility and migration

## Drawbacks

## Alternatives

## Open questions
34 changes: 34 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
## Type

- [ ] `feat` - New capability
- [ ] `fix` - Bug fix
- [ ] `docs` - Documentation only
- [ ] `test` - Test changes
- [ ] `chore` - Tooling, CI, maintenance

## Summary

<!-- One short paragraph: what changed and why -->

## Changes

-

## Validation

```bash
go test ./...
```

<!-- If parity relevant, include exact command and target used -->

## Checklist

- [ ] Code and docs follow project conventions
- [ ] Tests updated/added for behavior changes
- [ ] Parity contract/fixtures updated if API behavior changed
- [ ] Related design/docs updated if matcher semantics changed

## Resolves

Resolves #
46 changes: 46 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: ci

on:
push:
branches:
- main
pull_request:

jobs:
pr-title:
name: Validate PR title
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: amannn/action-semantic-pull-request@v5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

test:
name: Go tests + coverage
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Run unit tests with coverage
run: go test ./... -coverprofile=coverage.out -covermode=atomic
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
files: ./coverage.out
fail_ci_if_error: false

scripts:
name: Script smoke tests (skipped targets expected)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Run benchmark script smoke
run: bash scripts/run-all.sh
- name: Generate report from raw results
run: python3 scripts/generate-report.py
26 changes: 26 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: codeql

on:
push:
branches:
- main
pull_request:

jobs:
analyze:
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: ["go"]
steps:
- uses: actions/checkout@v5
- uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Generated benchmark artifacts
results/latest/raw/*.json
results/latest/summary.json
results/latest/report.md
82 changes: 82 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# PROJECT KNOWLEDGE BASE

**Generated:** 2026-02-07T17:40:00+02:00
**Commit:** 46f84cd
**Branch:** feat/benchmark-repo-hardening

## OVERVIEW
Benchmark harness for API parity and performance comparison across framework implementations.
Correctness is enforced first (parity), then benchmark scripts generate raw metrics and reports.

## STRUCTURE
```text
benchmarks/
├── cmd/parity-test/ # Go parity runner + matcher/fixture tests
├── test/fixtures/parity/ # Parity seed + endpoint scenarios
├── scripts/ # Parity-gated benchmark and report pipeline
├── docs/ # Architecture + operational guides
├── .github/workflows/ # CI + CodeQL
├── apps/ # Placeholder for framework app implementations
├── results/latest/ # Generated raw metrics, summary, report
├── Makefile
└── docker-compose.yml
```

## WHERE TO LOOK

| Task | Location | Notes |
|------|----------|-------|
| Run parity checks | `Makefile`, `scripts/parity-check.sh` | `PARITY_TARGET=... make parity-check` is canonical |
| Extend parity runner | `cmd/parity-test/main.go`, `cmd/parity-test/main_test.go` | Matcher semantics + fixture validation tests |
| Add/adjust contract cases | `test/fixtures/parity/scenarios/*.json` | Endpoint-grouped fixtures (`health`, `users-*`) |
| Change baseline test state | `test/fixtures/parity/seed.json` | Posted before scenarios when seed endpoint configured |
| Benchmark orchestration | `scripts/run-single.sh`, `scripts/run-all.sh` | Per-target parity gate then benchmark output emit |
| Reporting | `scripts/generate-report.py`, `results/latest/` | Builds `summary.json` and `report.md` from raw JSON |
| CI policy | `.github/workflows/ci.yml`, `.github/workflows/codeql.yml` | Semantic PR title check + Go tests + script smoke + CodeQL |


## CODE MAP
LSP project views unavailable in this environment (`no views`).
Use direct file map; Go entrypoint remains `cmd/parity-test/main.go` (`func main()`).

## CONVENTIONS
- Benchmark scripts are parity-gated per target: benchmark is skipped when health or parity fails.
- Raw benchmark outputs are one JSON file per framework under `results/latest/raw/`.
- Report generation is deterministic from raw artifacts (`summary.json`, `report.md`).
- CI runs three tracks: PR title semantics, Go tests, script/report smoke.

## ANTI-PATTERNS (THIS PROJECT)
- Do not benchmark before parity passes for the target implementation.
- Do not change matcher token semantics without updating fixture expectations and design doc.
- Do not treat generated files in `results/latest/` as hand-authored source-of-truth.

## UNIQUE STYLES
- Parity contract is fixture-first; runner logic is intentionally generic and target-agnostic.
- Scenarios stay endpoint-grouped (`users-*`, `health`) instead of a single aggregate fixture file.
- Benchmark scripts degrade gracefully by writing `skipped` records when targets are unavailable.

## COMMANDS
```bash
# Run full benchmark orchestration
make benchmark

# Generate benchmark report
make report

# Run unit tests
make test

# Run parity checks against a specific service
make parity-check-modkit
make parity-check-nestjs

# Generic parity invocation (set target URL)
PARITY_TARGET=http://localhost:3001 make parity-check

# Direct parity CLI invocation
go run ./cmd/parity-test -target http://localhost:3001 -fixtures test/fixtures/parity
```

## NOTES
- `apps/` is still a placeholder in this branch.
- `results/latest/` is generated output; contents vary between runs and environments.
26 changes: 26 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Code of Conduct

## Our pledge

We are committed to a harassment-free experience for everyone.

## Expected behavior

- be respectful and constructive
- assume good intent
- accept and give feedback professionally

## Unacceptable behavior

- harassment, insults, or discrimination
- doxxing or sharing private information without consent
- disruptive or hostile conduct

## Enforcement

Project maintainers may remove or reject content that violates this policy.

## Attribution

Adapted from Contributor Covenant, version 2.1:
https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
57 changes: 57 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Contributing to benchmarks

Thanks for contributing.

## Scope

This repository validates API parity and runs framework benchmarks. Keep changes focused on one of these areas:

- parity contract (`test/fixtures/parity`)
- parity runner (`cmd/parity-test`)
- benchmark orchestration (`scripts`, `Makefile`, `docker-compose.yml`)
- docs (`docs/`, `README.md`, `METHODOLOGY.md`)

## Prerequisites

- Go 1.25.7+
- Docker + Docker Compose (for local service runs)
- GNU Make

## Local validation

Run these before opening a PR:

```bash
go test ./...
TARGET=http://localhost:3001 bash scripts/parity-check.sh
```

If you changed scripts, also run shell linting if available:

```bash
shellcheck scripts/*.sh
```

## Pull request process

1. Create a branch from `main`.
2. Keep changes atomic and add/update tests when behavior changes.
3. Run local validation commands.
4. Fill out `.github/pull_request_template.md`.
5. Link relevant issues with `Resolves #<number>`.

## Contract rules

- Do not benchmark a framework before parity passes for that target.
- Do not change matcher semantics (`@any_number`, `@is_iso8601`) without updating fixtures and design docs.
- Keep fixture files endpoint-scoped (`users-*`, `health`) instead of creating a single large fixture file.

## Commit style

Use Conventional Commits when possible:

- `feat:` new functionality
- `fix:` bug fix
- `docs:` documentation only
- `test:` tests only
- `chore:` tooling/build/CI
Loading
Loading