feat: add profile quality scorecard with json and markdown output (#14)

.github/workflows/release-guard.yml

@@ -16,4 +16,4 @@ jobs:
       - name: Install dependencies
         run: python -m pip install -r requirements.txt
       - name: Check release metadata
-        run: python3 scripts/check_release_version.py --base origin/${{ github.base_ref }} --strict
+        run: python3 scripts/release_readiness.py --base origin/${{ github.base_ref }} --strict

.github/workflows/validate.yml

@@ -19,5 +19,7 @@ jobs:
         run: make compile
       - name: Validate repository profile distribution
         run: make validate
+      - name: Run profile quality scorecard
+        run: python3 scripts/scorecard.py
       - name: Smoke-test install path
         run: make smoke

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 All notable changes to this Hermes profile distribution are documented here.
 
+## 0.3.1
+
+- Added release readiness checklist workflow.
+
 ## 0.3.0
 
 - Clarified that this repository is a developer authoring system built on top of Hermes Agent's native profile distribution runtime.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: deps validate compile generate-smoke smoke release-check clean
+.PHONY: deps validate compile generate-smoke smoke scorecard test release-check clean
 
 PYTHON ?= python3
 BASE ?= origin/main
@@ -9,6 +9,7 @@ deps:
 
 validate:
 	$(PYTHON) scripts/validate_profile.py .
+	$(PYTHON) -m unittest discover -s tests -p "test_*.py"
 
 compile:
 	PYTHONDONTWRITEBYTECODE=1 $(PYTHON) -m py_compile scripts/*.py
@@ -21,8 +22,14 @@ generate-smoke:
 smoke:
 	scripts/smoke_install.sh
 
+scorecard:
+	$(PYTHON) scripts/scorecard.py
+
+test:
+	$(PYTHON) -m unittest discover -s tests -p "test_*.py"
+
 release-check:
-	$(PYTHON) scripts/check_release_version.py --base $(BASE)
+	$(PYTHON) scripts/release_readiness.py --base $(BASE)
 
 clean:
 	rm -rf $(GEN_ROOT) .pytest_cache .mypy_cache .ruff_cache htmlcov dist build

README.md

@@ -153,19 +153,87 @@ make smoke
 
 The smoke script validates the repository, compiles Python scripts without writing bytecode, generates and validates a profile from `templates/profile.params.yaml`, and installs into a temporary `HERMES_HOME` when the Hermes CLI is available. If you do not use `make`, run `python3 scripts/validate_profile.py .` and `scripts/smoke_install.sh` directly.
 
+## Profile Quality Scorecard
+
+In addition to structural validation, this repository includes a profile quality scorecard that reports release readiness and search/organic discovery alignment.
+
+To generate a scorecard report for your repository:
+
+```bash
+make scorecard
+```
+
+Or execute the script directly:
+
+```bash
+python3 scripts/scorecard.py [path/to/profile]
+```
+
+### Checks & Deductions
+
+The scorecard starts at `100` points and evaluates the repository against 12 criteria grouped by severity:
+
+| Check | Severity | Description | Deduction |
+| :--- | :--- | :--- | :--- |
+| **Required Root Files Presence** | `FAIL` | Checks for mandatory files (`distribution.yaml`, `SOUL.md`, etc.). | `-15` |
+| **Manifest Structure & Fields** | `FAIL` | Validates `distribution.yaml` format and fields. | `-15` |
+| **Environment Variables in Example** | `FAIL` | Ensures `env_requires` variables exist in `.env.EXAMPLE`. | `-15` |
+| **No Committed Runtime/Cache Files** | `FAIL` | Checks that no temporary, cache, or user database files are tracked. | `-15` |
+| **No Exposed API Keys or Secrets** | `FAIL` | Checks for standard API keys or secret structures. | `-15` |
+| **Skill Markdown Frontmatter** | `FAIL` | Checks that skill files under `skills/` contain valid frontmatter. | `-15` |
+| **Python Script Syntax** | `FAIL` | Compiles Python scripts in-memory to catch syntax errors. | `-15` |
+| **License File Presence** | `WARN` | Checks that a `LICENSE` file or metadata field exists. | `-5` |
+| **Install Command in README** | `WARN` | Verifies installation instructions are documented. | `-5` |
+| **GitHub Topic Metadata** | `WARN` | Verifies repository topic suggestions exist. | `-5` |
+| **Changelog Version Consistency** | `WARN` | Verifies `CHANGELOG.md` matches the current version. | `-5` |
+| **Smoke Testing Documentation** | `WARN` | Checks that smoke test instructions are in the README. | `-5` |
+
+### Output Formats
+
+- **Console/Terminal (Default)**: Visual output with colored status labels.
+- **JSON (`--json`)**: Deterministic and structured output for integrations:
+  ```bash
+  python3 scripts/scorecard.py --json
+  ```
+- **Markdown (`--markdown`)**: A formatted table suitable for pull request comments or release summaries:
+  ```bash
+  python3 scripts/scorecard.py --markdown
+  ```
+
+### Exit Codes
+
+- `0`: Scorecard passed with all checks OK, or with advisory warnings (`WARN`) only.
+- `1`: Scorecard failed due to one or more hard/critical errors (`FAIL`).
+- `2`: Command execution error (e.g., path does not exist).
+
+
 
 ## Release discipline
 
-For any change that affects profile behavior, generated files, config, docs, skills, scripts, or distribution metadata:
+For any change that affects profile behavior, generated files, config, docs, skills, scripts, or distribution metadata, a release readiness workflow must be run.
 
-1. Bump `version` in `distribution.yaml`.
+Before opening a pull request or tagging a release:
+
+1. Bump the `version` in `distribution.yaml`.
 2. Add a matching `## <version>` entry to `CHANGELOG.md`.
-3. Run the release guard before opening a pull request:
+3. Run the release readiness check:
 
 ```bash
 make release-check
 ```
 
+This runs `scripts/release_readiness.py`, which checks:
+- **Version Discipline**: Ensures version changes when release-relevant files are modified (relative to a base branch, e.g., `origin/main`).
+- **Changelog Validation**: Verifies that a matching release entry heading exists in `CHANGELOG.md`.
+- **Profile Validation**: Runs standard checks from `validate_profile.py` (formatting, skills, metadata).
+- **No Runtime Files**: Verifies no runtime, temporary, or cache files (e.g., `.env`, `state.db`) are committed.
+- **No Secrets**: Scans the workspace to prevent hardcoded secrets or credentials.
+- **Generator Smoke**: Runs a full template compile and verifies the generated profile passes validation.
+- **Install Smoke**: If Hermes CLI is available, verifies that the profile installs successfully and contains required files.
+- **Install Documentation**: Ensures that the `README.md` or other docs list the correct installation command for this repository.
+
+It generates a detailed Markdown report summarizing the checks, status (PASS, FAIL, SKIPPED), and any remediation hints.
+
 The GitHub Actions release guard enforces this on pull requests.
 
 ## Publish a generated profile

distribution.yaml

@@ -1,5 +1,5 @@
 name: hermes-profile-template
-version: 0.3.0
+version: 0.3.1
 description: "Starter distribution for building custom Hermes Agent profiles with AI assistance."
 hermes_requires: ">=0.12.0"
 author: "CodeGraphTheory"

docs/profile-distribution-contract.md

@@ -20,7 +20,7 @@ This template provides developer-facing authoring and release tooling for creati
 - deterministic YAML-driven generation with `scripts/generate_profile.py`
 - publish-time validation with `scripts/validate_profile.py`
 - isolated install smoke tests with `scripts/smoke_install.sh`
-- release metadata checks with `scripts/check_release_version.py`
+- release readiness checklist with `scripts/release_readiness.py`
 - repeatable GitHub metadata setup with `scripts/apply_github_metadata.py`
 - CI workflows for validation and release hygiene
 - catalog snippets and explicit template lineage metadata