feat: CI dev/prod parity, fix flaky builds, upgrade Node to 26 - #34
Merged
Conversation
## Problem
CI builds were flaky — the Docker smoke test would intermittently fail
because the Hermes Dashboard crashed on startup trying to rebuild the web
UI inside the container. This caused port 9119 (socat→dashboard) to never
respond, making the self-check exit with code 2.
## Root Cause
The dashboard's startup sequence runs a web UI staleness check and tries
to when it detects the dist is missing or
outdated. In CI, this npm install fails because the container doesn't
have the full dev dependencies installed at runtime — the web UI is
already built during the Docker image build step.
## Fix
- Pass to → Building web UI...
npm notice run web@0.0.0 build
npm notice run tsc -b && vite build
(!) Your Vite config uses features that are unsupported by `configLoader: 'native'`, which is planned to become the default in a future major version of Vite:
- `__dirname` (vite.config.ts:64:25). Use `import.meta.dirname` instead
Set `VITE_CONFIG_NATIVE_IGNORE_WARNING=true` to suppress this warning.
vite v8.2.0 building client environment for production...
�[2K
transforming...✓ 2216 modules transformed.
rendering chunks...
computing gzip size...
../hermes_cli/web_dist/index.html 1.47 kB │ gzip: 0.53 kB
../hermes_cli/web_dist/assets/Mondwest-Regular-CWscgue7.woff2 23.82 kB
../hermes_cli/web_dist/assets/RulesExpanded-Regular-l8uVympt.woff2 33.82 kB
../hermes_cli/web_dist/assets/RulesExpanded-Bold-DZA7s8Pa.woff2 35.01 kB
../hermes_cli/web_dist/assets/RulesCompressed-Regular-BSXFyF4x.woff2 37.76 kB
../hermes_cli/web_dist/assets/RulesCompressed-Medium-CA76_CrB.woff2 38.84 kB
../hermes_cli/web_dist/assets/Collapse-Bold-mgICk9-_.woff2 59.14 kB
../hermes_cli/web_dist/assets/Collapse-Regular-DysayoTY.woff2 62.81 kB
../hermes_cli/web_dist/assets/xterm-BrP-ENHg.css 3.93 kB │ gzip: 1.01 kB
../hermes_cli/web_dist/assets/index-mXYHKWVT.css 112.59 kB │ gzip: 17.61 kB
../hermes_cli/web_dist/assets/chat-activation-BiaqYo9D.js 0.04 kB │ gzip: 0.06 kB
../hermes_cli/web_dist/assets/page-header-context-BX4nlssA.js 0.15 kB │ gzip: 0.14 kB
../hermes_cli/web_dist/assets/useProfileScope-D4PhX9DW.js 0.26 kB │ gzip: 0.20 kB
../hermes_cli/web_dist/assets/usePageHeader-Ct7hu-9P.js 0.30 kB │ gzip: 0.23 kB
../hermes_cli/web_dist/assets/DeleteConfirmDialog-CsLNZYN8.js 0.42 kB │ gzip: 0.26 kB
../hermes_cli/web_dist/assets/useModalBehavior-BzfiHIf6.js 0.52 kB │ gzip: 0.33 kB
../hermes_cli/web_dist/assets/utils-CwZgPRDJ.js 0.53 kB │ gzip: 0.29 kB
../hermes_cli/web_dist/assets/rolldown-runtime-CbXtAM7H.js 0.58 kB │ gzip: 0.36 kB
../hermes_cli/web_dist/assets/mcp-server-create-B0VgoYzF.js 0.92 kB │ gzip: 0.46 kB
../hermes_cli/web_dist/assets/DocsPage-6AuSIQ8w.js 1.40 kB │ gzip: 0.83 kB
../hermes_cli/web_dist/assets/plugins-Bhfqz2DM.js 4.45 kB │ gzip: 2.09 kB
../hermes_cli/web_dist/assets/LogsPage-B-K9EZla.js 4.75 kB │ gzip: 2.00 kB
../hermes_cli/web_dist/assets/PairingPage-CgKs7_Hh.js 4.79 kB │ gzip: 1.64 kB
../hermes_cli/web_dist/assets/FilesPage-MfyaX1PW.js 9.80 kB │ gzip: 3.25 kB
../hermes_cli/web_dist/assets/ModelReloadConfirm-COUTk7QY.js 12.06 kB │ gzip: 4.35 kB
../hermes_cli/web_dist/assets/WebhooksPage-BWio2WVh.js 12.40 kB │ gzip: 3.72 kB
../hermes_cli/web_dist/assets/themes-BXlzFeo6.js 12.79 kB │ gzip: 4.19 kB
../hermes_cli/web_dist/assets/AnalyticsPage-zQ-l7u1o.js 13.59 kB │ gzip: 3.54 kB
../hermes_cli/web_dist/assets/ConfigPage-C-z7kjKT.js 14.68 kB │ gzip: 4.57 kB
../hermes_cli/web_dist/assets/ProfileBuilderPage-DsdwS39T.js 14.69 kB │ gzip: 4.38 kB
../hermes_cli/web_dist/assets/McpPage-DhgLzzBB.js 17.15 kB │ gzip: 4.99 kB
../hermes_cli/web_dist/assets/PluginsPage-HMKOdBmV.js 20.44 kB │ gzip: 5.66 kB
../hermes_cli/web_dist/assets/api-CUjvYA_E.js 21.73 kB │ gzip: 4.79 kB
../hermes_cli/web_dist/assets/ProfilesPage-DHk5RLOd.js 23.07 kB │ gzip: 6.45 kB
../hermes_cli/web_dist/assets/ModelsPage-BK7RmKPj.js 27.48 kB │ gzip: 7.40 kB
../hermes_cli/web_dist/assets/ChannelsPage-BGZijnAM.js 28.72 kB │ gzip: 7.94 kB
../hermes_cli/web_dist/assets/EnvPage-BRX6V_Z1.js 29.97 kB │ gzip: 8.14 kB
../hermes_cli/web_dist/assets/CronPage-DxeZaUwM.js 32.54 kB │ gzip: 9.24 kB
../hermes_cli/web_dist/assets/SkillsPage-B5DCnGsm.js 39.61 kB │ gzip: 10.57 kB
../hermes_cli/web_dist/assets/SessionsPage-BlWPuMwG.js 40.63 kB │ gzip: 11.87 kB
../hermes_cli/web_dist/assets/SystemPage-CNaFRAJr.js 40.66 kB │ gzip: 10.92 kB
../hermes_cli/web_dist/assets/ChatPage-C9tKnV9G.js 42.66 kB │ gzip: 14.34 kB
../hermes_cli/web_dist/assets/index-DbM3nk0u.js 44.94 kB │ gzip: 13.54 kB
../hermes_cli/web_dist/assets/vendor-BLReI8FQ.js 50.06 kB │ gzip: 17.82 kB
../hermes_cli/web_dist/assets/react-vendor-B6GYCG81.js 226.82 kB │ gzip: 72.67 kB
../hermes_cli/web_dist/assets/ui-Cc7bRenc.js 290.01 kB │ gzip: 94.82 kB
../hermes_cli/web_dist/assets/xterm-CXxU4Y2B.js 474.38 kB │ gzip: 122.64 kB
../hermes_cli/web_dist/assets/i18n-XsAPqR2h.js 477.54 kB │ gzip: 140.64 kB
✓ built in 460ms
✓ Web UI built in so
it serves the pre-built dist without trying to rebuild on startup
- This eliminates the race condition where socat binds to 9119 before
the dashboard is listening
## CI Workflow Improvements (Dev/Prod Parity)
- Start the test container via PUID=501 \
PGID=20 \
docker compose up -d (with ) instead
of a raw — this mirrors how the container runs in
production (same docker-compose, same env vars, same init scripts)
- Run self-check as user (the container's unprivileged user)
instead of root, matching production behavior
- Capture only on failure (not always), avoiding
noisy output on passing runs
- Allow env var to suppress in
the Makefile, preventing CI from hanging on log follow
## Other Changes
- Upgrade Node.js from 24.18.0 to 26.7.0 in Dockerfile
- Set to `/config` during container initialization so
Hermes starts in the home directory by default
- Update omniroute combo models to include opencode-zen variants
gitricko
force-pushed
the
gitricko-ci-devprod-parity
branch
from
August 16, 2026 03:44
fce9830 to
9808304
Compare
added 12 commits
August 16, 2026 03:50
PR #33 introduced a web UI build stamp persistence mechanism to fix CI flakiness, but the stamp is never actually used in Docker contexts: 1. Dockerfile wrote a build stamp to /opt/hermes-prebuilt/ via a Python script calling _write_web_ui_build_stamp() 2. 01-copy-web-ui-stamp.sh was supposed to copy the stamp from the build-time path to at container startup 3. But with --skip-build in start-hermes.sh, the staleness check is bypassed entirely — the stamp is never read This is dead code: the stamp is written, never copied (the init script runs but the stamp check is skipped), and never affects behavior. Removed: - Python stamp-writing block from Dockerfile (lines 62-71) - mkdir -p /opt/hermes-prebuilt from Dockerfile (line 56) - docker/01-copy-web-ui-stamp.sh (dead init script) The --skip-build flag in start-hermes.sh is the actual fix that prevents unnecessary rebuild attempts in CI/Docker contexts.
The CI workflow had a 150-second polling loop waiting for port 9119 before running self-check. This is redundant because self-check.sh already has its own 120-second polling loop that checks all 5 service ports (3000, 8888, 7352, 20128, 9119). The self-check's wait is sufficient — it polls every 5s and reports per-port status. The CI-level wait just added 150s of dead time on every run, even when services are already up.
Port the Ollama section from hermes-codespace self-check. Checks: 1. Binary — ollama is in PATH 2. API — responds on :11434 3. Model — nomic-embed-text is listed (with 15s retry + filesystem fallback) 4. Embedding — can generate embeddings (with 30s retry) All Ollama checks are warnings, not critical failures — Ollama may not be available in all contexts (CI, first boot, etc.).
Two bugs fixed: 1. CI race condition: self-check ran before start-hermes.sh finished initializing config. Now waits for all start-*.sh scripts to exit using pgrep before running self-check. 2. Exit code clobbered: self-check detected critical failures correctly but then crashed writing /tmp/health-report.json (permission denied), killing the script with exit 1 before reaching 'exit 2'. Now the report write is non-fatal so the correct exit code propagates.
pgrep returned nothing because init scripts hadn't started yet when docker compose up -d returns. Instead, wait for /config/.hermes/config.yaml to exist — this file is created early in start-hermes.sh and is a reliable signal that initialization is underway.
The config file is created at runtime by start-hermes.sh, not at build time. In CI, the self-check may run before the config is fully written. Making this a warning instead of critical prevents false CI failures while still surfacing the issue in the health report.
This reverts commit 2d168dd.
The config file is created by 'hermes config set' inside runuser -l abc, but may end up with root ownership. Add explicit chown/chmod after config creation to ensure abc can read it during self-check.
We don't know why self-check (as abc) can't find config.yaml when the CI (as root) finds it. Adding whoami, HOME, ls, and test output to understand the actual root cause.
Root cause: hermes config set creates config.yaml as root inside runuser -l abc, but the /config/.hermes directory ends up owned by root. Self-check (as abc) gets 'Permission denied' when trying to ls the directory. Fix: chown -R abc:abc /config/.hermes as root AFTER the runuser block completes (when all files are created). Removed earlier incorrect fix that tried chown inside runuser. Also removed debug output now that root cause is confirmed.
The chown was after the runuser block, but config init is skipped on fresh containers (logs dir doesn't exist). Docker build creates config.yaml as root, VOLUME preserves it, but abc can't traverse the root-owned directory. Moving chown before runuser ensures permissions are fixed regardless of whether config init runs.
Root cause: the config.yaml check found a root-owned file from the Docker build, but start-hermes.sh hadn't finished creating the runtime config yet. The check passed immediately but self-check (as abc) couldn't read the root-owned file. Fix: wait for port 9119 to respond, which means start-hermes.sh has finished its initialization and created the config as abc. This is the same signal the passing run relied on (old 150s wait). Also removed the incorrect chown fix from start-hermes.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI builds were flaky — the Docker smoke test intermittently failed with
two distinct root causes, both causing the self-check to exit with code 2.
Root Cause 1: Dashboard crash on startup (intermittent)
The dashboard's startup runs a web UI staleness check and tries
npm install --workspace webwhen it detects the dist is outdated.In CI this fails because the web UI is already built during the Docker
image build step — the container doesn't have full dev dependencies
at runtime. When the dashboard crashes, socat on port 9119 forwards
to nothing, and self-check reports "no response from http://localhost:9119".
Fix: Pass
--skip-buildtohermes dashboardinstart-hermes.shso it serves the pre-built dist without trying to rebuild on startup.
Root Cause 2: Self-check runs before init scripts finish
The CI workflow started the container and immediately ran self-check.
start-hermes.shruns as a background process — by the time self-checkexecuted, the config file either didn't exist yet or was the root-owned
Docker build artifact (not the runtime
abc-owned version). This caused:❌ Config no config at /config/.hermes/config.yaml(critical failure)self-check: line 398: /tmp/health-report.json: Permission denied(clobbered the exit code from 2 to 1)
Fix: Wait for port 9119 to respond (HTTP 200/302/401) before running
self-check. Port 9119 responding means
start-hermes.shhas finishedand created all runtime files as
abc. Also fixed the JSON report writewith
|| trueto prevent permission errors from overriding the exit code.Changes
CI Workflow (
docker-publish.yml)make startwithNO_LOGS=1(dev/prod parity)abc(matching production)Self-Check (
self-check.sh)ported from hermes-codespace (warnings only, not critical)
|| trueso permissionerrors don't clobber the exit code from 2 to 1
Startup (
start-hermes.sh)--skip-buildflag for dashboard startupDead Code Cleanup (Kapati Guidelines — references PR #33)
_write_web_ui_build_stamp()doesn't exist in v2026.8.13)docker/01-copy-web-ui-stamp.sh(dead init script, never invoked)mkdir -p /opt/hermes-prebuiltfrom DockerfileOther
terminal.cwdto$HOMEduring initNO_LOGS=1env var support in MakefileStability Verification
5 consecutive CI reruns — ALL PASSED ✅
Previous 10-rerun test (before the config race fix) showed 6/10 pass
(60%). After the port 9119 health check fix: 5/5 pass (100%).
Files Changed (7 files, +83 / -49 lines)
.github/workflows/docker-publish.ymldocker/Dockerfiledocker/start-hermes.sh--skip-buildflagdocker/self-check.shdocker/01-copy-web-ui-stamp.shMakefileNO_LOGS=1supportdocker/start-omniroute.sh