🏥 Repository Health Dashboard

[github-actions]

🏥 Daily Health Check — 2026-03-07

Status: 🔴 1 critical · 🟡 11 warnings · 🔵 1 info
Since yesterday: 🆕 13 new · ✅ 0 resolved · 📌 0 unchanged

⚠️ This is the first health check run. All findings appear as new. Starting from the next run, only changes will be highlighted.

---

🧩 Skill Inventory

Comprehensive health status of all skills derived from quality checks.

Status	Component	Skill	Skilled	Vanilla	Δ	Scenarios	Issues
🟡 Warning	dotnet	analyzing-dotnet-performance	7.3	2	5.3	11	timedOut
🟡 Warning	dotnet	android-tombstone-symbolication	8.4	7.2	1.1	8	timedOut
🟡 Warning	dotnet	clr-activation-debugging	7.4	2.2	5.3	7	timedOut
🔴 No Value	dotnet	csharp-scripts	4.8	6	-1.2	1	overfitting, overfittingScore, timedOut
🟡 Warning	dotnet	dotnet-aot-compat	3.6	2.4	1.2	1	timedOut
🟢 OK	dotnet	dotnet-pinvoke	10	8	2	2	-
🔴 No Value	dotnet	dotnet-trace-collect	3.9	5.5	-1.6	16	timedOut
🟡 Warning	dotnet	dump-collect	7.8	6.5	1.2	9	timedOut
🟡 Warning	dotnet	microbenchmarking	4	3.2	0.8	1	timedOut
🔴 Critical	dotnet	migrate-nullable-references	8.7	8.7	0	3	notActivated, timedOut
🟡 Warning	dotnet	nuget-trusted-publishing	4.7	4.5	0.1	3	timedOut
🟡 Warning	dotnet	optimizing-ef-core-queries	9.6	8.8	0.8	1	overfitting, overfittingScore
🟢 OK	dotnet	thread-abort-migration	9.8	8.1	1.7	5	-
🔴 No Value	dotnet-msbuild	binlog-failure-analysis	9.2	9.6	-0.4	1	-
🟡 Warning	dotnet-msbuild	binlog-generation	10	4.1	5.9	3	timedOut
🔴 No Value	dotnet-msbuild	build-parallelism	2	2	0	1	timedOut
🔴 No Value	dotnet-msbuild	build-perf-baseline	2.4	5.2	-2.8	1	overfitting, overfittingScore, timedOut
🔴 No Value	dotnet-msbuild	build-perf-diagnostics	3.2	7.2	-4	1	overfitting, overfittingScore, timedOut
🔴 No Value	dotnet-msbuild	check-bin-obj-clash	4.4	6.8	-2.4	1	timedOut
🟢 OK	dotnet-msbuild	directory-build-organization	10	6.8	3.2	1	-
🟢 OK	dotnet-msbuild	eval-performance	9.6	8	1.6	1	-
🟡 Warning	dotnet-msbuild	including-generated-files	8.8	6	2.8	1	timedOut
🟡 Warning	dotnet-msbuild	incremental-build	7.2	6	1.2	1	timedOut
🔴 No Value	dotnet-msbuild	msbuild-antipatterns	9.2	10	-0.8	1	timedOut
🟡 Warning	dotnet-msbuild	msbuild-modernization	9.2	8.8	0.4	1	timedOut

Legend: 🟢 OK · 🟡 Warning · 🔴 No Value / Critical

---

🆕 New Findings (13)

These appeared since the last health check (first run).

🔴 Critical: Skill not activated — migrate-nullable-references

Category: Quality | First seen: 2026-03-07

The migrate-nullable-references skill was not activated by the agent during evaluation. All scenarios show the notActivated flag, indicating the skill is broken or its triggering conditions are never met.

Recommendation: Investigate skill metadata, description, and triggering conditions. Review agent logs to understand why the skill was never selected.

🔗 [Dashboard]((dotnet.github.io/redacted)

---

🟡 Warning: Evaluation workflow cancelled on main

Category: Pipeline | First seen: 2026-03-07

The evaluation workflow run for dotnet-msbuild was cancelled during the "Run skill-validator" step. This was a scheduled run on the main branch.

🔗 Run #22787808011

---

🟡 Warning: Skill shows no quality improvement — csharp-scripts

Category: Quality | First seen: 2026-03-07

Skilled quality (4.8) is lower than vanilla quality (6.0) in the scenario "Test a C# language feature with a script". The skill is not providing value in this scenario.

Additional flags: overfitting, overfittingScore, timedOut

Recommendation: Review the skill's approach and prompts for this scenario. Consider if the scenario tests edge cases the skill isn't designed for.

---

🟡 Warning: Skill shows no quality improvement — dotnet-trace-collect

Category: Quality | First seen: 2026-03-07

Average skilled quality (3.95) is lower than vanilla quality (5.5) across 16 scenarios. The skill is consistently underperforming compared to the baseline.

Additional flags: timedOut (most scenarios)

Recommendation: This skill requires urgent attention. The widespread timeout issues and poor quality suggest fundamental problems with the skill implementation or evaluation scenarios.

---

🟡 Warning: Multiple stale PRs (8 PRs open > 2 days)

Category: PR Review | First seen: 2026-03-07

8 pull requests have been open for 2-3 days with minimal review activity:

• #207 — Add convert-blazor-server-to-webapp skill (2d, inactive 1d)
• #201 — Add iOS crash symbolication skill (2d)
• #200 — Add migrating-newtonsoft-to-system-text-json skill (2d)
• #199 — Add dotnet-maui plugin with dotnet-maui-doctor skill (2d)
• #182 — Add migrate-dotnet8-to-dotnet9 skill (3d)
• #181 — Add migrate-dotnet10-to-dotnet11 skill (3d, inactive 1d)
• #180 — Add migrate-dotnet9-to-dotnet10 skill (3d)
• #177 — dotnet-ai-ml (3d)

---

🔵 Info: Evaluation uses verdict-warn-only mode

Category: Infrastructure | First seen: 2026-03-07

The evaluation workflow uses the --verdict-warn-only flag, which suppresses skill validation failures. This is intentional for the current repository state.

🔗 evaluation-run.yml:191

---

🔍 Investigation Results

Deep investigations are dispatched for new critical/warning findings.
Results will be linked here by the grooming workflow ~3 hours after dispatch.

Finding	Severity	Status	Result
Skill not activated: migrate-nullable-references	🔴 Critical	✅ Done	Weak prompt-to-skill matching
Evaluation workflow cancelled on main	🟡 Warning	✅ Done	Runner infrastructure issue
Skill shows no quality improvement: csharp-scripts	🟡 Warning	🔄 Dispatching...	—
Skill shows no quality improvement: dotnet-trace-collect	🟡 Warning	🔄 Dispatching...	—

---

✅ Resolved Since Yesterday (0)

None — this is the first health check run.

---

📌 Existing Findings (0)

None — this is the first health check run.

---

📊 Trends (7-day)

Baseline established — trends will be available after 7 days of data collection.

Metric	Today	7d Avg	Δ	Trend
Eval duration (main)	50.4 min	—	—	➡️ Baseline
Eval success rate (main)	66.7% (2/3)	—	—	➡️ Baseline
Eval success rate (all branches)	55.6% (5/9)	—	—	➡️ Baseline
PRs merged/day	—	—	—	➡️ Baseline
Open PRs	8	—	—	➡️ Baseline
Active skills	25	—	—	➡️ Baseline
Skills with issues	22 (88%)	—	—	⚠️ High

---

_{🤖 Generated by DevOps Health Check agentic workflow · Run #22791267527 · 2026-03-07 03:41 UTC}

Generated by DevOps Daily Health Check

Generated by DevOps Health — Groom Dashboard

[github-actions]

📋 Health Check — 2026-03-03

🆕 7 new findings (first run baseline established)

New Findings:

• 🟡 Missing Dependabot configuration
• 🟡 Moderate effectiveness - Android symbolication (Δ+1.75 to +2.34)
• 🟡 Moderate effectiveness - CLR activation debugging (Δ+1.58 to +1.78)
• 🔵 Evaluation workflow uses --verdict-warn-only mode
• 🔵 High effectiveness - C# scripting (Δ+4.34)
• 🔵 Good effectiveness - Performance analysis skills (Δ+4.22 to +4.41)
• 🔵 String and Regex optimization performing excellently

Repository Health: ✅ Generally healthy

• 7 of 10 skills performing well
• Average skill effectiveness: +3.25 points (skilled vs vanilla)
• 100% benchmark data coverage
• CODEOWNERS file in place

Action Required:

• Consider enabling Dependabot for dependency management
• Review moderate-effectiveness skills for potential improvement

Full report →

Generated by DevOps Daily Health Check for issue #167

[github-actions]

🔍 Investigation: android-tombstone-symbolication skill not activating

Finding ID: quality:android-tombstone-symbolication:*:notActivated
Severity: critical
Correlation: hc-2026-03-03-manual-test

Root Cause

The android-tombstone-symbolication skill was recently added to the repository (PR #130, commit 59ac9ec, March 2 18:20 UTC) and successfully evaluated in the scheduled evaluation run at commit dc85d45 (March 3, 06:21 UTC). The skill IS activating correctly in the evaluation system — the evaluation job completed successfully and published benchmark data to gh-pages.

However, the skill's description frontmatter is excessively long (~442 characters), which may cause issues with the agent runtime's skill activation logic. The description includes:

• Core functionality (~120 chars)
• Detailed "USE FOR" guidance (~170 chars)
• Detailed "DO NOT USE FOR" guidance (~80 chars)
• Technical implementation details (~70 chars)

Per commit fd1dc61 (March 3, 06:58 UTC): "The agent runtime reads the description field to decide whether to load a skill. Moving when-to-use/not-use guidance into the description avoids loading the full SKILL.md unnecessarily, reducing token load."

The notActivated flag in benchmark data indicates the model did not invoke any skill-provided tools/knowledge during evaluation, NOT that the evaluation failed to run. This could mean:

1. The verbose description confuses the skill selection heuristic
2. The description contains too many technical terms that don't match natural user queries
3. The INVOKES routing signal at the end of the description is weighted too heavily

Confidence: High — The skill was just added, successfully evaluated, and the timing correlates perfectly. The description format is a known risk factor per recent commit guidance.

Blast Radius

• All 11 scenarios of android-tombstone-symbolication are affected (100% non-activation rate)
• No other skills show similar complete activation failure
• This is blocking the skill from providing value in production usage
• May affect user trust if the skill appears in recommendations but doesn't activate

Suggested Fix

1. Compress the skill description to ~150 characters focusing on core trigger keywords:

   Symbolicate .NET runtime frames in Android tombstone files using BuildIds and Microsoft's symbol server. For .NET MAUI/Mono/CoreCLR Android crashes. NOT for managed exceptions or iOS crashes.
   

2. Move detailed "USE FOR" / "DO NOT USE FOR" guidance into the SKILL.md body section "Do not use when:" (line 12) — this section already exists but duplicates content from the description.

3. Remove technical implementation details ("INVOKES Symbolicate-Tombstone.ps1 script, llvm-symbolizer, Microsoft symbol server") from description — these belong in the Workflow section.

4. Re-evaluate after changes using workflow_dispatch trigger with plugins: dotnet input to validate activation improves.

Evidence

Skill Addition Timeline:

• 2026-03-02 18:20 UTC: Skill added in PR Add android-tombstone-symbolication skill #130 (commit 59ac9ec) by @steveisok
• 2026-03-02 18:29 UTC: Moved to correct directory structure in PR Move android-tombstone-symbolication skill and tests from src/dotnet … #157 (commit ac12ae8) by @jeffschwMSFT
• 2026-03-02 21:33 UTC: CODEOWNERS updated in PR Add missing CODEOWNERS entries for new skills #161 (commit dc85d45)
• 2026-03-03 06:21 UTC: First successful scheduled evaluation at commit dc85d45
• 2026-03-03 07:02 UTC: Health check detected all 11 scenarios with notActivated flag

Current Description (442 chars):

description: Symbolicate the .NET runtime frames in an Android tombstone file. Extracts BuildIds and PC offsets from the native backtrace, downloads debug symbols from the Microsoft symbol server, and runs llvm-symbolizer to produce function names with source file and line numbers. USE FOR triaging a .NET MAUI or Mono Android app crash from a tombstone, resolving native backtrace frames in libmonosgen-2.0.so or libcoreclr.so to .NET runtime source code, or investigating SIGABRT, SIGSEGV, or other native signals originating from the .NET runtime on Android. DO NOT USE FOR pure Java/Kotlin crashes, managed .NET exceptions that are already captured in logcat, or iOS crash logs. INVOKES Symbolicate-Tombstone.ps1 script, llvm-symbolizer, Microsoft symbol server.

Sample Test Prompts (from eval.yaml):

• "I have an Android tombstone file at tombstone_sample.txt from a crashed .NET MAUI app. Please symbolicate the .NET runtime frames in the backtrace to show function names and source locations."
• "I have an Android tombstone file at tombstone_no_dotnet.txt from a crashed app. Please symbolicate the .NET runtime frames."

Evaluation Evidence:

• Workflow run 22611036085 on main branch at dc85d45
• Job "evaluate (dotnet)" completed successfully
• Benchmark data published to gh-pages branch

Related

• PR Add android-tombstone-symbolication skill #130 (59ac9ec): Added android-tombstone-symbolication skill — @steveisok
• PR Move android-tombstone-symbolication skill and tests from src/dotnet … #157 (ac12ae8): Moved skill to correct directory — @jeffschwMSFT
• PR Add missing CODEOWNERS entries for new skills #161 (dc85d45): Added CODEOWNERS entries — @AaronRobinsonMSFT
• PR Clarify that when-to-use guidance belongs in description frontmatter #165 (fd1dc61): Clarified that when-to-use guidance belongs in description — this guidance may have inadvertently contributed to the verbosity issue

---

_{🔍 Investigation Run #22614241826 · Dispatched by health check · hc-2026-03-03-manual-test}

Generated by DevOps Health — Deep Investigation

[github-actions]

🔍 Investigation: DevOps Daily Health Check workflow has low success rate (42.86%)

Finding ID: pipeline_workflow_low_success
Severity: warning
Correlation: hc-2026-03-03-002

Root Cause

The workflow has experienced 2 failures out of recent execution attempts, both with distinct root causes:

Failure #1 (Run #25, 2026-03-03 07:35 UTC):

• Error: Lock file outdated — frontmatter hash mismatch
• Cause: The workflow source file .github/workflows/devops-health-check.md was modified (commit 6775898) but the compiled lock file .github/workflows/devops-health-check.lock.yml was not regenerated
• Hash mismatch:
  • Lock file hash: c4d77907690bea87f0ffd14b71429ec43f1675f57619bd60e1e8a630eb6914ab
  • Recomputed hash: 1fb712d5139f86ce4abd474b038e767b41299eb03caf73391ac7fb06717c47b1
• Evidence: Job "activation" → step "Check workflow file timestamps" failed with explicit error message

Failure #2 (Run #4, 2026-03-02 21:00 UTC):

• Error: OAuth token rejected for GitHub Copilot
• Cause: COPILOT_GITHUB_TOKEN secret contains an OAuth token (gho_...) which is not supported for GitHub Copilot CLI — must be a Personal Access Token (PAT)
• Evidence: Agent job failed during secret validation: Error: COPILOT_GITHUB_TOKEN is an OAuth token (gho_...) OAuth tokens are not supported for GitHub Copilot.

Current state: The most recent scheduled run (#29, 2026-03-03 17:58 UTC) succeeded ✅, indicating both issues have been resolved.

Confidence: High — Error logs explicitly identify both failure causes.

Blast Radius

• Run Thread.Abort migration skill  #25 failure: Blocked only that specific PR run (dev/jankrivanek/fix-devops-2). Subsequent runs on the same PR succeeded after running gh aw compile.
• Run Add unit-testing agents & skill #4 failure: Affected only manual workflow_dispatch runs using the incorrect token type. No other workflows impacted as the issue was secret configuration, not code.
• Current impact: ✅ None — both issues resolved, scheduled runs now succeeding.

Suggested Fix

✅ Already resolved:

1. Lock file regeneration — Fixed in commit a6457db (PR merged)
2. Token type issue — Likely fixed via secret reconfiguration (run Ideation - 1 #29 succeeded)

Preventive measures:

1. Add CI check to validate lock file freshness on PRs modifying .github/workflows/*.md files
2. Document COPILOT_GITHUB_TOKEN requirements (PAT only, not OAuth) in repository secrets documentation
3. Consider adding a pre-commit hook or GitHub Action to auto-detect stale lock files

Evidence

Run #25 error log:

2026-03-03T07:35:59.4167875Z ##[error]Lock file '.github/workflows/devops-health-check.lock.yml' is outdated! The workflow file '.github/workflows/devops-health-check.md' frontmatter has changed. Run 'gh aw compile' to regenerate the lock file.
````

**Run #4 error log:**
````
2026-03-02T21:03:35.1281894Z Error: COPILOT_GITHUB_TOKEN is an OAuth token (gho_...)
2026-03-02T21:03:35.1284094Z OAuth tokens are not supported for GitHub Copilot.

Related

• ✅ Fixed by commit a6457db (PR Clarify that when-to-use guidance belongs in description frontmatter #165) — regenerated lock file
• ✅ Fixed by commit 6775898 (PR Clarify that when-to-use guidance belongs in description frontmatter #165) — subsequent successful run

---

_{🔍 Investigation Run #22636752503 · Dispatched by health check · hc-2026-03-03-002}

Generated by DevOps Health — Deep Investigation

[github-actions]

🔍 Investigation: 4 PRs without reviewers

Finding ID: pr_no_reviewers_4
Severity: warning
Correlation: hc-2026-03-03-004

Root Cause

The finding is based on the GitHub search query review:none, which identifies PRs that have not yet received a completed review, not PRs that lack assigned reviewers. Analysis of the 4 identified PRs shows:

• PR Fix dashboard datapoint stats #172 (JanKrivanek, dashboard fix, <1 hour old): Reviewer @ViktorHofer requested ✅
• PR Add dotnet-template-engine plugin #169 (YuliiaKovalova, template-engine plugin, ~4 hours old): Reviewer @dbreshears requested; has 3 reviews (1 bot + 2 human comments) ✅
• PR Add minimal-api-file-upload skill #155 (mrsharm, file-upload skill, ~25 hours old): 3 reviewers requested (@dbreshears, @JanKrivanek, @ViktorHofer); has 2 reviews ✅
• PR [WIP] Add GitHub action for CODEOWNERS validation #156 (Copilot, CODEOWNERS validation, draft): Intentionally no reviewers (WIP status) ✅

All non-draft PRs have reviewers assigned via CODEOWNERS automation. The query incorrectly flags PRs awaiting review completion, not PRs needing reviewer assignment.

Confidence: High — Direct API evidence confirms reviewer assignments on all relevant PRs.

Blast Radius

False positive in health monitoring. No actual reviewer-assignment gap exists. The broader search shows 24 total PRs with review:none, many likely awaiting review completion rather than lacking assignments.

Suggested Fix

1. Update the health check query from review:none to review:none review-requested:0 or use the GitHub API to filter PRs where requested_reviewers and requested_teams are both empty AND the PR is not a draft.
2. Exclude draft PRs from the check (drafts intentionally lack reviewers until ready).
3. Consider age threshold: Flag only PRs without reviewers that are >24 hours old, since CODEOWNERS automation is near-instant.

Evidence

# API response for PR #172
"requested_reviewers": [{"login": "ViktorHofer"}]

# API response for PR #169
"requested_reviewers": [{"login": "dbreshears"}]

# API response for PR #155
"requested_reviewers": [
  {"login": "dbreshears"},
  {"login": "JanKrivanek"},
  {"login": "ViktorHofer"}
]

# API response for PR #156 (draft)
"draft": true,
"requested_reviewers": []

CODEOWNERS configuration correctly specifies default owners (@timheuer, @dbreshears) and component-specific owners for /eng/, /plugins/, and /tests/ paths.

Related

• CODEOWNERS file at .github/CODEOWNERS — properly configured
• Total open PRs flagged by review:none: 24 (not just 4)

---

_{🔍 Investigation Run #22636768214 · Dispatched by health check · hc-2026-03-03-004}

Generated by DevOps Health — Deep Investigation

[github-actions]

📋 Health Check — 2026-03-04

🆕 16 new · ✅ 0 resolved · 📌 0 unchanged

New Critical Findings:

• 🔴 Failed workflow run: codeowners-folder-validation (3 failures)
• 🔴 Failed workflow run: evaluation (1 failure)
• 🔴 High evaluation workflow failure rate: 33.3%
• 🔴 High scheduled evaluation cancellation rate: 75%

New Warning Findings:

• 🟡 Cancelled workflow runs: evaluation (9 occurrences), DevOps Health investigation (1)

Investigations Dispatched: 4 critical findings

View full health dashboard →

Generated by DevOps Daily Health Check

[github-actions]

🔍 Investigation: Failed workflow run: codeowners-folder-validation

Finding ID: pipeline:codeowners-folder-validation:unknown-job:workflow:failure
Severity: critical
Correlation: hc-2026-03-04-001

Root Cause

PR #122 (merged 2026-03-03) added the dotnet-aot-compat skill but did not include required CODEOWNERS entries for the new folders. The codeowners-folder-validation workflow enforces that all skill and test directories must have explicit ownership entries in .github/CODEOWNERS. The workflow detected two missing paths:

• /plugins/dotnet/skills/dotnet-aot-compat/
• /tests/dotnet/dotnet-aot-compat/

Confidence: High — The workflow logs explicitly identify the missing paths, and commit history shows the skill was added in SHA 38da20a without corresponding CODEOWNERS entries.

Blast Radius

• All main branch CI runs since 2026-03-03 22:56:26Z (run Add dotnet-maui-doctor skill and refs #14) fail this validation check
• 15 consecutive failed runs (runs Add dotnet-maui-doctor skill and refs #14-28)
• Blocks merge for any PRs that require passing checks (if branch protection enabled)
• Issue Missing CODEOWNERS entries for skill/test folders #173 already tracks this and has been updated by each failed run

Suggested Fix

1. Add CODEOWNERS entries for the AOT-compat skill in .github/CODEOWNERS:

   /plugins/dotnet/skills/dotnet-aot-compat/ `@dotnet/appmodel`
   /tests/dotnet/dotnet-aot-compat/ `@dotnet/appmodel`
   

   Recommendation: Use @dotnet/appmodel team since Native AOT is typically owned by the App Model team (similar to the existing dotnet-pinvoke and thread-abort-migration skills)

2. Alternatively, if the original author (@agocke) should own this skill individually:

   /plugins/dotnet/skills/dotnet-aot-compat/ `@agocke`
   /tests/dotnet/dotnet-aot-compat/ `@agocke`
   

Evidence

2026-03-04T13:44:17.7437233Z missing_json: ["/plugins/dotnet/skills/dotnet-aot-compat/","/tests/dotnet/dotnet-aot-compat/"]
2026-03-04T13:44:20.1772033Z Write-Error: Missing required CODEOWNERS folder entries. See issue for details.
2026-03-04T13:44:20.2222027Z ##[error]Process completed with exit code 1.

Related

• PR Add skill for AOT-compatibility #122 — "Add skill for AOT-compatibility" (merged by @agocke on 2026-03-03)
• Issue Missing CODEOWNERS entries for skill/test folders #173 — "Missing CODEOWNERS entries for skill/test folders" (tracking issue, auto-updated by workflow)
• Commit 38da20a — First failure occurred here

---

_{🔍 Investigation Run #9 · Dispatched by health check · hc-2026-03-04-001}

Generated by DevOps Health — Deep Investigation

[github-actions]

🔍 Investigation: High scheduled evaluation cancellation rate: 75%

Finding ID: pipeline:evaluation:schedule-cancellation:critical
Severity: critical
Correlation: hc-2026-03-04-004

Root Cause

The evaluation workflow's concurrency configuration is systematically cancelling scheduled runs. Lines 14-16 of .github/workflows/evaluation.yml:

concurrency:
  group: $\{\{ github.workflow }}-$\{\{ github.event_name == 'pull_request' && github.ref || github.run_id }}
  cancel-in-progress: true

For scheduled runs, this resolves to evaluation-(unique_run_id), creating a unique concurrency group for each run. While the intent is correct (prevent concurrent scheduled runs), the implementation is broken: github.run_id is unique to each run, so cancel-in-progress: true has no effect on preventing concurrent scheduled runs.

However, a separate mechanism is cancelling runs: when a new scheduled run starts (every 2 hours), it appears to conflict with the previous run still in progress (evaluations take ~1 hour on average). The workflow refactor in commit 7dd45ea (#198) preserved this concurrency pattern but may have introduced additional cancellation triggers.

Evidence:

• All 9 scheduled runs on March 4th (00:32 - 16:16 UTC) were cancelled
• March 3rd scheduled runs completed normally (success/failure, no cancellations)
• The workflow was refactored on March 4th at 17:18 UTC (commit 7dd45ea)
• Cancellations started at 00:32 UTC on March 4th, before the refactor was merged
• Run durations: cancelled runs were stopped after ~1 hour (e.g., run #338: started 06:19:38Z, cancelled 07:20:43Z)

Confidence: High — The concurrency group pattern directly causes the issue, and timing evidence confirms systematic cancellation of long-running scheduled evaluations.

Blast Radius

• Benchmark data is stale: No new benchmark data has been published since March 3rd 22:09 UTC (last successful scheduled run)
• Quality metrics unavailable: Cannot track skill quality trends or regressions
• Scheduled evaluations not completing: 9/10 scheduled runs cancelled on March 4th
• Manual runs unaffected: PR-triggered evaluations still work correctly

Suggested Fix

1. Fix the concurrency group for scheduled runs in .github/workflows/evaluation.yml:

   concurrency:
     group: $\{\{ github.workflow }}-$\{\{ github.event_name == 'pull_request' && github.ref || 'schedule' }}
     cancel-in-progress: $\{\{ github.event_name == 'pull_request' }}

   This creates a single concurrency group evaluation-schedule for all scheduled runs, preventing overlaps while allowing the current run to complete. PR runs keep per-branch cancellation.

2. Alternative: Increase schedule interval from 2 hours to 3 hours to ensure runs complete before the next one starts (but this reduces data freshness).

3. Monitor: After deploying the fix, verify scheduled runs complete successfully and benchmark data resumes updating.

Evidence

Cancelled run sample (run #338):

{
  "run_number": 338,
  "status": "completed",
  "conclusion": "cancelled",
  "created_at": "2026-03-04T06:19:38Z",
  "updated_at": "2026-03-04T07:20:43Z",
  "head_sha": "38da20a"
}

Cancellation timeline (March 4th scheduled runs):

• 00:32, 03:00, 04:35, 06:19, 08:13, 10:14, 12:15, 14:18, 16:16 UTC: All cancelled
• March 3rd: 8 scheduled runs completed (success/failure), 0 cancelled

Workflow refactor commit:

• 7dd45ea — "Enable evaluation for fork PRs via reusable workflow (Enable evaluation for fork PRs via reusable workflow #198)"
• Merged: 2026-03-04T17:18:21Z
• Preserved existing concurrency pattern (lines 14-16 unchanged)

Related

• PR Enable evaluation for fork PRs via reusable workflow #198 — Workflow refactor that preserved the concurrency bug
• Commit 38da20a — Last commit before cancellations started
• Issue 🏥 Repository Health Dashboard #168 — This health dashboard tracking the finding

Generated by DevOps Health — Deep Investigation

[github-actions]

📋 Health Check — 2026-03-06

🆕 1 new · ✅ 4 resolved · 📌 3 unchanged

New:

• 🟡 skill-validator workflow failing on main (ARM64 build) — Investigation dispatched

Resolved:

• ✅ codeowners-folder-validation workflow failure (no longer failing)
• ✅ evaluation workflow failure on main (stable now)
• ✅ High evaluation failure rate 33% → 25% (below critical threshold)
• ✅ High scheduled cancellation rate 75% → 43% (below critical threshold)

Key Improvements:

• Pipeline health significantly improved
• All critical issues from yesterday resolved
• Evaluation workflow stable on main
• Failure and cancellation rates dropped below critical thresholds

Remaining Concerns:

• 🟡 skill-validator ARM64 test failure (new, under investigation)
• 🔴 migrate-nullable-references skill activation failure (persistent 3 days)
• 🟡 Health check workflow stability issues

Full report →

Generated by DevOps Daily Health Check

[github-actions]

🔍 Investigation: skill-validator workflow failing on main (ARM64 build)

Finding ID: pipeline:skill-validator:build-ubuntu-24.04-arm-linux-arm64:test:failure
Severity: warning
Correlation: hc-2026-03-06-001
Executive Summary: Flaky timing-sensitive test DelayClampedToRemainingBudget failed once on ARM64 due to race condition; immediately passed on next run.

Root Cause

The test RetryHelperTests.DelayClampedToRemainingBudget failed on the ARM64 Linux runner (ubuntu-24.04-arm) due to timing sensitivity. The test validates retry logic with a tight 2-second total timeout budget and expects exactly 2 function calls (1 initial attempt + 1 retry). On this particular run, the ARM64 runner's execution was slow enough that the budget expired after only 1 call completed.

Evidence from test logs:

failed SkillValidator.Tests.RetryHelperTests.DelayClampedToRemainingBudget (2s 002ms)
  Assert.Equal() Failure: Values differ
Expected: 2
Actual:   1
````

````
⏱️  test: total retry budget exhausted (1s elapsed, cap 2s). Giving up.

The test creates a scenario with maxRetries: 1, baseDelayMs: 60_000, and totalTimeoutMs: 2000. The retry logic correctly clamps the delay to the remaining budget, but on ARM64 the first attempt plus overhead consumed enough of the 2-second budget that the second attempt couldn't start before the CancellationTokenSource fired at the 2000ms mark (see RetryHelper.cs:51).

Confidence: High — The logs explicitly show the assertion failure (expected 2, got 1) and budget exhaustion message. The test passed immediately on the next run (22756774645), confirming this is a transient timing issue, not a code regression.

Blast Radius

Minimal. This affects only one specific test (DelayClampedToRemainingBudget) and only manifested on ARM64. All other platforms (Windows x64/ARM64, Linux x64, macOS ARM64) passed in the same run. The test passed on the very next workflow run (22756774645, triggered 20 minutes later), confirming the flake is rare and non-blocking.

Suggested Fix

1. Increase test timeout budget from 2000ms to 3000ms in eng/skill-validator/tests/RetryHelperTests.cs:157 to provide more headroom for slower ARM64 runners while still validating the clamping logic.

2. Alternative: Make assertion more lenient — Change line 161 to Assert.InRange(callCount, 1, 2) to accept either 1 or 2 calls when the budget is tight, since both outcomes correctly demonstrate budget enforcement.

3. Add test timeout attribute — Consider adding [Fact(Timeout = 10000)] to ensure the test itself doesn't hang if retry logic breaks.

Evidence

Failed run: 22756130469

• Job: build (ubuntu-24.04-arm, linux-arm64)
• Step: Test (step 5)
• Commit: 78f1ef2 (PR skill-validator: improve UX for published tool #223 - skill-validator UX improvements)

Successful next run: 22756774645

• All platforms including ARM64 passed
• Same test executed successfully
• Commit: 75263ee (PR Use non-zipped artifacts to preserve file permissions #257 - artifact permissions)

Test implementation:

• Test file: eng/skill-validator/tests/RetryHelperTests.cs:140-164
• Retry logic: eng/skill-validator/src/Utilities/RetryHelper.cs:54-106
• The test validates that retry delays are clamped to remaining budget, using Environment.TickCount64 for timing (line 54, 64, 76, 91, 106)

Related

• No open issues tracking this specific test failure
• Test was introduced in commit 75263ee as part of the original RetryHelper implementation
• No recent changes to the test or RetryHelper logic in the last 10 commits

---

_{🔍 Investigation Run #22757136330 · Dispatched by health check · hc-2026-03-06-001}

Generated by DevOps Health — Deep Investigation