Skip to content

Fix collect_data; route kind:benchmarks sections - #156

Merged
mjeremicTT merged 3 commits into
tenstorrent:mainfrom
dcvijeticTT:fix/benchmark-telemetry-collector
Sep 15, 2026
Merged

mjeremicTT merged 3 commits into
tenstorrent:mainfrom
dcvijeticTT:fix/benchmark-telemetry-collector

Conversation

@dcvijeticTT

@dcvijeticTT dcvijeticTT commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Update benchmark collection to support tt-inference-server’s report format introduced at the end of July, while preserving legacy reports.

  • Route kind: "benchmarks" sections through the detailed mapper, preserving metrics, operating points, tool metadata, and target checks. Now kind "benchmarks" gets tagged with run_type: benchmarks in the DB
  • Add missing image/audio metrics and expand Whisper sweep records.
  • Filter report filenames and improve handling of unavailable or invalid metric values.

Copilot AI lite review requested due to automatic review settings September 14, 2026 13:10
@dcvijeticTT
dcvijeticTT requested a review from a team as a code owner September 14, 2026 13:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved review issues were identified.

Pull request overview

Updates collect_data to support kind: "benchmarks" sections, broader report formats, and partial-failure handling.

Changes:

  • Filters benchmark report filenames.
  • Expands metric mapping and validation.
  • Routes structured benchmark data through the detailed mapper.
  • Tracks failures without blocking report uploads.
File summaries
File Summary
.github/actions/collect_data/test/test_benchmark_mapper.py Adds coverage for routing, metrics, discovery, and failure handling.
.github/actions/collect_data/src/shared.py Adds failure-state reset support.
.github/actions/collect_data/src/generate_data.py Resets and reports partial collection failures.
.github/actions/collect_data/src/benchmark.py Implements report filtering, routing, mapping, and validation.
Review details
  • Files reviewed: 4/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 15, 2026 08:35
@dcvijeticTT
dcvijeticTT force-pushed the fix/benchmark-telemetry-collector branch from d5c2ca3 to 19d04ef Compare September 15, 2026 08:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved moderate issues affect report determinism and migration correctness.

Review details

Suppressed comments (1)

.github/actions/collect_data/src/benchmark.py:70

  • When both model_spec_*.json and runtime_model_spec_*.json exist for one job, this whitelist puts both paths into the same report group, but create_json_from_report selects whichever path happens to come first and parses the other as a benchmark report. Because os.walk does not guarantee that order, the model identity/config can vary between runs; select a canonical spec deterministically and exclude all other spec files from report processing.
    "model_spec_",
    "runtime_model_spec_",
  • Files reviewed: 4/5 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 15, 2026 08:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deterministically select one model-spec file and skip all model-spec files during benchmark report iteration.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 4/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread .github/actions/collect_data/src/benchmark.py
@vmilosevic

Copy link
Copy Markdown
Contributor

Please fill in the PR description with what is changed, why and how, few sentences is enough, thanks!

@mjeremicTT
mjeremicTT merged commit 4de7abe into tenstorrent:main Sep 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants