perf(build): add per-stage timing telemetry for postprocess#336
Closed
gzenz wants to merge 12 commits into
Closed
Conversation
gzenz
force-pushed
the
perf/build-timing-telemetry-v2
branch
3 times, most recently
from
April 21, 2026 19:07
b77240c to
11b2aba
Compare
- Remove duplicate macOS-copy module files (analysis 2.py, enrich 2.py, enrich 3.py, exports 2.py, exports 3.py, graph_diff 2.py, jedi_resolver 2.py, memory 2.py, memory 3.py, token_benchmark 2.py) that tripped ruff N999. - Sort imports in main.py (ruff I001). - Drop unused parse_git_diff_ranges import in tools/review.py (ruff F401). - Add type-ignore for FastMCP._tool_manager private attribute access in main.py. CI on main has been red since PR tirth8205#94. This fixes it.
MCP tool calls were opening a fresh GraphStore (and SQLite connection) on every invocation. Cache one GraphStore per db_path and reuse it, falling back to a fresh connection if the cached one is dead. Thread-safe via a module-level lock.
Exact known-answer assertions for compute_risk_score and trace_flows, plus error-path coverage for parser on malformed input and module-cache eviction. - TestRiskScoreExact: 6 tests pinning risk score math (untested, tested, security keyword, caller fractions, 20-caller cap) - TestFlowExact: 4 tests for linear chain depth, cycles, single-file criticality, and non-test neighbors - TestParserErrorPaths: 8 tests for binary files, unknown extensions, syntax errors, empty files, CRLF, deeply nested AST, unicode - TestCacheEviction: oldest-half module cache eviction
Add three helpers: - `_resolve_star_imports`: walks top-level `import_from_statement` nodes, detects `wildcard_import`, resolves the source module to a file, and merges the exported names into the caller's import_map. - `_get_exported_names`: returns the public names a module exports, respecting `__all__` when present. Caches results per resolved path with a threading lock so concurrent parses share work safely. - `_extract_dunder_all`: parses `__all__ = [...]` at module scope. Wires star-import expansion into `parse_bytes` and the notebook concat path. Enables resolution of calls that come in via wildcard imports (e.g. `from constants import *` then `use_constant()`).
gzenz
force-pushed
the
perf/build-timing-telemetry-v2
branch
from
May 7, 2026 16:51
11b2aba to
3bbdb45
Compare
When receiver filtering emits "Foo.bar" (uppercase receiver), resolve it to "/path/file::Foo.bar" by looking up the class entry in the file symbol table. Closes a resolution gap where class-qualified bare calls stayed unresolved after the receiver filter rewrite.
Add _resolve_cross_file_class_methods as a postprocess step that scans Class nodes, builds a name -> file::ClassName map (dropping colliding names), and rewrites bare ClassName.method edge targets to the qualified file::ClassName.method form. Complements the per-file resolver in parser.py which only sees symbols in the calling file.
The cross-file ClassName.method resolver dropped any class whose name appeared in more than one file. That was too strict: codebases commonly reuse short class names across feature packages (util/ClassA.kt and test/ClassA.kt, multiple platform-specific Impl files, etc.). The call site almost always means the version in the closest package. Rebuild class_candidates as a name -> [qn, ...] map. For each edge with a bare ClassName.method target, pick the class qn whose file path shares the deepest common directory prefix with the edge's calling file. If the top match is not unique, fall back to leaving the target bare (rather than mis-resolving).
…odes The parser already extracted decorator tuples for test detection but discarded them before emitting NodeInfo. flows._has_framework_decorator and refactor._is_entry_point expect this data on nodes[...].extra; with no source data, entry-point detection never fires and framework-exposed callables (FastAPI routes, Click commands, Spring handlers) are flagged as dead. Wire class-level decorators (Python decorated_definition, Java/Kotlin modifiers.annotation/marker_annotation) and function-level decorators (existing tuple) into NodeInfo.extra["decorators"]. flows.py patterns already cover FastAPI/Click/Spring/Angular/Celery/pytest.
_resolve_call_targets only consulted the same-file symbol table, so every
call to an imported library symbol (Column, useState, Depends, etc.) stayed
bare. For Python-heavy repos this is the dominant unresolved class and
halves the CALLS resolution rate.
Pass the file's import_map through to _resolve_call_targets and add a
third resolution arm: bare target -> imported module -> {module}::{name}.
Also matches ClassName.method when ClassName is imported. Same-file
symbols still take precedence.
Notebook parse path merges import_maps across per-language cell groups
before resolving, so Python/R notebooks see all imports.
- Dedupe callers/callees/inheritors by qualified_name (repeated CALLS edges were surfacing the same node multiple times). - tests_for: use store.get_transitive_tests (direct TESTED_BY + one CALLS hop) instead of only direct TESTED_BY edges. - Disambiguation: when multiple candidates match a bare name and exactly one is a production node (not is_test), prefer it silently instead of returning 'ambiguous'.
Webpack/rollup bundle outputs and AWS CDK synth artifacts are generated code that pollutes the graph. Add them to DEFAULT_IGNORE_PATTERNS.
Log duration of each postprocess stage (signatures, FTS, flows, communities, summaries) and include them in the build result under 'postprocess_timing'. Also log which summary table skipped on sqlite OperationalError (was silent rollback).
gzenz
force-pushed
the
perf/build-timing-telemetry-v2
branch
from
May 7, 2026 17:21
3bbdb45 to
42f2f04
Compare
Owner
|
The per-stage timing idea is useful, but head |
tirth8205
added a commit
that referenced
this pull request
Jul 17, 2026
Port the safe structured telemetry from PR #336 onto the current build pipeline. Co-authored-by: Gideon Zenz <91069374+gzenz@users.noreply.github.com>
tirth8205
added a commit
that referenced
this pull request
Jul 17, 2026
Port the safe structured telemetry from PR #336 onto the current build pipeline. Co-authored-by: Gideon Zenz <91069374+gzenz@users.noreply.github.com>
tirth8205
added a commit
that referenced
this pull request
Jul 17, 2026
Port the safe structured telemetry from PR #336 onto the current build pipeline. Co-authored-by: Gideon Zenz <91069374+gzenz@users.noreply.github.com>
Owner
|
Resolved and merged through the current-main replacement PR #637.\n\nWhat was retained from this contribution:\n- structured post-process timing fields in the build result;\n- minimal mode reports signatures_s and fts_s;\n- full mode additionally reports flows_s, communities_s, and summaries_s;\n- no timing data is printed to stdout.\n\nThe replacement was rebased onto protected main, preserves Gideon Zenz's co-author attribution, and merged as dea8d47.\n\nVerification:\n- focused timing tests: 3 passed;\n- full local suite: 1,507 passed, 1 skipped, 2 xpassed;\n- Ruff, graph review, and diff review passed;\n- all 11 GitHub checks passed, including Python 3.10–3.13 and the Windows daemon/file-handle job. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per-stage timing logs for postprocess so slow stages are visible in build output.
Stacked PR approach
This is PR 11/16 in a stack that ports closed #158 into small, reviewable slices based on your feedback. Each branch contains this commit plus every earlier branch in the stack, so it applies cleanly on its own. The intended merge order is bottom-to-top.
Stack (merge order):
After you merge PR N, I rebase PRs N+1..16 on current main and force-push. Each PR ends up as a single commit on top of the previous one. Review only the newest commit in this PR; the rest will already be on main (or in earlier PRs in the stack).
All 16 branches are rebased on 072ab80. None re-attempts the closed lang-handler refactor.
Roadmap: restoring v7 loop-test parity
Measured impact on loop-test fixtures (Gadgetbridge Kotlin/Java, HealthAgent Python/TS):
Still deferred after this stack