Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,17 @@ disabled. A successful probe reports granted permission; a recognized
remediation; other failures remain `degraded` with bounded diagnostics. Passive
capability discovery therefore reports that an active probe is required, and a
capture plan must bind and recheck the active report immediately before running
the workload.
the workload. A denied probe is reported as `permission_required`, which is
unusable for sampling until the host's `perf_event_open` policy is changed;
the report includes the observed kernel restriction and a remediation to grant
the required event access before refreshing capabilities.

Workloads that declare `nvcc` receive a separate bounded CUDA toolkit preflight.
It compiles a tiny header probe and distinguishes an installed compiler from a
usable development toolkit. Missing `cuda_runtime.h` is recorded as
`environment_blocked` with the compiler diagnostic and a remediation to install
the CUDA development headers. Flameox does not build the workload during this
check.

#### `perfetto`

Expand Down
22 changes: 14 additions & 8 deletions docs/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@ the exact active executable and fixed launcher arguments. It is read-only and
fails on configuration drift instead of repairing it implicitly.

Client launchers contain the absolute path of a verified managed runtime and
the fixed arguments `mcp serve --project-root .`. Setup never passes `--init`.
the fixed arguments `mcp serve --project-root .`. Setup never passes `--init` or
an external workspace; hosts that need one can add `--workspace PATH` to an
explicit server invocation.
Each MCP client therefore binds flameox's project root to the client's launch
directory. Client setup is complete when the launcher is verified; each checkout
still requires deliberate project initialization through `flameox init`,
Expand Down Expand Up @@ -236,8 +238,8 @@ manifest.
### MCP

```text
flameox mcp serve [--init]
flameox mcp inspect
flameox mcp serve [--init] [--workspace PATH]
flameox mcp inspect [--workspace PATH]
```

`serve` uses stdio exclusively. `--init` performs the additive workspace
Expand Down Expand Up @@ -386,11 +388,12 @@ execution explicitly. Planning still does not execute the workload;

#### `initialize_workspace`

Additive and idempotent. Initializes only the MCP server's fixed project root.
Additive and idempotent. Initializes only the MCP server's fixed project root,
or the explicit workspace root selected when the server started.
If the server already owns an initialized workspace, the call returns its current
status without replacing the detached-capture manager. It cannot select an
external path or configure workloads. Hosts may instead start the server using
`flameox mcp serve --init`.
status without replacing the detached-capture manager. It cannot switch to a
different path or configure workloads. Hosts may instead start the server using
`flameox mcp serve --init --workspace PATH`.

#### `workload_configuration_status`

Expand Down Expand Up @@ -459,7 +462,10 @@ request cleanup. The operation accepts only adapter names reported by
`list_capabilities` as managed setup actions: `coverage`, `memray`, `perfetto`,
`py-spy`, `pytest`, and `torch.profiler`. It installs the published FlameOx
extra into the active managed Python runtime and stages the pinned user-space
Trace Processor under `.diagnostics/tools` for `perfetto`. Its receipt includes
Trace Processor under the active workspace's `tools/` directory for `perfetto`.
Setup progress uses `validating_request`, `installing_packages`,
`staging_trace_processor`, `verifying`, and `completed`; a staging failure keeps
the staging phase and bounded cause in its durable status. Its receipt includes
the workspace identity, exact request digest, named phase, bounded progress,
item outcomes, cancellation/cleanup state, and next recovery action. It never
runs the declared workload, mutates source, installs arbitrary packages,
Expand Down
8 changes: 5 additions & 3 deletions docs/storage-and-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@ If none exists, commands that mutate state fail with a remediation suggesting

An explicit `--workspace` overrides discovery. The resolved workspace must be
inside the selected project root unless the user explicitly supplies an
external absolute path through the CLI. MCP tools cannot choose an arbitrary
external workspace, but artifact import explicitly permits the fixed project
root or the system temporary directory as bounded source roots.
external path through the CLI or MCP server startup. The project root remains
the workload and source root when an external workspace is selected. MCP tools
cannot switch workspace roots after startup, but artifact import explicitly
permits the fixed project root or the system temporary directory as bounded
source roots.

In a Git repository, `flameox init` adds `.diagnostics/` to
`.git/info/exclude` when it is not already ignored. It does not edit the
Expand Down
127 changes: 121 additions & 6 deletions src/flameox/adapters/pytest.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ class PytestExtractionResult(ContractModel):
executed_count: int
passed_count: int
failed_count: int
environment_blocked_count: int
Comment thread
morluto marked this conversation as resolved.
skipped_count: int
errored_count: int
unexecuted_count: int
Expand Down Expand Up @@ -168,6 +169,20 @@ def extract(self, run_id: str) -> PytestExtractionResult:
outcome_counts = _test_outcomes(phases)
executed = set(phases)
unexecuted = collected - executed
compiler_diagnostic = self._external_compiler_diagnostic(run)
environment_blocked_nodes, environment_blocked_count, classified_counts = (
self._classify_external_compiler_outcomes(
phases,
unexecuted,
compiler_diagnostic is not None,
)
)
self._apply_external_compiler_evidence(
rows,
limitations,
environment_blocked_nodes,
compiler_diagnostic,
)
first_observed = min(failure_observed, default=None)
first_reported = min(failure_reported, default=None)
collection_started = [
Expand All @@ -189,9 +204,10 @@ def extract(self, run_id: str) -> PytestExtractionResult:
("pytest.tests.collected", len(collected)),
("pytest.tests.executed", len(executed)),
("pytest.tests.passed", outcome_counts["passed"]),
("pytest.tests.failed", outcome_counts["failed"]),
("pytest.tests.skipped", outcome_counts["skipped"]),
("pytest.tests.errored", outcome_counts["errored"]),
("pytest.tests.failed", classified_counts["failed"]),
("pytest.tests.environment_blocked", environment_blocked_count),
("pytest.tests.skipped", classified_counts["skipped"]),
("pytest.tests.errored", classified_counts["errored"]),
("pytest.tests.unexecuted", len(unexecuted)),
("pytest.fixture_setup.total", fixture_setup_ns),
):
Expand Down Expand Up @@ -273,9 +289,10 @@ def extract(self, run_id: str) -> PytestExtractionResult:
collected_count=len(collected),
executed_count=len(executed),
passed_count=outcome_counts["passed"],
failed_count=outcome_counts["failed"],
skipped_count=outcome_counts["skipped"],
errored_count=outcome_counts["errored"],
failed_count=classified_counts["failed"],
environment_blocked_count=environment_blocked_count,
skipped_count=classified_counts["skipped"],
errored_count=classified_counts["errored"],
unexecuted_count=len(unexecuted),
fixture_setup_count=fixture_setup_count,
fixture_setup_ns=fixture_setup_ns,
Expand All @@ -291,6 +308,92 @@ def extract(self, run_id: str) -> PytestExtractionResult:
limitations=tuple(limitations),
)

def _external_compiler_diagnostic(self, run: RunManifest) -> str | None:
outputs: list[str] = []
for registration in run.artifacts:
if registration.kind is not ArtifactKind.PROCESS_OUTPUT:
continue
if registration.role not in {"stdout", "stderr"}:
continue
try:
payload = ArtifactStore(self.workspace).get(registration.artifact_id)
outputs.append(
payload.payload_path.read_bytes()[: 64 * 1024].decode(errors="replace")
)
Comment thread
morluto marked this conversation as resolved.
except (DomainError, OSError, UnicodeError):
continue
combined = " ".join(outputs)
lowered = combined.casefold()
markers = (
"cuda_runtime.h: no such file or directory",
"fatal error: cuda_runtime.h",
"nvcc fatal",
"cannot find -lcudart",
"hip_runtime.h: no such file or directory",
"clang: error: no such file or directory",
"gcc: fatal error",
)
if not any(marker in lowered for marker in markers):
return None
return (
" ".join(combined.split())[:500] or "External compiler returned no diagnostic detail."
)
Comment thread
morluto marked this conversation as resolved.

@staticmethod
def _apply_external_compiler_evidence(
rows: list[dict[str, Any]],
limitations: list[str],
environment_blocked_nodes: set[str],
compiler_diagnostic: str | None,
) -> None:
if environment_blocked_nodes:
for row in rows:
dimensions = row.get("dimensions")
nodeid = dimensions.get("nodeid") if isinstance(dimensions, dict) else None
if nodeid in environment_blocked_nodes:
assert isinstance(dimensions, dict)
dimensions["classification"] = "environment_blocked"
dimensions["original_outcome"] = dimensions.get("outcome", "unknown")
Comment thread
morluto marked this conversation as resolved.
if compiler_diagnostic is not None:
limitations.extend(
(
"An external compiler prerequisite failure was detected; affected pytest "
"outcomes are environment-blocked evidence, not a confirmed application "
"defect.",
f"External compiler diagnostic: {compiler_diagnostic}",
)
)

@staticmethod
def _classify_external_compiler_outcomes(
phases: dict[str, dict[str, str]],
unexecuted: set[str],
compiler_failed: bool,
) -> tuple[set[str], int, dict[str, int]]:
outcome_counts = _test_outcomes(phases)
environment_blocked_nodes = {
nodeid
for nodeid, reports in phases.items()
if compiler_failed and _test_outcome(reports) in {"failed", "errored"}
}
Comment thread
morluto marked this conversation as resolved.
environment_blocked_count = len(environment_blocked_nodes)
if compiler_failed and not environment_blocked_nodes:
environment_blocked_count = len(unexecuted)
blocked_outcomes = {
outcome: sum(
1
for nodeid in environment_blocked_nodes
if _test_outcome(phases[nodeid]) == outcome
)
for outcome in ("failed", "errored")
}
classified_counts = {
**outcome_counts,
"failed": outcome_counts["failed"] - blocked_outcomes["failed"],
"errored": outcome_counts["errored"] - blocked_outcomes["errored"],
}
return environment_blocked_nodes, environment_blocked_count, classified_counts

def _registration(self, run: RunManifest) -> Any:
matches = [item for item in run.artifacts if item.kind is ArtifactKind.TEST_EXECUTION]
if len(matches) != 1:
Expand Down Expand Up @@ -391,6 +494,18 @@ def _test_outcomes(phases: dict[str, dict[str, str]]) -> dict[str, int]:
return counts


def _test_outcome(reports: dict[str, str]) -> str:
if reports.get("setup") == "failed" or reports.get("teardown") == "failed":
return "errored"
if reports.get("call") == "failed":
return "failed"
if "skipped" in reports.values():
return "skipped"
if reports.get("call") == "passed":
return "passed"
return "errored"


def _completion(
events: list[dict[str, Any]],
*,
Expand Down
57 changes: 54 additions & 3 deletions src/flameox/adapters/setup_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -316,14 +316,21 @@ def install_trace_processor(
)
atomic_write_text(workspace.paths.config, updated.to_toml())
temporary = None
except DomainError:
raise
except DomainError as exc:
raise _annotate_staging_error(exc) from exc
Comment thread
morluto marked this conversation as resolved.
except (OSError, urllib.error.URLError, ValueError) as exc:
category = _staging_failure_category(exc)
detail = _bounded_staging_detail(exc)
raise DomainError(
ErrorCode.PROCESS_FAILED,
"FlameOx could not stage the managed Trace Processor.",
retryable=True,
details={"next_tool": "prepare_capabilities", "adapter": "perfetto"},
details={
"next_tool": "prepare_capabilities",
"adapter": "perfetto",
"failure_category": category,
"failure_detail": detail,
},
remediation=(
"Retry prepare_capabilities; if the download remains unavailable, install "
"the official user-space binary or configure analysis.trace_processor_path.",
Expand All @@ -335,6 +342,50 @@ def install_trace_processor(
return TraceProcessorInstallation(TRACE_PROCESSOR_VERSION, target, True)


def _annotate_staging_error(error: DomainError) -> DomainError:
"""Retain a bounded cause when a staging helper already raised a domain error."""
details = dict(error.details)
details.setdefault("failure_category", _domain_failure_category(error))
details.setdefault(
"failure_detail",
_bounded_staging_detail(details.get("error") or error.message),
)
details.setdefault("phase", "staging_trace_processor")
return DomainError(
error.code,
error.message,
retryable=error.retryable,
details=details,
remediation=error.remediation,
run_id=error.run_id,
)


def _domain_failure_category(error: DomainError) -> str:
if error.code is ErrorCode.PROCESS_CANCELLED:
return "cancelled"
if error.code is ErrorCode.PROCESS_TIMEOUT:
return "timeout"
if error.code is ErrorCode.ARTIFACT_TOO_LARGE:
return "download_limit"
if error.code is ErrorCode.CAPABILITY_UNAVAILABLE:
return "unsupported_platform"
return "verification"


def _staging_failure_category(error: BaseException) -> str:
if isinstance(error, urllib.error.URLError):
return "network"
if isinstance(error, OSError):
return "filesystem"
return "verification"


def _bounded_staging_detail(error: object) -> str:
detail = " ".join(str(error).split())
return detail[:500] or "The staging operation returned no diagnostic detail."


def _verify_trace_processor(
executable: Path,
*,
Expand Down
Loading
Loading