Add Data Loop denial diagnostics

[Richie Gomez (richiemsft)]

📖 Description

Adds a deterministic Data Loop JSON sibling for every successful Learning Mode denial analysis while leaving canonical denials.json filtering and policy behavior unchanged.

One ETL analysis now produces both views:

• denials.json retains the existing canonical, actionable denials.
• denials.data-loop.json retains the bounded, deduplicated superset, including events excluded as unusableResourcePath, unsupported object types, malformed payloads, decoder limits, and unsupported event schemas.

Long diagnostic values remain individually identifiable with a bounded prefix, SHA-256 digest, and suffix. Identity and username redaction occurs before hashing or retaining context.

The guarded WPR path keeps known-provider event ordinals aligned through selection and replay. Brokered capability event 28 is scoped using its payload ProcessId, including partial-decode recovery, rather than the broker header PID.

Native capture and plm stop share a transactional paired-output writer. Both JSON files are staged and synchronized before promotion, serialized by a per-pair lock, promoted without clobbering, and rolled back with file-identity and content-digest ownership checks. PLM collision validation includes the derived Data Loop sibling.

Namespace ETL validation confirms the previously reported MXC output gaps remain identifiable in Data Loop output:

• Devices: 1/1
• Base Named Objects: 1/1
• Shell Namespaces: 11/11
• Scheduled Tasks: 1/1
• Event Log Channels: 3/3

🔗 References

Stacked on #847.

🔍 Validation

• learning_mode_core: 39 tests passed
• learning_mode_windows: 218 tests passed
• appcontainer_common: 243 tests passed
• plm: 224 tests passed, 1 ignored
• Clippy passed with warnings denied across all four affected crates
• Release wxc-exec.exe and plm.exe build
• Guarded-WPR VM validation with canonical and typed Data Loop groups, zero overflow, and no username leakage
• Nine-axis adversarial review completed with zero unresolved High or Medium findings

✅ Checklist

• Signed the Contributor License Agreement
• Linked to an issue
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)
• If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

• Bug fix
• Feature
• Task

---

GitHub Actions runs the PR validation build automatically. The ADO pipeline
(MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHub
Actions build; it runs on merge to main, and Microsoft reviewers with write access can trigger it
on a PR with /azp run. See docs/pull-requests.md.

If the dependency-feed-check check fails on a new dependency, the crate must be added to
the feed before the PR can pass. See docs/pull-requests.md
for the steps.

Microsoft Reviewers: Open in CodeFlow

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds bounded, redacted Data Loop diagnostics alongside Learning Mode denial output and integrates them across native capture, guarded WPR, and audit workflows.

Changes:

• Adds the Data Loop model, aggregation, sanitization, and paired output.
• Adds process-scoped guarded-WPR ETL relogging and retention.
• Reworks --audit to consume captureDenials artifacts directly.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.github/copilot-instructions.md	Updates Learning Mode architecture guidance.
README.md	Revises audit-mode documentation.
docs/learning-mode/capabilities.md	Documents Data Loop and retained ETL behavior.
src/Cargo.lock	Records the new dependency edge.
src/backends/appcontainer/common/src/appcontainer_runner.rs	Supports guarded ETL transfer.
src/backends/appcontainer/common/src/base_container_runner.rs	Integrates retained guarded captures.
src/backends/appcontainer/common/src/capture_output.rs	Adds paired denial/Data Loop output.
src/backends/appcontainer/common/src/guarded_capture.rs	Extends guarded-capture interfaces.
src/backends/learning_mode/windows/src/capability_dacl.rs	Adds diagnostic metadata to recovered denials.
src/backends/learning_mode/windows/src/etl_decode.rs	Aggregates Data Loop outcomes and selects relog events.
src/backends/learning_mode/windows/src/etl_filter.rs	Adds process-scoped ETL relogging.
src/backends/learning_mode/windows/src/extractors.rs	Adds sanitization and diagnostic classification.
src/backends/learning_mode/windows/src/lib.rs	Exports ETL filtering.
src/backends/learning_mode/windows/src/tdh_decode.rs	Categorizes recoverable decoding failures.
src/core/learning_mode_core/src/analyze.rs	Carries and bounds Data Loop results.
src/core/learning_mode_core/src/data_loop.rs	Defines the Data Loop model and writer.
src/core/learning_mode_core/src/lib.rs	Exports Data Loop APIs.
src/core/learning_mode_core/src/model.rs	Makes classifications sortable.
src/core/mxc_engine/src/guarded_capture.rs	Implements guarded trace transfer.
src/core/mxc_engine/src/lib.rs	Exports audit runner resolution.
src/core/mxc_engine/src/run.rs	Adds audit-specific resolver entry point.
src/core/wxc/Cargo.toml	Adds Learning Mode core dependency.
src/core/wxc/src/audit.rs	Relocates and post-processes capture artifacts.
src/core/wxc/src/main.rs	Routes audit through captureDenials.
src/host/plm/readme.md	Documents guarded analysis and validation.
src/host/plm/src/analysis.rs	Returns canonical documents after writing.
src/host/plm/src/elevated.rs	Adds filtered analysis/trace transfer protocol.
src/host/plm/src/log.rs	Consumes guardian-returned analysis directly.
src/host/plm/src/secure_scratch.rs	Manages filtered ETL scratch storage.
src/host/plm/src/stop.rs	Extracts canonical-denial post-processing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/core/learning_mode_core/src/data_loop.rs:184

• Once the signature set contains only canonical groups, every new distinct canonical event calls evict_one_noncanonical_group, which scans all 4,096 entries before overflowing it. At the 1,000,000-event bound this can perform billions of comparisons, turning an intentionally bounded diagnostic path into a CPU bottleneck. Track whether any noncanonical group exists (or partition/index the two classes) so the full scan is skipped when eviction cannot succeed.

        while self.signatures.len() >= MAX_DATA_LOOP_GROUPS
            || retained_bytes.saturating_add(serialized_len) > max_bytes
        {
            if !signature.reason.is_canonical_denial()
                || !self.evict_one_noncanonical_group(Some(retained_bytes))
            {

src/host/plm/src/elevated.rs:2341

• This helper is now also used for guarded captureDenials retention, so create_dir_all makes a missing outputPath parent succeed only when guarded WPR plus retainEtl is selected. Native and non-retained captures still fail, and the documented contract says the parent must already exist. Preflight the configured output directory consistently before capture, rather than creating it as a side effect of ETL transfer.
  src/core/learning_mode_core/src/paired_output.rs:56
• Every capture creates a unique .pair.lock file, but Drop only unlocks it and never removes it. Because capture output names include a per-run ID, these lock files are never reused and accumulate indefinitely in the configured output directory (or the system temp directory for managed outputs). Use an ephemeral/delete-on-close lock strategy, or add race-safe cleanup after the transaction.

impl Drop for OutputPairLock {
    fn drop(&mut self) {
        let _ = std::fs::File::unlock(&self.file);
    }

.github/copilot-instructions.md:216

• This architecture entry still says the output layer writes a single JSON document, but this PR makes learning_mode_core own the DataLoopDocument and paired-output transaction as well. Leaving the repository guidance singular will mislead future changes about the required canonical/Data Loop pair; update this entry to describe both artifacts and their paired emission contract.

- `learning_mode_core` is the **cross-platform learning-mode / captureDenials model + output emitter**: `DeniedResource` (+ `ResourceType`/`AccessType`), `DenialSummary`, the `DenialAnalyzer` decode trait, and `emit` — which writes the on-disk denials deliverable as a **single JSON document** `{ "denials": [...], "summary": {...} }` (`write_document` / `DenialsDocument`) and defines the serializable `DenialsOutputPointer`. It carries no OS-specific code (must not depend on any `backends/*` crate); the Windows ETL decoder implementing `DenialAnalyzer` lives in `backends/learning_mode/windows`. When `processContainer.captureDenials` is set, native PSEC/V2 seals and decodes a managed ETL locally, while guarded WPR relogs its host-wide source into a process-scoped ETL before analysis; both routes write the same canonical JSON through shared output plumbing and return neutral `wxc_common` metadata. Explicit `retainEtl` preserves the native sealed trace or the guarded process-scoped relogged trace after a terminal wait; abandonment discards it. `wxc-exec` serializes the metadata as the one-line stderr pointer at the CLI boundary; Rust/C#/FFI callers receive it programmatically. Each denial's `resource` field holds the file path or the AppContainer capability name; capability denials resolve their capability SID to a friendly name via `backends/learning_mode/windows`'s `capability_names` (well-known `S-1-15-3-…` SID → policy name; custom hashed SIDs fall back to the SID string).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/core/learning_mode_core/src/paired_output.rs:118

• CreateNew is the captureDenials path, whose canonical filename contains a fresh run ID, but acquiring this lock creates a correspondingly unique .pair.lock file and Drop only unlocks it. Every capture therefore leaves another permanent lock file in the output directory. The no-clobber promotions already serialize competing CreateNew writers, so avoid creating a lock for that policy and retain the stable lock only for Replace.

    let _pair_lock = OutputPairLock::acquire(operation, canonical_path)?;

src/core/wxc/src/audit.rs:203

• If this second move fails, capture.output_path already points to final_denials, but its implicitly derived Data Loop sibling remains beside source_denials. Because the Data Loop path is intentionally omitted from metadata, the metadata is no longer truthful and callers cannot locate the sibling after this error. Relocate the canonical/Data Loop pair transactionally (including rollback and metadata restoration) before moving the ETL.

    move_file(source_data_loop, final_data_loop)?;

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/core/wxc/src/audit.rs:224

• This cleanup can delete another audit's destination. A concurrent writer may create destination after the preflight check; if copy then fails, remove_file(destination) has no ownership proof. std::fs::copy can also truncate an existing destination, so the fallback is not actually no-clobber. Copy into an owned temporary file and promote it with create-new/no-clobber semantics, cleaning up only that temporary file.

        return Ok(());
    }
    ensure_destination_available(source, destination)?;
    match std::fs::rename(source, destination) {

src/core/learning_mode_core/src/lib.rs:17

• This changes learning_mode_core from the documented single-document emission model to a paired canonical/Data Loop contract and adds shared transactional output behavior, but .github/copilot-instructions.md still describes only one JSON document. Update that repository architecture guidance in this PR so future work does not bypass the required sibling output.

//!    written to the canonical JSON output that host applications read to
//!    regenerate policy. A deterministic Data Loop sibling contains bounded,
//!    sensitive-value-redacted signatures for excluded decoder outcomes.

src/backends/learning_mode/windows/src/etl_decode.rs:332

• At the 24-property limit, the newly inserted resource entry can be truncated immediately: the properties are sorted by BTreeMap, and lowercase resource sorts after the usual capitalized TDH property names. Distinct capability/UI denials then lose their resolved identity and collapse into the same Data Loop signature. Reserve a slot for this synthetic identity (or model it outside the bounded property list) before applying bound_properties.

            properties.insert(
                "resource".to_string(),
                if raw.resource_type == learning_mode_core::ResourceType::File {
                    crate::extractors::REDACTED_PATH.to_string()
                } else {
                    resource.to_string()
                },
            );
        }
        let properties =
            crate::extractors::bound_properties(properties.into_iter().collect::<Vec<_>>());

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/backends/learning_mode/windows/src/etl_decode.rs:576

• This helper charges each fixture here and then charges it again inside handle_decoded_event when the event is in scope. It therefore reaches the processing limit at half the production event count and even charges fixtures that production rejects before begin_event, so tests using this helper do not mirror the real callback's accounting.

    for event in events {
        if !accumulator.begin_event() {
            break;
        }
        handle_decoded_event(&event.parts, event.pid, event.filetime, &mut accumulator);

src/core/learning_mode_core/src/data_loop.rs:185

• Once the retained table contains only canonical groups, every additional distinct canonical outcome calls evict_one_noncanonical_group, which reverse-scans all 4,096 entries before overflowing. Since analysis now continues up to 1,000,000 events after the canonical limit, an adversarial trace can cause roughly four billion group comparisons. Track whether any noncanonical groups remain (or maintain them separately) so the all-canonical overflow path is O(1).

            if !signature.reason.is_canonical_denial()
                || !self.evict_one_noncanonical_group(Some(retained_bytes))
            {
                self.record_overflow(signature.reason.is_canonical_denial());