TPM Tooling gaps for Intelligent Systems Delivery

[namitts82]

As we all have realised, Intelligent systems don't behave like traditional software projects. Outcomes are probabilistic, model performance shifts over time, and progress can't be measured by task completion alone. Yet most TPM tooling (scoping, scheduling, risk tracking, RAG dashboards) still assumes deterministic delivery. This creates a structural gap: projects report green while decisions remain unsafe. We've been working on closing that gap with tools and processes built specifically for probabilistic delivery.

What We Are Doing

We have kicked off an initiative to build TPM tooling specifically for intelligent systems delivery. We studied how Google, Microsoft, and AWS define production readiness for ML systems and mapped where standard TPM practices hold up and where they break down. From that research, we are redefining 10 core TPM disciplines (scope, schedule, risk, quality, progress tracking, dependencies, stakeholder management, definition of done, operational readiness, reporting and change control) for environments where behaviour is learned from data and adapts to context. Each discipline gets purpose built tools and processes. Like, confidence dashboards instead of RAG status, hypothesis driven scope packs instead of fixed requirements, timeboxed learning cycles instead of milestone driven schedules, kill criteria and risk heatmaps that account for drift and silent failure modes, etc.

Where We Are

Research across three industry frameworks is complete. Ten TPM responsibility areas have been redefined with evidence based rationale. Twenty seven tools and processes have been documented with structure, templates, and alignment to the responsibility model. These include artifacts like the Confidence Dashboard, Decision Ledger, Uncertainty Framing Contract, Quality Assurance Pack (covering data, model, system, and drift gates), Model Operations Readiness Checklist, and Monitoring Signal Map. Everything is catalogued and queryable. Next step is piloting on live engagements to validate what works.

What's Next

We are hoping to integrate this as an accelerator into HVE Core so it becomes part of the standard toolkit available to all teams. Before that, we need to pilot on real engagements and iterate based on feedback.

If this problem resonates with you or you want to help shape the tooling, reach out to any of us. We welcome contributors, reviewers, and teams willing to pilot.

Namit (namit.t@microsoft.com)
David (david.ratcliffe@microsoft.com)
Parag (paragalurkar@microsoft.com)

[hodnick]

Namit, David, Parag, this resonates. I like the framing because it puts the TPM role where it belongs for intelligent systems: helping the team make safe decisions under uncertainty, not just reporting task progress.

A project can be green on scope, schedule, and risks while still being unsafe if the team has not earned confidence in model behavior, data quality, evaluation coverage, operator workflow, rollback posture, or drift detection.

A few additions I would consider as you move toward pilots:

1. Tie confidence to decisions: A confidence dashboard should name the decision it supports: demo to leadership, expand a pilot, remove human review, increase traffic, commit to a customer milestone, or move to production. Confidence should also have a shelf life. If the evidence ages, the model changes, or the operating context shifts, the status should decay until refreshed

2. Make the MVE pattern explicit: In HVE practice we already use Minimum Viable Experiments to test the riskiest assumption before building the full solution. I would make that a formal TPM scoping pattern for intelligent systems: hypothesis, evidence needed, timebox, pass/fail threshold, kill criteria, and owner

3. Plan evaluation capacity like delivery capacity: We are getting better at asking, “how much code can we meaningfully review today?” For intelligent systems, the companion planning question is, “how much model behavior can we meaningfully evaluate this week?” If evaluation capacity is finite, it should shape schedule and scope up front

4. Measure feedback loop velocity: I would track the time from production signal to evaluation dataset to prompt/model/system change to redeploy. That is closer to the heartbeat of an intelligent system than story completion or milestone percent complete

5. Watch context drift: Prompts, model versions, evaluation sets, ADRs, business rules, telemetry assumptions, and customer workflows all age. A lightweight context freshness check may be as important as a model drift check, especially in HVE where markdown becomes working memory for both humans and agents

The caution I would add is process weight. We should pilot the smallest useful pack first, probably the Confidence Dashboard, Decision Ledger, Uncertainty Framing Contract, and one evaluation/readiness artifact. If the tooling helps a crew make a safer decision faster, it belongs in HVE Core. If it mostly creates another status surface, it will give back the gains we are trying to protect.

I would be happy to help review or pilot from the TPM Guild side.

[namitts82]

Thanks for the thoughtful response. This is exactly the kind of feedback we were hoping for.

I really like your additions, especially the points on tying confidence to specific decisions, making MVE a formal scoping pattern, and treating evaluation capacity as a real delivery constraint. The feedback loop velocity and context freshness ideas also resonate strongly, because they get much closer to how these systems actually evolve in practice.

I also agree with your caution on process weight. The intent is not to create another status surface, but to help teams make safer decisions faster. Starting with the smallest useful pack makes a lot of sense.

We are currently exploring possibilities for integrating the code and artifacts into the HVE Core repo so this can become easier to pilot and iterate on with real crews. We will definitely seek your help from the TPM Guild side as we shape this up further.

Really appreciate you leaning in. We’ll keep you posted as we forward.

[musaabhasan]

The TPM gap is real because intelligent systems can be "on schedule" while still being operationally unsafe. I would model project health around evidence maturity, not only task completion.

A useful delivery dashboard could track:

• capability readiness: does the system perform the intended task across representative scenarios?
• evaluation readiness: are datasets, rubrics, baselines, and regression gates versioned?
• safety readiness: are abuse cases, policy boundaries, and escalation paths tested?
• observability readiness: can the team trace failures to prompt, model, retrieval, tool, or policy changes?
• drift readiness: is there a process for monitoring behavior after release?
• human readiness: are reviewers, support teams, and operators able to interpret failures?

For TPM reporting, I would make each milestone include a confidence statement: what evidence changed, what uncertainty remains, and what decision the evidence supports. That creates a better language for probabilistic delivery than green/yellow/red status alone.

[namitts82]

Thanks Musaab, this is a great framing. I completely agree that project health for intelligent systems needs to move beyond task completion and toward evidence maturity.

Your readiness breakdown is very useful. Capability, evaluation, safety, observability, drift, and human readiness are exactly the kinds of dimensions that can expose risk even when the project looks green from a traditional scope or schedule view.

I also really like the milestone confidence statement idea: what evidence changed, what uncertainty remains, and what decision the evidence now supports. That is very aligned with where we are trying to take the Confidence Dashboard and Decision Ledger patterns.

We are exploring how to integrate these artifacts into HVE Core and make them practical for pilots without adding too much process weight. I’ll fold this feedback into the working model, and we’d value your input as we shape it further.

[WilliamBerryiii]

@namitts82, David, Parag — thank you for opening this up. The framing lands, and the comments from Sam and Musaab on tying confidence to specific decisions, MVE as a formal scoping pattern, evaluation capacity as a delivery constraint, and evidence-maturity readiness are exactly the shape we'd want HVE Core to absorb.

Before we look at integration mechanics, we'd like to ask the three of you for a generic, OSS-postable BRD/PRD we can review and discuss in the open. A public artifact lets the broader community (and downstream forks) reason about fit, contribute, and pilot without back-channel context. A few prompts to shape it, not constrain it:

Inputs and dependencies

• What artifacts does the toolkit consume? Are any HVE Core agents (Task Researcher, Task Planner, Code Reviewer, RAI Planner, Security Planner, SSSC Planner) expected as upstream producers, and if so, which are hard prerequisites versus optional enrichers?
• Does it pull from a backlog (ADO, GitHub Issues, Jira), and if so, is the integration read-only for context, write-back for work items, or both? Our existing backlog agents (ado-*, github-backlog-*, jira-*) follow a discovery → planning → handoff pattern that may be worth aligning with.
• What's the minimum useful input set for the smallest pilot pack (Confidence Dashboard, Decision Ledger, Uncertainty Framing Contract, plus one readiness artifact)?

Outputs and surfaces

• Are outputs primarily human-readable reports, machine-readable artifacts under .copilot-tracking/, or work items handed off to a backlog manager? Each surface has different durability and review expectations.
• Where does the Confidence Dashboard live? Repo-resident markdown, a generated report, or a backlog field projection?

RAI posture

• We'd like to understand how the toolkit treats scoring. We've deliberately avoided composite risk scores in the RAI Planner because they tend to compress qualitative signals into a number that gets optimized rather than discussed. If confidence dashboards expose numeric thresholds, how are they framed to support a decision rather than become a status surface (echoing Sam's caution on process weight)?
• Is the RAI Planner a recommended companion or a hard prerequisite for any artifact that touches model behavior, evaluation coverage, or operator workflow?

Open standards alignment

• Which open industry frameworks shape the report structure? Candidates worth naming explicitly: NIST AI RMF 1.0 (Govern, Map, Measure, Manage), ISO/IEC 42001 (AI management system), ISO/IEC 23894 (AI risk), Google PAIR / Model Cards, and the in-flight evaluation work from MLCommons. Anchoring to one or more lets contributors map findings into frameworks they already report against.
• For evidence and provenance, are you considering in-toto attestations or SLSA-style provenance for evaluation runs and decision ledger entries?

Chain of custody

• Should toolkit artifacts (Decision Ledger entries, Confidence Dashboard snapshots, Quality Assurance Pack outputs, Model Operations Readiness checklists) be eligible for keyless signing the way our planners are? The RAI Planner ends with an optional npm run rai:sign that produces a SHA-256 manifest and Sigstore bundle, and our releases ship Sigstore-attested SPDX SBOMs. Treating evidence artifacts as signable would give pilots a tamper-evident record of what evidence supported which decision at which point in time, which matches the "shelf life" idea Sam raised about confidence decay.

Architecture fit

Please fold integration mechanics into the BRD/PRD itself rather than landing implementation first. Two specifics worth covering in that document:

• How each of the 27 tools maps to HVE Core's existing primitives (skill, prompt, instruction, agent, or subagent) under the .github/{skills,prompts,instructions,agents}/{collection-id}/ convention. Naming the primitive up front lets the toolkit inherit the standard validation, packaging, and signing pipelines instead of bolting them on later.
• Where the toolkit composes with existing agents and subagents rather than duplicating them. Task Researcher, Task Planner, Code Reviewer, RAI Planner, Security Planner, SSSC Planner, the backlog agents (ado-*, github-backlog-*, jira-*), the DT Coach, and the shared Researcher Subagent all already cover adjacent ground. The architecture stays consistent when new tools call into these (or are called by them) instead of standing alone. For example, evaluation readiness signals could flow through the RAI Planner's evidence register, and pilot work items could hand off through the existing backlog handoff contracts.

Once the BRD/PRD lands, an experimental collection (collections/experimental.collection.yml) is a low-friction landing zone with explicit experimental-maturity expectations, which lets pilots move fast without locking the API while the integration shape settles.

Happy to review a draft when you have one, and to help shape the smallest pilot pack into the repo conventions.

[namitts82]

Good day @WilliamBerryiii,

We took some time to finalize it, but the BRD is now ready. Could you please review it and share your feedback when you get a chance?

Foglight BRD Pull Request