diff --git a/README.md b/README.md index cda7b50..57a84f5 100644 --- a/README.md +++ b/README.md @@ -640,9 +640,20 @@ IDs, labels, annotations, workload identity, endpoints, and unknown fields. Any partial, duplicate-key, identity/window-mismatched, oversized, or invalid-total response is rejected as a whole and emits zero facts; invalid rows are never filtered into a partial success. A successful empty allocation map also emits zero facts, while missing OpenCost coverage is never estimated. This -library path adds no OpenCost client, service discovery, credentials, persistence, -billing, optimization, mutation, fleet/team rollup, currency conversion, freshness objective, or -GPU-utilization claim. The current CLI and Hub do not fetch or display these facts yet. +package can preserve each successful projection in a per-scope snapshot, including an empty fact +set, and combine snapshots for one exact window into a deterministic workspace USD total. The +caller supplies the complete expected-scope set; output names every expected, reported, +successful-empty, and missing scope, and missing scopes never contribute synthetic zero cost. +Every fact is revalidated before all component and total amounts are summed with exact decimal +arithmetic. A rollup carries the source window end only when at least one scope reported and selects +no stale threshold. + +The rollup is an offline workspace computation core, not a live Hub feature. This library path adds +no OpenCost client, port-forward, service or ingress discovery, arbitrary endpoint, credentials, +Kubernetes Service-proxy RBAC, OCM transport, persistence, runtime wiring, per-team attribution, +UI/API, billing, optimization, mutation, currency conversion, freshness objective, or +GPU-utilization claim. The current CLI and Hub do not fetch, persist, roll up, or display these +facts yet. Every verdict includes its rule, exact cited signals, confidence state, missing lenses, and an advisory command or PR change for the operator to inspect and run. The brain performs no I/O and diff --git a/docs/EPICS.md b/docs/EPICS.md index e41c10d..36a7948 100644 --- a/docs/EPICS.md +++ b/docs/EPICS.md @@ -3174,6 +3174,18 @@ flowchart TD **Key risk / guardrail.** A partial rollup read as complete. Guardrail: coverage is always shown. +**Current bounded slice (F13.2a, #284).** `internal/connector/opencost` preserves every successful +F13.1a projection in a per-scope snapshot, including a complete empty allocation set, and computes +one deterministic workspace USD total for an exact caller-bound window. The caller supplies the +unique expected cluster set; output separately names expected, reported, successful-empty, and +missing scopes, so missing OpenCost coverage never becomes zero cost or a complete rollup. Every +fact is revalidated against workspace, cluster, namespace, window, currency, lens, provenance, +canonical payload, and native identity before all monetary components and totals are summed with +exact decimal arithmetic. The rollup uses the window end as observation time only when at least one +scope reported. It adds no live transport, endpoint, credential, persistence, Hub/runtime wiring, +team/label attribution, UI, stale threshold, conversion, billing, optimization, GPU-efficiency +inference, or write path, and therefore does not complete F13.2. + ### F13.3 — GPU cost columns (DCGM) **What it is.** GPU cost/utilization columns in the fleet cost view where DCGM metrics exist — diff --git a/docs/SITH-NOTION.md b/docs/SITH-NOTION.md index 37f7208..51ba8ed 100644 --- a/docs/SITH-NOTION.md +++ b/docs/SITH-NOTION.md @@ -3087,6 +3087,18 @@ flowchart TD **Key risk / guardrail.** A partial rollup read as complete. Guardrail: coverage is always shown. +**Current bounded slice (F13.2a, #284).** `internal/connector/opencost` preserves every successful +F13.1a projection in a per-scope snapshot, including a complete empty allocation set, and computes +one deterministic workspace USD total for an exact caller-bound window. The caller supplies the +unique expected cluster set; output separately names expected, reported, successful-empty, and +missing scopes, so missing OpenCost coverage never becomes zero cost or a complete rollup. Every +fact is revalidated against workspace, cluster, namespace, window, currency, lens, provenance, +canonical payload, and native identity before all monetary components and totals are summed with +exact decimal arithmetic. The rollup uses the window end as observation time only when at least one +scope reported. It adds no live transport, endpoint, credential, persistence, Hub/runtime wiring, +team/label attribution, UI, stale threshold, conversion, billing, optimization, GPU-efficiency +inference, or write path, and therefore does not complete F13.2. + ### F13.3 — GPU cost columns (DCGM) **What it is.** GPU cost/utilization columns in the fleet cost view where DCGM metrics exist — diff --git a/docs/adr/0012-opencost-coverage-aware-workspace-rollup.md b/docs/adr/0012-opencost-coverage-aware-workspace-rollup.md new file mode 100644 index 0000000..8ad1743 --- /dev/null +++ b/docs/adr/0012-opencost-coverage-aware-workspace-rollup.md @@ -0,0 +1,90 @@ +# ADR 0012: Coverage-aware workspace rollup for OpenCost cost facts + +**Status:** Accepted +**Date:** 2026-07-18 +**Decision owners:** E13 / F13.2a ([#284](https://github.com/ArdurAI/sith/issues/284)) + +## Context + +ADR 0011 and issue 282 define an exact-decimal USD projector for one already-authorized OpenCost +namespace-allocation response. A successful response may contain an empty allocation map and +correctly emit zero facts. A later fleet rollup cannot infer from zero facts whether the cluster +reported an empty result or never reported at all. Treating both cases as zero cost would violate +E13's central coverage guardrail. + +The live access path is also unresolved. OpenCost documents its API on port 9003 through an +operator-run Kubernetes port-forward and notes that deployments may expose a Service or Ingress. +Sith has local, Hub, and security-held OCM environments, but no accepted contract assigns OpenCost +endpoint discovery, authentication, TLS, or credential forwarding to one of them. Those choices +must not leak into the normalization or aggregation core. + +Per-team grouping is similarly premature: the normalized cost fact deliberately discards labels, +and Sith does not yet have a canonical team-attribution identity for a namespace. Guessing from a +workload or arbitrary label would create unstable cross-tenant accounting semantics. + +## Decision + +1. `ProjectNamespaceCostSnapshot` wraps a successful F13.1a projection in a value-only envelope + containing its exact workspace, cluster scope, UTC window, trusted USD unit, and facts. Presence + of the snapshot is the reporting signal; an empty fact slice is a successful empty report. +2. `RollupWorkspaceCosts` accepts one explicit expected-scope set plus at most one successful + snapshot per reporting scope. Expected scopes are bounded, unique, and caller-authoritative. +3. Every snapshot must match the requested workspace, exact UTC window, and USD unit. Every fact is + revalidated against the closed cost taxonomy, TELEMETRY lens, cluster/namespace entity, + OpenCost provenance and protocol, canonical payload bytes, native SHA-256 identity, and source + observation time. +4. Invalid, duplicate, foreign, stale-marked, oversized, or ambiguous input aborts the entire + operation and returns no partial rollup. Duplicate namespaces within one cluster are rejected. +5. All fifteen component, adjustment, and total values are parsed as exact rational decimals and + summed independently. Output uses canonical five-decimal strings; binary floating point is not + used. +6. Coverage separately names expected, reported, successful-empty, and missing scopes. A missing + scope contributes no fact and no synthetic zero. `complete` is true only when every expected + scope has a successful snapshot. +7. The rollup preserves the allocation-window end as `observed_at` when at least one scope + reported. With no report, `observed_at` is absent. No collection time or stale objective is + invented. +8. The computation is bounded to 256 scopes, 1,024 facts per scope, 4,096 facts total, 8 MiB of + normalized payload, a 256 KiB result, and an aggregate magnitude derived from the per-fact cost + bound. + +## Consequences + +- A workspace total can never silently present partial OpenCost coverage as complete. +- Successful empty reports remain distinguishable from unavailable OpenCost without retaining raw + responses or adding a sentinel fact. +- The result retains aggregate amounts, coverage metadata (expected, reported, successful-empty, + and missing categories plus `complete`), and optional `observed_at` only. Namespace names, + provider IDs, labels, annotations, workload identity, endpoints, credentials, and unknown source + fields do not survive. +- Historical evidence remains tied to its source window, allowing F13.4 to select a freshness + objective later without retroactively changing fact semantics. +- This is an offline workspace computation core. It does not provide the live F13.1 adapter, + persistence, Hub/runtime composition, an API or UI, team rollups, or F13.2 completion. +- Runtime expense is bounded local CPU and memory. The slice creates no cloud resource, network + call, storage, telemetry-volume, egress, or recurring-service cost. + +## Alternatives considered + +- **Treat zero facts as zero cost:** rejected because it conflates successful empty coverage with a + missing cluster. +- **Emit a synthetic zero-cost fact:** rejected because a sentinel would look like observed + namespace cost and contaminate the fact model. +- **Accept arbitrary OpenCost URLs and credentials in the core:** held because this requires an + explicit SSRF, redirect, TLS, endpoint-provenance, and credential-forwarding decision. +- **Use the Kubernetes Service proxy:** held because it adds `services/proxy` RBAC and does not + solve the security-held OCM transport. +- **Group by an arbitrary team label now:** rejected because no canonical, tenant-scoped team + identity exists and F13.1a intentionally discards labels. +- **Use `float64`:** rejected because deterministic fleet totals require exact decimal behavior. +- **Stamp collection time or choose a stale threshold:** rejected because rereading historical + evidence must not make it fresh, and the objective belongs to F13.4. + +## Primary references + +- [OpenCost allocation API](https://opencost.io/docs/integrations/api/) +- [OpenCost installation and access](https://opencost.io/docs/installation/install/) +- [OpenCost v1.120.2](https://github.com/opencost/opencost/releases/tag/v1.120.2) +- [ADR 0011](0011-opencost-namespace-cost-facts.md) +- [F13.1a issue 282](https://github.com/ArdurAI/sith/issues/282) +- [E13 transport escalation](https://github.com/ArdurAI/sith/issues/31#issuecomment-5013914477) diff --git a/docs/adr/README.md b/docs/adr/README.md index 6c1eb3a..c68ae36 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -21,6 +21,7 @@ decision rests on an external fact, that fact is web-verified and cited (see als | [0009](0009-release-supply-chain.md) | Reproducible and identity-bound release supply chain | Accepted | | [0010](0010-native-local-desktop-shell.md) | Native local desktop shell | Accepted | | [0011](0011-opencost-namespace-cost-facts.md) | Exact-decimal USD boundary for OpenCost namespace cost facts | Accepted | +| [0012](0012-opencost-coverage-aware-workspace-rollup.md) | Coverage-aware workspace rollup for OpenCost cost facts | Accepted | Planning ADRs remain **Proposed** until their implementation lane accepts or rejects them. Implementation-specific ADRs may be **Accepted** when the corresponding shipped slice provides diff --git a/internal/connector/opencost/boundary_test.go b/internal/connector/opencost/boundary_test.go index 23a46c1..5c2522a 100644 --- a/internal/connector/opencost/boundary_test.go +++ b/internal/connector/opencost/boundary_test.go @@ -36,37 +36,56 @@ var allowedProductionImports = map[string]bool{ var allowedProductionFiles = map[string]string{ "project.go": "bc5a1b2919a50cd0213deb2881c25a3fd43beed790c9d3cbfeb9e147cb72dc96", + "rollup.go": "1dccb66e42dde4be827d2b2ef3a8cfea624dbce374934fd9e8340849cc3ba093", } var allowedProductionDeclarations = map[string]bool{ "func:ProjectNamespaceCosts": true, + "func:ProjectNamespaceCostSnapshot": true, + "func:RollupWorkspaceCosts": true, "func:buildFact": true, "func:consumeUniqueJSON": true, "func:decodeOptionalField": true, "func:matchingDelimiter": true, + "func:namespaceCostNativeID": true, + "func:newCostAccumulator": true, "func:objectFields": true, + "func:observationCostValue": true, + "func:parseCanonicalCost": true, "func:parseCanonicalTime": true, "func:parseCostAmount": true, "func:rejectCaseAliases": true, "func:rejectDuplicateJSON": true, + "func:sortedScopeKeys": true, "func:validCostLiteral": true, "func:validateAllocation": true, "func:validateAllocationWindow": true, "func:validateCanonicalTime": true, + "func:validateNamespaceCostFact": true, + "func:validateNamespaceCostSnapshot": true, "func:validateProjection": true, "func:validateResponse": true, "func:validateText": true, + "func:validateWorkspaceRollupRequest": true, "method:allocationProperties.UnmarshalJSON": true, "method:allocationRecord.UnmarshalJSON": true, "method:allocationResponse.UnmarshalJSON": true, "method:allocationWindow.UnmarshalJSON": true, + "method:costAccumulator.add": true, + "method:costAccumulator.amounts": true, "type:AllocationQuery": true, + "type:CostAmounts": true, + "type:NamespaceCostSnapshot": true, "type:Projection": true, + "type:WorkspaceCostCoverage": true, + "type:WorkspaceCostRollup": true, + "type:WorkspaceRollupRequest": true, "type:allocationProperties": true, "type:allocationRecord": true, "type:allocationResponse": true, "type:allocationWindow": true, "type:costField": true, + "type:costAccumulator": true, "type:namespaceCostObservation": true, "value:Kind": true, "value:ProtocolVersion": true, @@ -83,6 +102,11 @@ var allowedProductionDeclarations = map[string]bool{ "value:maxJSONDepth": true, "value:maxQueryWindow": true, "value:maxResponseBytes": true, + "value:maxRollupCostUnits": true, + "value:maxRollupFacts": true, + "value:maxRollupInputBytes": true, + "value:maxRollupPayloadBytes": true, + "value:maxRollupScopes": true, } func TestProjectorHasNoIOCredentialPersistenceOrMutationSeam(t *testing.T) { diff --git a/internal/connector/opencost/rollup.go b/internal/connector/opencost/rollup.go new file mode 100644 index 0000000..04eaa9c --- /dev/null +++ b/internal/connector/opencost/rollup.go @@ -0,0 +1,461 @@ +// SPDX-License-Identifier: Apache-2.0 + +package opencost + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "math/big" + "sort" + "time" + + "k8s.io/apimachinery/pkg/util/validation" + + "github.com/ArdurAI/sith/internal/fleet" +) + +const ( + maxRollupScopes = 256 + maxRollupFacts = 4_096 + maxRollupInputBytes = 8 << 20 + maxRollupPayloadBytes = 256 << 10 +) + +const maxRollupCostUnits = maxCostUnits * maxRollupFacts + +// NamespaceCostSnapshot records one successful, already-authorized per-scope projection. +// Presence of a snapshot is the reporting signal, so an empty Facts slice is distinct from a +// scope that has no snapshot. +type NamespaceCostSnapshot struct { + Workspace string `json:"workspace"` + Scope string `json:"scope"` + WindowStart time.Time `json:"window_start"` + WindowEnd time.Time `json:"window_end"` + CurrencyCode string `json:"currency"` + Facts []fleet.GraphFact `json:"facts"` +} + +// WorkspaceRollupRequest supplies the complete expected scope set and each successful scope +// snapshot for one exact allocation window. +type WorkspaceRollupRequest struct { + Workspace string `json:"workspace"` + WindowStart time.Time `json:"window_start"` + WindowEnd time.Time `json:"window_end"` + CurrencyCode string `json:"currency"` + ExpectedScopes []string `json:"expected_scopes"` + Snapshots []NamespaceCostSnapshot `json:"snapshots"` +} + +// WorkspaceCostCoverage makes partial cost reporting impossible to confuse with complete +// coverage. ReportedScopes includes both populated and successful-empty snapshots. +type WorkspaceCostCoverage struct { + ExpectedScopes []string `json:"expected_scopes"` + ReportedScopes []string `json:"reported_scopes"` + EmptyScopes []string `json:"empty_scopes"` + MissingScopes []string `json:"missing_scopes"` + Complete bool `json:"complete"` +} + +// CostAmounts is the exact, canonical component breakdown retained by the workspace rollup. +type CostAmounts struct { + CPUCost string `json:"cpu_cost"` + CPUCostAdjustment string `json:"cpu_cost_adjustment"` + GPUCost string `json:"gpu_cost"` + GPUCostAdjustment string `json:"gpu_cost_adjustment"` + RAMCost string `json:"ram_cost"` + RAMCostAdjustment string `json:"ram_cost_adjustment"` + PVCost string `json:"pv_cost"` + PVCostAdjustment string `json:"pv_cost_adjustment"` + NetworkCost string `json:"network_cost"` + NetworkCostAdjustment string `json:"network_cost_adjustment"` + LoadBalancerCost string `json:"load_balancer_cost"` + LoadBalancerCostAdjustment string `json:"load_balancer_cost_adjustment"` + SharedCost string `json:"shared_cost"` + ExternalCost string `json:"external_cost"` + TotalCost string `json:"total_cost"` +} + +// WorkspaceCostRollup is one deterministic, read-only workspace total. ObservedAt is nil only +// when no expected scope reported a successful snapshot. +type WorkspaceCostRollup struct { + Workspace string `json:"workspace"` + WindowStart time.Time `json:"window_start"` + WindowEnd time.Time `json:"window_end"` + CurrencyCode string `json:"currency"` + ObservedAt *time.Time `json:"observed_at,omitempty"` + NamespaceFacts int `json:"namespace_facts"` + Coverage WorkspaceCostCoverage `json:"coverage"` + Amounts CostAmounts `json:"amounts"` +} + +// ProjectNamespaceCostSnapshot preserves the successful per-scope reporting envelope around the +// existing fact projector. It performs no additional I/O and returns a zero snapshot on error. +func ProjectNamespaceCostSnapshot(input Projection) (NamespaceCostSnapshot, error) { + facts, err := ProjectNamespaceCosts(input) + if err != nil { + return NamespaceCostSnapshot{}, err + } + return NamespaceCostSnapshot{ + Workspace: input.Workspace, Scope: input.Scope, + WindowStart: input.Query.WindowStart, WindowEnd: input.Query.WindowEnd, + CurrencyCode: input.CurrencyCode, Facts: facts, + }, nil +} + +// RollupWorkspaceCosts validates and aggregates successful per-scope snapshots. Missing expected +// scopes remain explicit coverage gaps; they never contribute a synthetic zero-cost fact. +func RollupWorkspaceCosts(input WorkspaceRollupRequest) (WorkspaceCostRollup, error) { + expected, err := validateWorkspaceRollupRequest(input) + if err != nil { + return WorkspaceCostRollup{}, err + } + + byScope := make(map[string]NamespaceCostSnapshot, len(input.Snapshots)) + reported := make([]string, 0, len(input.Snapshots)) + empty := make([]string, 0, len(input.Snapshots)) + accumulator := newCostAccumulator() + factCount := 0 + inputBytes := 0 + for index, snapshot := range input.Snapshots { + if _, exists := byScope[snapshot.Scope]; exists { + return WorkspaceCostRollup{}, fmt.Errorf("OpenCost workspace rollup snapshot %d duplicates scope", index) + } + if _, exists := expected[snapshot.Scope]; !exists { + return WorkspaceCostRollup{}, fmt.Errorf("OpenCost workspace rollup snapshot %d has unexpected scope", index) + } + if err := validateNamespaceCostSnapshot(input, snapshot, accumulator, &factCount, &inputBytes); err != nil { + return WorkspaceCostRollup{}, fmt.Errorf("OpenCost workspace rollup snapshot %d: %w", index, err) + } + byScope[snapshot.Scope] = snapshot + reported = append(reported, snapshot.Scope) + if len(snapshot.Facts) == 0 { + empty = append(empty, snapshot.Scope) + } + } + + sort.Strings(reported) + sort.Strings(empty) + missing := make([]string, 0, len(expected)-len(reported)) + for scope := range expected { + if _, exists := byScope[scope]; !exists { + missing = append(missing, scope) + } + } + sort.Strings(missing) + + amounts, err := accumulator.amounts() + if err != nil { + return WorkspaceCostRollup{}, fmt.Errorf("OpenCost workspace rollup amounts: %w", err) + } + rollup := WorkspaceCostRollup{ + Workspace: input.Workspace, WindowStart: input.WindowStart, WindowEnd: input.WindowEnd, + CurrencyCode: input.CurrencyCode, NamespaceFacts: factCount, + Coverage: WorkspaceCostCoverage{ + ExpectedScopes: sortedScopeKeys(expected), ReportedScopes: reported, + EmptyScopes: empty, MissingScopes: missing, Complete: len(missing) == 0, + }, + Amounts: amounts, + } + if len(reported) != 0 { + observedAt := input.WindowEnd + rollup.ObservedAt = &observedAt + } + encoded, err := json.Marshal(rollup) + if err != nil { + return WorkspaceCostRollup{}, fmt.Errorf("encode OpenCost workspace rollup: %w", err) + } + if len(encoded) > maxRollupPayloadBytes { + return WorkspaceCostRollup{}, fmt.Errorf("OpenCost workspace rollup exceeds %d bytes", maxRollupPayloadBytes) + } + return rollup, nil +} + +func validateWorkspaceRollupRequest(input WorkspaceRollupRequest) (map[string]struct{}, error) { + if err := validateText("workspace", input.Workspace, maxIdentityText); err != nil { + return nil, err + } + if input.CurrencyCode != currencyUSD { + return nil, fmt.Errorf("OpenCost workspace rollup currency must be USD") + } + if err := validateCanonicalTime("window start", input.WindowStart); err != nil { + return nil, err + } + if err := validateCanonicalTime("window end", input.WindowEnd); err != nil { + return nil, err + } + window := input.WindowEnd.Sub(input.WindowStart) + if window <= 0 || window > maxQueryWindow { + return nil, fmt.Errorf("OpenCost workspace rollup window is invalid") + } + if len(input.ExpectedScopes) > maxRollupScopes { + return nil, fmt.Errorf("OpenCost workspace rollup expected scope count exceeds %d", maxRollupScopes) + } + if len(input.Snapshots) > len(input.ExpectedScopes) || len(input.Snapshots) > maxRollupScopes { + return nil, fmt.Errorf("OpenCost workspace rollup snapshot count is invalid") + } + expected := make(map[string]struct{}, len(input.ExpectedScopes)) + for index, scope := range input.ExpectedScopes { + if err := validateText("scope", scope, maxIdentityText); err != nil || bytes.ContainsRune([]byte(scope), '/') { + return nil, fmt.Errorf("OpenCost workspace rollup expected scope %d is invalid", index) + } + if _, exists := expected[scope]; exists { + return nil, fmt.Errorf("OpenCost workspace rollup expected scope %d is duplicated", index) + } + expected[scope] = struct{}{} + } + return expected, nil +} + +func validateNamespaceCostSnapshot( + request WorkspaceRollupRequest, + snapshot NamespaceCostSnapshot, + accumulator *costAccumulator, + factCount *int, + inputBytes *int, +) error { + if snapshot.Workspace != request.Workspace || snapshot.CurrencyCode != request.CurrencyCode || + !snapshot.WindowStart.Equal(request.WindowStart) || !snapshot.WindowEnd.Equal(request.WindowEnd) { + return fmt.Errorf("snapshot envelope does not match requested workspace, window, and currency") + } + if err := validateCanonicalTime("snapshot window start", snapshot.WindowStart); err != nil { + return err + } + if err := validateCanonicalTime("snapshot window end", snapshot.WindowEnd); err != nil { + return err + } + if err := validateText("scope", snapshot.Scope, maxIdentityText); err != nil || + bytes.ContainsRune([]byte(snapshot.Scope), '/') { + return fmt.Errorf("snapshot scope is invalid") + } + if len(snapshot.Facts) > maxAllocations { + return fmt.Errorf("snapshot fact count exceeds %d", maxAllocations) + } + if *factCount+len(snapshot.Facts) > maxRollupFacts { + return fmt.Errorf("workspace fact count exceeds %d", maxRollupFacts) + } + + namespaces := make(map[string]struct{}, len(snapshot.Facts)) + for index, fact := range snapshot.Facts { + observation, observedBytes, err := validateNamespaceCostFact(request, snapshot.Scope, fact) + if err != nil { + return fmt.Errorf("fact %d: %w", index, err) + } + if _, exists := namespaces[observation.Namespace]; exists { + return fmt.Errorf("fact %d duplicates namespace", index) + } + namespaces[observation.Namespace] = struct{}{} + *inputBytes += observedBytes + if *inputBytes > maxRollupInputBytes { + return fmt.Errorf("workspace fact payload exceeds %d bytes", maxRollupInputBytes) + } + if err := accumulator.add(observation); err != nil { + return fmt.Errorf("fact %d amounts: %w", index, err) + } + } + *factCount += len(snapshot.Facts) + return nil +} + +func validateNamespaceCostFact( + request WorkspaceRollupRequest, + scope string, + fact fleet.GraphFact, +) (namespaceCostObservation, int, error) { + if err := fact.Validate(request.Workspace); err != nil { + return namespaceCostObservation{}, 0, fmt.Errorf("graph fact is invalid") + } + if fact.Fact.Kind != fleet.FactCost || fact.Lens != fleet.LensTelemetry || + fact.Fact.Workspace != request.Workspace || fact.Fact.Source != scope || + !fact.Fact.ObservedAt.Equal(request.WindowEnd) || fact.Fact.Stale || fact.Fact.StaleFor != "" { + return namespaceCostObservation{}, 0, fmt.Errorf("fact envelope is invalid") + } + if fact.Fact.Ref.SourceKind != Kind || fact.Fact.Ref.Scope != scope || + fact.Fact.Ref.Kind != "NamespaceCost" || fact.Fact.Ref.Name != "allocation" || + fact.Fact.Ref.Namespace == "" || fact.Fact.Ref.Attributes != nil || fact.Fact.Display != nil { + return namespaceCostObservation{}, 0, fmt.Errorf("fact resource identity or display metadata is invalid") + } + if fact.Fact.Provenance.Adapter != Kind || fact.Fact.Provenance.ProtocolV != ProtocolVersion || + fact.Fact.Provenance.DeepLink != "" || fact.Fact.Provenance.Collector != "" { + return namespaceCostObservation{}, 0, fmt.Errorf("fact provenance is invalid") + } + if fact.Entity == nil || *fact.Entity != (fleet.EntityRef{ + Cluster: scope, Namespace: fact.Fact.Ref.Namespace, + }) { + return namespaceCostObservation{}, 0, fmt.Errorf("fact entity is invalid") + } + if len(fact.Fact.Observed) == 0 || len(fact.Fact.Observed) > maxFactPayloadBytes || + !json.Valid(fact.Fact.Observed) { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload is invalid") + } + if err := rejectDuplicateJSON(fact.Fact.Observed); err != nil { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload is not canonical") + } + var observation namespaceCostObservation + if err := json.Unmarshal(fact.Fact.Observed, &observation); err != nil { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload is invalid") + } + canonical, err := json.Marshal(observation) + if err != nil || !bytes.Equal(canonical, fact.Fact.Observed) { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload is not canonical") + } + if observation.Namespace != fact.Fact.Ref.Namespace || observation.Currency != request.CurrencyCode || + !observation.WindowStart.Equal(request.WindowStart) || !observation.WindowEnd.Equal(request.WindowEnd) { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload identity, window, or currency is invalid") + } + if len(validation.IsDNS1123Label(observation.Namespace)) != 0 { + return namespaceCostObservation{}, 0, fmt.Errorf("fact payload namespace is invalid") + } + if err := validateCanonicalTime("fact window start", observation.WindowStart); err != nil { + return namespaceCostObservation{}, 0, err + } + if err := validateCanonicalTime("fact window end", observation.WindowEnd); err != nil { + return namespaceCostObservation{}, 0, err + } + nativeID, err := namespaceCostNativeID(request.Workspace, scope, fact.Fact.Observed) + if err != nil || fact.Fact.Provenance.NativeID != nativeID { + return namespaceCostObservation{}, 0, fmt.Errorf("fact native identity is invalid") + } + return observation, len(fact.Fact.Observed), nil +} + +type costAccumulator struct { + values map[string]*big.Rat +} + +func newCostAccumulator() *costAccumulator { + values := make(map[string]*big.Rat, len(costFields)) + for _, field := range costFields { + values[field.JSONName] = new(big.Rat) + } + return &costAccumulator{values: values} +} + +func (accumulator *costAccumulator) add(observation namespaceCostObservation) error { + if accumulator == nil || len(accumulator.values) != len(costFields) { + return fmt.Errorf("cost accumulator is invalid") + } + total := new(big.Rat) + for _, field := range costFields { + value := observationCostValue(observation, field.JSONName) + amount, err := parseCanonicalCost(value, field.AllowNegative) + if err != nil { + return fmt.Errorf("%s is invalid", field.JSONName) + } + if field.PartOfTotal { + total.Add(total, amount) + } + accumulator.values[field.JSONName].Add(accumulator.values[field.JSONName], amount) + if new(big.Rat).Abs(new(big.Rat).Set(accumulator.values[field.JSONName])).Cmp( + new(big.Rat).SetInt64(maxRollupCostUnits), + ) > 0 { + return fmt.Errorf("%s aggregate exceeds limit", field.JSONName) + } + } + reportedTotal, err := parseCanonicalCost(observation.TotalCost, false) + if err != nil { + return fmt.Errorf("totalCost is invalid") + } + difference := new(big.Rat).Sub(reportedTotal, total) + difference.Abs(difference) + if difference.Cmp(new(big.Rat).SetFrac64(1, 10_000)) > 0 { + return fmt.Errorf("totalCost does not match components") + } + return nil +} + +func (accumulator *costAccumulator) amounts() (CostAmounts, error) { + if accumulator == nil || len(accumulator.values) != len(costFields) { + return CostAmounts{}, fmt.Errorf("cost accumulator is invalid") + } + value := func(name string) string { + return accumulator.values[name].FloatString(costScale) + } + return CostAmounts{ + CPUCost: value("cpuCost"), CPUCostAdjustment: value("cpuCostAdjustment"), + GPUCost: value("gpuCost"), GPUCostAdjustment: value("gpuCostAdjustment"), + RAMCost: value("ramCost"), RAMCostAdjustment: value("ramCostAdjustment"), + PVCost: value("pvCost"), PVCostAdjustment: value("pvCostAdjustment"), + NetworkCost: value("networkCost"), NetworkCostAdjustment: value("networkCostAdjustment"), + LoadBalancerCost: value("loadBalancerCost"), + LoadBalancerCostAdjustment: value("loadBalancerCostAdjustment"), + SharedCost: value("sharedCost"), ExternalCost: value("externalCost"), + TotalCost: value("totalCost"), + }, nil +} + +func parseCanonicalCost(value string, allowNegative bool) (*big.Rat, error) { + if !validCostLiteral(value) || len(value) < costScale+2 || + value[len(value)-costScale-1] != '.' { + return nil, fmt.Errorf("cost value is not canonical") + } + amount, ok := new(big.Rat).SetString(value) + if !ok || (!allowNegative && amount.Sign() < 0) || amount.FloatString(costScale) != value { + return nil, fmt.Errorf("cost value is not canonical") + } + if new(big.Rat).Abs(new(big.Rat).Set(amount)).Cmp(new(big.Rat).SetInt64(maxCostUnits)) > 0 { + return nil, fmt.Errorf("cost value exceeds limit") + } + return amount, nil +} + +func observationCostValue(observation namespaceCostObservation, name string) string { + switch name { + case "cpuCost": + return observation.CPUCost + case "cpuCostAdjustment": + return observation.CPUCostAdjustment + case "gpuCost": + return observation.GPUCost + case "gpuCostAdjustment": + return observation.GPUCostAdjustment + case "ramCost": + return observation.RAMCost + case "ramCostAdjustment": + return observation.RAMCostAdjustment + case "pvCost": + return observation.PVCost + case "pvCostAdjustment": + return observation.PVCostAdjustment + case "networkCost": + return observation.NetworkCost + case "networkCostAdjustment": + return observation.NetworkCostAdjustment + case "loadBalancerCost": + return observation.LoadBalancerCost + case "loadBalancerCostAdjustment": + return observation.LoadBalancerCostAdjustment + case "sharedCost": + return observation.SharedCost + case "externalCost": + return observation.ExternalCost + case "totalCost": + return observation.TotalCost + default: + return "" + } +} + +func namespaceCostNativeID(workspace, scope string, observed json.RawMessage) (string, error) { + identity, err := json.Marshal(struct { + Workspace string `json:"workspace"` + Scope string `json:"scope"` + Observed json.RawMessage `json:"observed"` + }{Workspace: workspace, Scope: scope, Observed: observed}) + if err != nil { + return "", err + } + digest := sha256.Sum256(identity) + return "sha256:" + hex.EncodeToString(digest[:]), nil +} + +func sortedScopeKeys(scopes map[string]struct{}) []string { + result := make([]string, 0, len(scopes)) + for scope := range scopes { + result = append(result, scope) + } + sort.Strings(result) + return result +} diff --git a/internal/connector/opencost/rollup_test.go b/internal/connector/opencost/rollup_test.go new file mode 100644 index 0000000..454e7dd --- /dev/null +++ b/internal/connector/opencost/rollup_test.go @@ -0,0 +1,511 @@ +// SPDX-License-Identifier: Apache-2.0 + +package opencost + +import ( + "bytes" + "encoding/json" + "reflect" + "slices" + "strings" + "testing" + "time" + + "github.com/ArdurAI/sith/internal/fleet" +) + +func TestProjectNamespaceCostSnapshotPreservesSuccessfulEmptyCoverage(t *testing.T) { + t.Parallel() + input := testProjection(t) + + snapshot, err := ProjectNamespaceCostSnapshot(input) + if err != nil { + t.Fatalf("ProjectNamespaceCostSnapshot() error = %v", err) + } + if snapshot.Workspace != input.Workspace || snapshot.Scope != input.Scope || + snapshot.CurrencyCode != input.CurrencyCode || + !snapshot.WindowStart.Equal(input.Query.WindowStart) || + !snapshot.WindowEnd.Equal(input.Query.WindowEnd) || + snapshot.Facts == nil || len(snapshot.Facts) != 0 { + t.Fatalf("successful empty snapshot = %#v", snapshot) + } + + facts, err := ProjectNamespaceCosts(input) + if err != nil { + t.Fatalf("ProjectNamespaceCosts() error = %v", err) + } + if !reflect.DeepEqual(snapshot.Facts, facts) { + t.Fatalf("snapshot facts = %#v, compatibility facts = %#v", snapshot.Facts, facts) + } + + invalid := input + invalid.CurrencyCode = "EUR" + failed, err := ProjectNamespaceCostSnapshot(invalid) + if err == nil || !reflect.DeepEqual(failed, NamespaceCostSnapshot{}) { + t.Fatalf("invalid projection = %#v, %v", failed, err) + } +} + +func TestRollupWorkspaceCostsSurfacesPartialAndEmptyCoverage(t *testing.T) { + t.Parallel() + populated := testCostSnapshot(t, "cluster-a", "payments", "infra") + empty := testCostSnapshot(t, "cluster-b") + request := WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, + ExpectedScopes: []string{"cluster-c", "cluster-b", "cluster-a"}, + Snapshots: []NamespaceCostSnapshot{empty, populated}, + } + + rollup, err := RollupWorkspaceCosts(request) + if err != nil { + t.Fatalf("RollupWorkspaceCosts() error = %v", err) + } + if rollup.Workspace != request.Workspace || !rollup.WindowStart.Equal(testWindowStart) || + !rollup.WindowEnd.Equal(testWindowEnd) || rollup.CurrencyCode != currencyUSD || + rollup.ObservedAt == nil || !rollup.ObservedAt.Equal(testWindowEnd) || + rollup.NamespaceFacts != 2 { + t.Fatalf("rollup envelope = %#v", rollup) + } + wantCoverage := WorkspaceCostCoverage{ + ExpectedScopes: []string{"cluster-a", "cluster-b", "cluster-c"}, + ReportedScopes: []string{"cluster-a", "cluster-b"}, + EmptyScopes: []string{"cluster-b"}, + MissingScopes: []string{"cluster-c"}, + Complete: false, + } + if !reflect.DeepEqual(rollup.Coverage, wantCoverage) { + t.Fatalf("coverage = %#v, want %#v", rollup.Coverage, wantCoverage) + } + wantAmounts := CostAmounts{ + CPUCost: "2.50000", CPUCostAdjustment: "-0.10000", + GPUCost: "4.00000", GPUCostAdjustment: "0.00000", + RAMCost: "1.50000", RAMCostAdjustment: "0.02000", + PVCost: "1.00000", PVCostAdjustment: "0.00000", + NetworkCost: "0.50000", NetworkCostAdjustment: "-0.02000", + LoadBalancerCost: "0.20000", LoadBalancerCostAdjustment: "0.00000", + SharedCost: "0.40000", ExternalCost: "0.60000", TotalCost: "10.60000", + } + if rollup.Amounts != wantAmounts { + t.Fatalf("amounts = %#v, want %#v", rollup.Amounts, wantAmounts) + } + encoded, err := json.Marshal(rollup) + if err != nil { + t.Fatalf("marshal rollup: %v", err) + } + if len(encoded) > maxRollupPayloadBytes { + t.Fatalf("rollup bytes = %d", len(encoded)) + } + for _, forbidden := range []string{ + "payments", "infra", "do-not-retain-provider-id", "do-not-retain-label", + "do-not-retain-annotation", "do-not-retain-controller", "do-not-retain-endpoint", + } { + if bytes.Contains(encoded, []byte(forbidden)) { + t.Fatalf("rollup retained forbidden detail %q: %s", forbidden, encoded) + } + } +} + +func TestRollupWorkspaceCostsIsDeterministicAcrossSnapshotAndFactOrder(t *testing.T) { + t.Parallel() + alpha := testCostSnapshot(t, "cluster-a", "zeta", "alpha") + beta := testCostSnapshot(t, "cluster-b", "payments") + left := WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-b", "cluster-a"}, + Snapshots: []NamespaceCostSnapshot{alpha, beta}, + } + right := cloneRollupRequest(t, left) + slices.Reverse(right.ExpectedScopes) + slices.Reverse(right.Snapshots) + slices.Reverse(right.Snapshots[1].Facts) + + leftRollup, err := RollupWorkspaceCosts(left) + if err != nil { + t.Fatalf("left rollup: %v", err) + } + rightRollup, err := RollupWorkspaceCosts(right) + if err != nil { + t.Fatalf("right rollup: %v", err) + } + if !reflect.DeepEqual(leftRollup, rightRollup) { + t.Fatalf("input order changed rollup:\nleft = %#v\nright = %#v", leftRollup, rightRollup) + } + if !leftRollup.Coverage.Complete || len(leftRollup.Coverage.MissingScopes) != 0 || + leftRollup.NamespaceFacts != 3 || leftRollup.Amounts.TotalCost != "15.90000" { + t.Fatalf("complete rollup = %#v", leftRollup) + } +} + +func TestRollupWorkspaceCostsDistinguishesNoReportsFromSuccessfulEmpty(t *testing.T) { + t.Parallel() + missing, err := RollupWorkspaceCosts(WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-b", "cluster-a"}, + }) + if err != nil { + t.Fatalf("missing rollup: %v", err) + } + if missing.ObservedAt != nil || missing.NamespaceFacts != 0 || missing.Coverage.Complete || + !reflect.DeepEqual(missing.Coverage.MissingScopes, []string{"cluster-a", "cluster-b"}) || + missing.Amounts != zeroCostAmounts() { + t.Fatalf("all-missing rollup = %#v", missing) + } + + empty, err := RollupWorkspaceCosts(WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-a"}, + Snapshots: []NamespaceCostSnapshot{testCostSnapshot(t, "cluster-a")}, + }) + if err != nil { + t.Fatalf("empty rollup: %v", err) + } + if empty.ObservedAt == nil || !empty.ObservedAt.Equal(testWindowEnd) || + !empty.Coverage.Complete || + !reflect.DeepEqual(empty.Coverage.ReportedScopes, []string{"cluster-a"}) || + !reflect.DeepEqual(empty.Coverage.EmptyScopes, []string{"cluster-a"}) || + len(empty.Coverage.MissingScopes) != 0 || empty.Amounts != zeroCostAmounts() { + t.Fatalf("successful-empty rollup = %#v", empty) + } + + noScopes, err := RollupWorkspaceCosts(WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, + }) + if err != nil { + t.Fatalf("zero-scope rollup: %v", err) + } + if noScopes.ObservedAt != nil || !noScopes.Coverage.Complete || + noScopes.Coverage.ExpectedScopes == nil || noScopes.Coverage.ReportedScopes == nil || + noScopes.Coverage.EmptyScopes == nil || noScopes.Coverage.MissingScopes == nil { + t.Fatalf("zero-scope rollup = %#v", noScopes) + } +} + +func TestRollupWorkspaceCostsRejectsInvalidRequestsAtomically(t *testing.T) { + t.Parallel() + base := WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-a"}, + Snapshots: []NamespaceCostSnapshot{testCostSnapshot(t, "cluster-a", "payments")}, + } + fixedOffset := time.FixedZone("offset", -5*60*60) + tests := map[string]func(*WorkspaceRollupRequest){ + "workspace": func(value *WorkspaceRollupRequest) { value.Workspace = " workspace-a" }, + "currency": func(value *WorkspaceRollupRequest) { value.CurrencyCode = "EUR" }, + "window start": func(value *WorkspaceRollupRequest) { value.WindowStart = time.Time{} }, + "window end": func(value *WorkspaceRollupRequest) { value.WindowEnd = value.WindowStart }, + "window duration": func(value *WorkspaceRollupRequest) { + value.WindowEnd = value.WindowStart.Add(maxQueryWindow + time.Second) + }, + "expected duplicate": func(value *WorkspaceRollupRequest) { + value.ExpectedScopes = append(value.ExpectedScopes, value.ExpectedScopes[0]) + }, + "expected invalid": func(value *WorkspaceRollupRequest) { value.ExpectedScopes[0] = "cluster/a" }, + "expected excessive": func(value *WorkspaceRollupRequest) { + value.ExpectedScopes = make([]string, maxRollupScopes+1) + for index := range value.ExpectedScopes { + value.ExpectedScopes[index] = "cluster-" + strings.Repeat("a", index/26) + string(rune('a'+index%26)) + } + value.Snapshots = nil + }, + "snapshot excessive": func(value *WorkspaceRollupRequest) { + value.Snapshots = append(value.Snapshots, cloneCostSnapshot(t, value.Snapshots[0])) + }, + "snapshot unexpected": func(value *WorkspaceRollupRequest) { value.Snapshots[0].Scope = "cluster-b" }, + "snapshot workspace": func(value *WorkspaceRollupRequest) { value.Snapshots[0].Workspace = "workspace-b" }, + "snapshot currency": func(value *WorkspaceRollupRequest) { value.Snapshots[0].CurrencyCode = "EUR" }, + "snapshot window": func(value *WorkspaceRollupRequest) { value.Snapshots[0].WindowEnd = value.WindowEnd.Add(time.Second) }, + "snapshot non-UTC": func(value *WorkspaceRollupRequest) { + value.Snapshots[0].WindowStart = value.WindowStart.In(fixedOffset) + }, + "snapshot fact count": func(value *WorkspaceRollupRequest) { + value.Snapshots[0].Facts = make([]fleet.GraphFact, maxAllocations+1) + }, + } + for name, mutate := range tests { + name, mutate := name, mutate + t.Run(name, func(t *testing.T) { + t.Parallel() + candidate := cloneRollupRequest(t, base) + mutate(&candidate) + assertRollupErrorIsAtomic(t, candidate) + }) + } +} + +func TestRollupWorkspaceCostsRevalidatesFactsAtomically(t *testing.T) { + t.Parallel() + base := WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-a"}, + Snapshots: []NamespaceCostSnapshot{testCostSnapshot(t, "cluster-a", "payments")}, + } + tests := map[string]func(*fleet.GraphFact){ + "workspace": func(fact *fleet.GraphFact) { fact.Fact.Workspace = "workspace-b" }, + "kind": func(fact *fleet.GraphFact) { fact.Fact.Kind = fleet.FactAlert }, + "lens": func(fact *fleet.GraphFact) { fact.Lens = fleet.LensLive }, + "source": func(fact *fleet.GraphFact) { fact.Fact.Source = "cluster-b" }, + "observed at": func(fact *fleet.GraphFact) { + fact.Fact.ObservedAt = fact.Fact.ObservedAt.Add(time.Second) + }, + "stale": func(fact *fleet.GraphFact) { fact.Fact.Stale = true }, + "stale detail": func(fact *fleet.GraphFact) { fact.Fact.StaleFor = "1m" }, + "source kind": func(fact *fleet.GraphFact) { fact.Fact.Ref.SourceKind = "other" }, + "scope": func(fact *fleet.GraphFact) { fact.Fact.Ref.Scope = "cluster-b" }, + "resource kind": func(fact *fleet.GraphFact) { + fact.Fact.Ref.Kind = "Namespace" + }, + "resource name": func(fact *fleet.GraphFact) { fact.Fact.Ref.Name = "other" }, + "attributes": func(fact *fleet.GraphFact) { fact.Fact.Ref.Attributes = map[string]string{} }, + "display": func(fact *fleet.GraphFact) { fact.Fact.Display = []fleet.DisplayField{} }, + "adapter": func(fact *fleet.GraphFact) { fact.Fact.Provenance.Adapter = "other" }, + "protocol": func(fact *fleet.GraphFact) { fact.Fact.Provenance.ProtocolV = "other" }, + "native identity": func(fact *fleet.GraphFact) { + fact.Fact.Provenance.NativeID = "sha256:" + strings.Repeat("0", 64) + }, + "deep link": func(fact *fleet.GraphFact) { fact.Fact.Provenance.DeepLink = "https://secret.invalid" }, + "collector": func(fact *fleet.GraphFact) { fact.Fact.Provenance.Collector = "secret" }, + "entity nil": func(fact *fleet.GraphFact) { fact.Entity = nil }, + "entity mismatch": func(fact *fleet.GraphFact) { + fact.Entity.Namespace = "other" + }, + "payload whitespace": func(fact *fleet.GraphFact) { + fact.Fact.Observed = append([]byte(" "), fact.Fact.Observed...) + rebindFactNativeID(t, fact) + }, + "payload unknown": func(fact *fleet.GraphFact) { + var document map[string]any + mustUnmarshal(t, fact.Fact.Observed, &document) + document["provider_id"] = "secret" + fact.Fact.Observed = mustMarshal(t, document) + rebindFactNativeID(t, fact) + }, + "payload duplicate": func(fact *fleet.GraphFact) { + fact.Fact.Observed = bytes.Replace( + fact.Fact.Observed, + []byte(`{"namespace":"payments",`), + []byte(`{"namespace":"payments","namespace":"payments",`), + 1, + ) + rebindFactNativeID(t, fact) + }, + "payload oversized": func(fact *fleet.GraphFact) { + fact.Fact.Observed = append(fact.Fact.Observed, bytes.Repeat([]byte(" "), maxFactPayloadBytes)...) + rebindFactNativeID(t, fact) + }, + "payload namespace": func(fact *fleet.GraphFact) { + mutateFactObservation(t, fact, func(value *namespaceCostObservation) { value.Namespace = "Invalid" }) + fact.Fact.Ref.Namespace = "Invalid" + fact.Entity.Namespace = "Invalid" + rebindFactNativeID(t, fact) + }, + "payload window": func(fact *fleet.GraphFact) { + mutateFactObservation(t, fact, func(value *namespaceCostObservation) { + value.WindowEnd = value.WindowEnd.Add(time.Second) + }) + rebindFactNativeID(t, fact) + }, + "payload currency": func(fact *fleet.GraphFact) { + mutateFactObservation(t, fact, func(value *namespaceCostObservation) { value.Currency = "EUR" }) + rebindFactNativeID(t, fact) + }, + "payload noncanonical amount": func(fact *fleet.GraphFact) { + mutateFactObservation(t, fact, func(value *namespaceCostObservation) { value.CPUCost = "1.25" }) + rebindFactNativeID(t, fact) + }, + "payload component total": func(fact *fleet.GraphFact) { + mutateFactObservation(t, fact, func(value *namespaceCostObservation) { value.TotalCost = "5.40000" }) + rebindFactNativeID(t, fact) + }, + } + for name, mutate := range tests { + name, mutate := name, mutate + t.Run(name, func(t *testing.T) { + t.Parallel() + candidate := cloneRollupRequest(t, base) + mutate(&candidate.Snapshots[0].Facts[0]) + assertRollupErrorIsAtomic(t, candidate) + }) + } + + duplicate := cloneRollupRequest(t, base) + duplicate.Snapshots[0].Facts = append( + duplicate.Snapshots[0].Facts, + duplicate.Snapshots[0].Facts[0], + ) + assertRollupErrorIsAtomic(t, duplicate) +} + +func TestCanonicalRollupCostContract(t *testing.T) { + t.Parallel() + for value, valid := range map[string]bool{ + "0.00000": true, "1.25000": true, "-0.05000": true, + "0": false, "1.25": false, "01.00000": false, "-0.00000": false, + "1.250000": false, "1e2": false, " 1.00000": false, + } { + value, valid := value, valid + t.Run(value, func(t *testing.T) { + t.Parallel() + _, err := parseCanonicalCost(value, true) + if (err == nil) != valid { + t.Fatalf("parseCanonicalCost(%q) error = %v, valid = %t", value, err, valid) + } + }) + } +} + +func FuzzRollupWorkspaceCostsNeverPanicsOrReturnsPartial(f *testing.F) { + seed := testCostSnapshot(f, "cluster-a", "payments") + f.Add([]byte(seed.Facts[0].Fact.Observed), uint8(0), "cluster-a") + f.Add([]byte(`{"namespace":"payments"}`), uint8(1), "cluster-b") + f.Add([]byte(`{"namespace":"payments","namespace":"payments"}`), uint8(2), "cluster/a") + f.Fuzz(func(t *testing.T, payload []byte, mode uint8, scope string) { + request := WorkspaceRollupRequest{ + Workspace: "workspace-a", WindowStart: testWindowStart, WindowEnd: testWindowEnd, + CurrencyCode: currencyUSD, ExpectedScopes: []string{"cluster-a"}, + Snapshots: []NamespaceCostSnapshot{testCostSnapshot(t, "cluster-a", "payments")}, + } + switch mode % 4 { + case 0: + request.Snapshots[0].Facts[0].Fact.Observed = append([]byte(nil), payload...) + nativeID, err := namespaceCostNativeID( + request.Workspace, + request.Snapshots[0].Scope, + request.Snapshots[0].Facts[0].Fact.Observed, + ) + if err == nil { + request.Snapshots[0].Facts[0].Fact.Provenance.NativeID = nativeID + } + case 1: + request.ExpectedScopes = []string{scope} + request.Snapshots[0].Scope = scope + case 2: + request.Snapshots[0].Facts = append( + request.Snapshots[0].Facts, + request.Snapshots[0].Facts[0], + ) + case 3: + request.ExpectedScopes = append(request.ExpectedScopes, scope) + } + rollup, err := RollupWorkspaceCosts(request) + if err != nil { + if !reflect.DeepEqual(rollup, WorkspaceCostRollup{}) { + t.Fatalf("error returned partial rollup: %#v, %v", rollup, err) + } + return + } + if rollup.NamespaceFacts > maxRollupFacts || + rollup.Coverage.Complete != (len(rollup.Coverage.MissingScopes) == 0) || + !slices.IsSorted(rollup.Coverage.ExpectedScopes) || + !slices.IsSorted(rollup.Coverage.ReportedScopes) || + !slices.IsSorted(rollup.Coverage.EmptyScopes) || + !slices.IsSorted(rollup.Coverage.MissingScopes) { + t.Fatalf("invalid successful rollup: %#v", rollup) + } + for _, value := range costAmountValues(rollup.Amounts) { + if _, err := parseCanonicalCost(value, true); err != nil { + t.Fatalf("noncanonical successful amount %q: %v", value, err) + } + } + encoded, err := json.Marshal(rollup) + if err != nil || len(encoded) > maxRollupPayloadBytes { + t.Fatalf("successful rollup encoding = %d bytes, %v", len(encoded), err) + } + }) +} + +func testCostSnapshot(t testing.TB, scope string, namespaces ...string) NamespaceCostSnapshot { + t.Helper() + input := projectionFor(t, "workspace-a", scope, testWindowStart, testWindowEnd, namespaces...) + snapshot, err := ProjectNamespaceCostSnapshot(input) + if err != nil { + t.Fatalf("ProjectNamespaceCostSnapshot() error = %v", err) + } + return snapshot +} + +func cloneCostSnapshot(t testing.TB, value NamespaceCostSnapshot) NamespaceCostSnapshot { + t.Helper() + encoded := mustMarshal(t, value) + var cloned NamespaceCostSnapshot + mustUnmarshal(t, encoded, &cloned) + return cloned +} + +func cloneRollupRequest(t testing.TB, value WorkspaceRollupRequest) WorkspaceRollupRequest { + t.Helper() + encoded := mustMarshal(t, value) + var cloned WorkspaceRollupRequest + mustUnmarshal(t, encoded, &cloned) + return cloned +} + +func mutateFactObservation( + t testing.TB, + fact *fleet.GraphFact, + mutate func(*namespaceCostObservation), +) { + t.Helper() + var observation namespaceCostObservation + mustUnmarshal(t, fact.Fact.Observed, &observation) + mutate(&observation) + fact.Fact.Observed = mustMarshal(t, observation) +} + +func rebindFactNativeID(t testing.TB, fact *fleet.GraphFact) { + t.Helper() + nativeID, err := namespaceCostNativeID(fact.Fact.Workspace, fact.Fact.Ref.Scope, fact.Fact.Observed) + if err != nil { + t.Fatalf("namespaceCostNativeID() error = %v", err) + } + fact.Fact.Provenance.NativeID = nativeID +} + +func assertRollupErrorIsAtomic(t testing.TB, request WorkspaceRollupRequest) { + t.Helper() + rollup, err := RollupWorkspaceCosts(request) + if err == nil || !reflect.DeepEqual(rollup, WorkspaceCostRollup{}) { + t.Fatalf("RollupWorkspaceCosts() = %#v, %v; want zero rollup and error", rollup, err) + } +} + +func zeroCostAmounts() CostAmounts { + return CostAmounts{ + CPUCost: "0.00000", CPUCostAdjustment: "0.00000", + GPUCost: "0.00000", GPUCostAdjustment: "0.00000", + RAMCost: "0.00000", RAMCostAdjustment: "0.00000", + PVCost: "0.00000", PVCostAdjustment: "0.00000", + NetworkCost: "0.00000", NetworkCostAdjustment: "0.00000", + LoadBalancerCost: "0.00000", LoadBalancerCostAdjustment: "0.00000", + SharedCost: "0.00000", ExternalCost: "0.00000", TotalCost: "0.00000", + } +} + +func costAmountValues(amounts CostAmounts) []string { + return []string{ + amounts.CPUCost, amounts.CPUCostAdjustment, + amounts.GPUCost, amounts.GPUCostAdjustment, + amounts.RAMCost, amounts.RAMCostAdjustment, + amounts.PVCost, amounts.PVCostAdjustment, + amounts.NetworkCost, amounts.NetworkCostAdjustment, + amounts.LoadBalancerCost, amounts.LoadBalancerCostAdjustment, + amounts.SharedCost, amounts.ExternalCost, amounts.TotalCost, + } +} + +func mustMarshal(t testing.TB, value any) []byte { + t.Helper() + encoded, err := json.Marshal(value) + if err != nil { + t.Fatalf("marshal test value: %v", err) + } + return encoded +} + +func mustUnmarshal(t testing.TB, document []byte, target any) { + t.Helper() + if err := json.Unmarshal(document, target); err != nil { + t.Fatalf("unmarshal test value: %v", err) + } +} diff --git a/sessions/2026-07-18-e13-opencost-workspace-rollup.md b/sessions/2026-07-18-e13-opencost-workspace-rollup.md new file mode 100644 index 0000000..ff08fb5 --- /dev/null +++ b/sessions/2026-07-18-e13-opencost-workspace-rollup.md @@ -0,0 +1,107 @@ +# Session — 2026-07-18 — E13 OpenCost coverage-aware workspace rollup + +**Builder:** Gnani Rahul · **Branch:** `gnanirahulnutakki/feat/e13-opencost-workspace-rollup` +**Slice:** [#284](https://github.com/ArdurAI/sith/issues/284), E13 +[#31](https://github.com/ArdurAI/sith/issues/31) · **Status:** complete local proof + +## [G] Goal + +Preserve successful-empty per-cluster OpenCost coverage and compute one deterministic workspace USD +total for an explicit window without guessing the blocked live transport or team-attribution +contracts. + +## [S] Scope + +- One value-only snapshot envelope around each successful F13.1a projection. +- One explicit expected cluster set and zero or one successful snapshot per reporting cluster. +- Exact workspace, scope, UTC window, USD, fact taxonomy, entity, provenance, payload, and native + identity revalidation. +- Exact component and total aggregation plus sorted expected/reported/empty/missing coverage. +- No client, endpoint, discovery, credentials, port-forward, Kubernetes Service proxy, OCM + transport, persistence, runtime wiring, team grouping, UI/API, stale objective, conversion, + billing, optimization, GPU efficiency, mutation, or execution. + +## [A] Decision and implementation + +- Reconciled every open issue and current `dev`; all higher-priority residuals remain + landed-to-current-contract or explicitly human/upstream blocked, and no competing PR is open. +- Verified current access guidance in the official OpenCost API and installation documentation. + OpenCost recommends operator port forwarding and permits deployment-specific Service/Ingress + exposure; Sith has no accepted owner for discovery, authentication, or TLS. +- Recorded the live-transport options and recommended local-versus-brokered split on parent issue + 31. F13.1 transport remains held pending GR confirmation; no URL, credential, RBAC grant, or + network path was guessed. +- Opened issue 284 and recorded the accepted contract in Notion and the EXTENDED Obsidian Sith + project before source work. +- Added a successful projection snapshot that preserves an empty fact set as reported coverage. +- Added a deterministic workspace rollup with exact decimal aggregation, explicit coverage, nil + observation time when nothing reported, whole-input atomic failure, privacy-minimized output, and + fixed scope/fact/byte/magnitude bounds. +- Extended the AST boundary so the package remains value-only and cannot acquire network, + credential, persistence, process, planning, mutation, or execution capability unnoticed. + +## [T] Focused proof + +- Package tests pass with the race detector and 91.9% statement coverage. +- Positive coverage proves sorted partial coverage, successful-empty versus missing distinction, + exact component totals, deterministic order independence, absent observation time when no scope + reports, and non-retention of namespace/source metadata beyond authorized cluster coverage. +- Adversarial coverage revalidates workspace, scope, UTC window, currency, taxonomy, entity, + provenance, native identity, canonical JSON, namespace identity, amount precision, component + totals, duplication, count, and size bounds with zero partial rollup on every error. +- Native Go fuzzing completed exactly 50,000 generated rollup executions with four workers after + correcting one fuzz-harness-only invalid-RawMessage identity helper; the production path did not + panic or emit a partial/invalid rollup. +- The projection and rollup fuzz boundaries each pass exactly 50,000 generated executions with four + workers. Full repository CI passes formatting, vet, lint with zero issues, `govulncheck` with no + reachable vulnerabilities, race coverage, shell policy tests, Prometheus rules, performance, + end-to-end tests, and the production build. +- PostgreSQL forced-RLS isolation and two 50,000-execution cross-workspace fuzzers pass. The + reproducible four-platform release, SPDX SBOM, checksum, Homebrew, and multi-platform OCI proof + passes. The pinned Kubernetes 1.36.1 kind suite passes in 238.257 seconds. +- CodeRabbit's first complete review found one minor ADR result-shape omission. The ADR now names + the retained coverage categories, `complete`, and optional `observed_at`; source behavior was + unchanged. The second complete review covers all changed files and reports zero findings. +- Manual red-team confirms that `complete` is relative to the caller-authoritative expected set and + that successful-empty presence is a trusted caller claim; the transport-free core cannot prove + either claim independently and does not pretend to. Late invalid input remains atomic, authorized + cluster coverage is the only retained identity, total and component fields are summed + independently, limits fail closed, errors omit raw inputs, race proof is green, and the AST wall + admits no I/O or authority seam. +- The changed-file credential sweep has zero high-signal matches. Live GitHub queues are 0 open + Dependabot, 0 code-scanning, and 0 secret-scanning alerts; no competing PR is open; exact + `origin/dev` remains `3d45529624a275652d4c6793271859dfe5add152`. + +## [S] Security, reliability, and cost + +The rollup revalidates normalized facts and returns no prefix on an invalid later scope or fact. +Only aggregate amounts plus explicit authorized cluster coverage survive; namespace names and raw +provider/workload/source metadata do not. Work is bounded in memory and CPU and creates no network, +cloud API, storage, egress, logging-volume, credential, privilege, or recurring-service cost. + +## [P] Primary sources + +- [OpenCost allocation API](https://opencost.io/docs/integrations/api/) +- [OpenCost installation and access](https://opencost.io/docs/installation/install/) +- [OpenCost v1.120.2](https://github.com/opencost/opencost/releases/tag/v1.120.2) +- [ADR 0011](../docs/adr/0011-opencost-namespace-cost-facts.md) +- [ADR 0012](../docs/adr/0012-opencost-coverage-aware-workspace-rollup.md) + +## [N] Next + +Create one signed DCO/GSTACK commit, publish a PR to `dev`, require exact-head CI/CodeQL/hosted +CodeRabbit, merge while preserving the signed head, and require exact post-merge `dev` evidence +before closing issue 284. + +## [C] Checkpoint #1 + +Issue, decision, code, focused race coverage, adversarial tests, exact 50,000-execution fuzz proof, +README, mirrored roadmap, ADR, and GSTACK journal are present in the EXTENDED worktree. + +## [C] Checkpoint #2 + +The complete local CI, isolation, reproducible release, and real-cluster gates pass from the +EXTENDED worktree. Both OpenCost fuzz boundaries pass 50,000 executions. The first complete +CodeRabbit review's sole documentation finding is corrected and the second review reports zero +findings. Manual red-team, credential scan, security queues, exact base, and no-competing-PR checks +are clean; signed publication remains the fail-closed gate.