diff --git a/README.md b/README.md index 0244843..d9dc548 100644 --- a/README.md +++ b/README.md @@ -616,6 +616,19 @@ retention, repository-to-workload correlation, alert, typed intent, PEP handoff, execution. The cache-backed `sith investigate` path cannot produce R9 until a future reader supplies validated workflow-run graph facts and explicit TIMELINE coverage. +The brain's existing R3 CrashLoop rule can also consume the bounded Elasticsearch +`search/ecs-v1` facts produced from an already-fetched, complete Search API response. The graph +bridge requires exact Elasticsearch source/provenance, an attached Pod identity, a SHA-256 +native/resource identity that recomputes from the retained sanitized aggregate and Pod, and the +closed `logs.cause` values `panic`, `missing-config`, or +`dependency-failure`. It revalidates source bounds, discards count/container/window metadata, and +preserves only the Pod, cause, last classified event time, source, and stale flag. Fact presence +does not infer TELEMETRY coverage, and evidence for one Pod cannot strengthen another. Raw logs, +index/document IDs, query text, labels, URLs, credentials, and user data do not enter the brain or +CLI output. This path adds no Elasticsearch HTTP client, endpoint/index configuration, credential, +query execution, persistence, fleet correlation, typed intent, mutation, or execution; the +cache-backed `sith investigate` command still does not fetch Elasticsearch data. + Every verdict includes its rule, exact cited signals, confidence state, missing lenses, and an advisory command or PR change for the operator to inspect and run. The brain performs no I/O and imports no connector planning, execution, intent, PEP, MCP, or local-operation path. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 52c5df9..0e15aaf 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -337,6 +337,17 @@ sensitive human inspection guidance and adds no GitHub client, token, storage, a typed intent, policy handoff, mutation, or execution. The cache-backed local CLI still has no workflow-run fetch path. +The existing **R3** rule now also accepts the reviewed Elasticsearch `search/ecs-v1` graph fact +through a narrow bridge. Only an attached Pod TELEMETRY `FactDerived` with exact Elasticsearch +source/provenance, matching scope and namespace, a revalidated SHA-256 native/resource identity +bound to the retained workspace, Pod, aggregate, and collection fields, and a closed `logs.cause` +payload can become an observation. The accepted values remain `panic`, +`missing-config`, and `dependency-failure`; count, container, event-window metadata, and the source +fact payload are discarded after validation. The last classified event time and stale flag are +preserved, while TELEMETRY coverage remains entirely caller-declared. Evidence attached to another +Pod cannot strengthen the CrashLoop verdict. This bridge adds no Elasticsearch client, endpoint, +index, query execution, credential, persistence, correlation, typed intent, mutation, or execution. + ## Integration waves (E12) — the connector coverage the brain needs Connectors ship in four waves (`docs/specs/E2-readfed-brain-integrations.md` §4), each scored by @@ -372,6 +383,10 @@ need:** Raw messages are classified in memory and discarded. Missing cluster identity, partial or failed shards, `_source`, unknown fields, and ambiguous values fail closed. This slice adds no HTTP client, index discovery, credentials, persistence, or writes. + Issue #280 connects those already-reviewed facts to R3 without widening the source contract: + exact `elasticsearch` / `search/ecs-v1` provenance and Pod identity are validated again, only the + closed cause classification enters the brain, cross-Pod evidence stays separate, and declared + TELEMETRY coverage and staleness remain authoritative. Issue #216 establishes the AWS autoscaler-evidence contract: one already-fetched EKS `DescribeNodegroup` response becomes a bounded LIVE inventory fact and a bounded LIVE provider-health fact attached to an already-trusted Sith cluster. The response's partition, diff --git a/docs/adr/0008-deterministic-advisory-brain.md b/docs/adr/0008-deterministic-advisory-brain.md index 64ee6b8..2c0e7c2 100644 --- a/docs/adr/0008-deterministic-advisory-brain.md +++ b/docs/adr/0008-deterministic-advisory-brain.md @@ -107,6 +107,40 @@ contract in [Workflow runs](https://docs.github.com/en/rest/actions/workflow-run and the closed conclusion vocabulary in [Checks](https://docs.github.com/en/rest/guides/using-the-rest-api-to-interact-with-checks). +### 2026-07-18 extension: Elasticsearch log-cause graph bridge for R3 + +R3 continues to use its existing generic `logs.cause` TELEMETRY strengthener. The extension is a +narrow graph admission path for the already-reviewed Elasticsearch projector, not a new rule. +`FromGraphFacts` first validates the workspace-bounded graph, then admits only an attached +TELEMETRY `FactDerived` whose resource is an Elasticsearch `LogSignal`, source kind and provenance +adapter are both `elasticsearch`, protocol is `search/ecs-v1`, source/scope/namespace agree, and +the entity carries exactly one Pod identity. The source hash uses only the retained workspace, +scope, namespace, Pod, optional container, cause, count, first/last event times, and collection +time, allowing the bridge to recompute both the full native ID and resource-name prefix. Extra +resource attributes, display fields, entity dimensions, cross-Pod retargeting, and malformed, +noncanonical, or mismatched SHA-256 identities fail closed. + +The exact-case payload is closed to `key`, `value`, `count`, `first_event_at`, `last_event_at`, and +optional `container`. The bridge revalidates the source projector's byte, count, event-window, +clock-skew, and text bounds, accepts only `logs.cause` with `panic`, `missing-config`, or +`dependency-failure`, then discards count, container, and the source payload. The resulting +observation retains only exact Pod identity, the closed cause, last classified event time, source, +and stale flag. Graph fact presence never creates TELEMETRY coverage; missing, unavailable, stale, +or observation-stale TELEMETRY evidence therefore cannot produce a fully confirmed R3 result. +Entity grouping uses exact scope, namespace, kind, and name, so evidence attached to one Pod cannot +strengthen another Pod or create a fleet-wide cause claim. + +Raw messages are classified and discarded by the source projector before this bridge runs. Index +and document IDs, query text, labels, URLs, credentials, user data, and container/count metadata do +not enter the observation, citation, replay, or renderer. The extension adds bounded in-memory JSON +validation only: no HTTP/TLS client, endpoint/index configuration, API key, mapping discovery, +query execution, pagination, retention, socket, filesystem, database, process, correlation, typed +intent, dispatch, mutation, or cloud resource. Elastic documents the underlying Search API and +selected-field behavior in [Search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) +and [Retrieve selected fields](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields), +and the normalized identity fields in [ECS orchestrator fields](https://www.elastic.co/docs/reference/ecs/ecs-orchestrator) +and [Filebeat Kubernetes fields](https://www.elastic.co/docs/reference/beats/filebeat/exported-fields-kubernetes-processor). + ## Consequences - Investigations are offline, reproducible, replayable, and inspectable. The same observation @@ -115,12 +149,12 @@ and the closed conclusion vocabulary in telemetry-dependent cause variant. This is intentionally less confident than an opaque guess. - Phase L does not auto-port-forward to Prometheus/Loki, retain telemetry series, read Git desired state, or infer absent evidence. Those connectors can later emit the same observation contract. -- The existing cache-backed CLI does not fetch Argo Applications or GitHub workflow runs. R8 and R9 - are available only to callers that already possess validated graph facts and explicitly declare - TIMELINE coverage. +- The existing cache-backed CLI does not fetch Argo Applications, GitHub workflow runs, or + Elasticsearch logs. R8, R9, and Elasticsearch-strengthened R3 are available only to callers that + already possess validated graph facts and explicitly declare the relevant lens coverage. - There is no hosted or cloud cost. Runtime cost is one existing tier-1 hydration pass plus in-memory rule evaluation over the returned records or an existing bounded graph. No extra - Kubernetes watch, Argo/GitHub request, storage, or network egress is introduced. + Kubernetes watch, Argo/GitHub/Elasticsearch request, storage, or network egress is introduced. - Advisory strings are not shell execution. Operators remain responsible for reviewing and running a suggestion with their own kubeconfig identity. diff --git a/docs/specs/E2-readfed-brain-integrations.md b/docs/specs/E2-readfed-brain-integrations.md index b9a5403..add7fa9 100644 --- a/docs/specs/E2-readfed-brain-integrations.md +++ b/docs/specs/E2-readfed-brain-integrations.md @@ -403,6 +403,20 @@ hub). Weights are indicative, to be tuned against real incidents. advisory* accordingly. - **Coverage gate.** LIVE + logs(TELEMETRY) are the crux; DESIRED/TIMELINE disambiguate. +**Implemented Elasticsearch graph seam (#280).** The bounded `search/ecs-v1` projector described +in #214 can feed this existing signal without exposing raw logs to the brain. `FromGraphFacts` +accepts only an attached Pod TELEMETRY `FactDerived` whose source kind and provenance adapter are +both `elasticsearch`, protocol is exact, source/scope/namespace agree, and the SHA-256 +native/resource identity recomputes from the retained workspace, Pod, aggregate, and collection +fields. The payload contains only `key`, `value`, `count`, +`first_event_at`, `last_event_at`, and optional `container`. The key must be `logs.cause`; the value +must be `panic`, `missing-config`, or `dependency-failure`; source projector count, time-window, +clock-skew, and optional-container bounds are revalidated. Only the Pod identity, cause, last event +time, source, and stale flag survive as an observation. Count and container metadata are discarded. +Caller-declared coverage is copied exactly and never inferred from fact presence. Different Pods +remain different evaluator entities, so log evidence for one Pod cannot strengthen another Pod's +CrashLoop. This is an in-memory graph bridge, not an Elasticsearch reader or freshness claim. + #### R4 — Config drift (live diverged from desired) - **Symptom.** Live ≠ desired for a workload/Application (Argo `OutOfSync`, or `diff` verb shows a @@ -595,7 +609,7 @@ the *reasoning*. Decision deferred to the owner (see §7). | **GitHub** | RA + TA | di, rd, qy, df, **pl/ex** (`gitops.open-pr`), vf | DESIRED (manifests), TIMELINE (commits/PR/deploys) | **T1** | read=local, write=hub | Bounded pure projectors now normalize caller-fetched merged-PR and completed workflow-run failure evidence; the HTTP/token reader remains future. `gitops.open-pr` is the first governed write (P2). | | **ArgoCD** | RA + BR + TA | di, rd, qy, **df**, pl/ex (`argocd.sync`,`argocd.rollback`), vf | DESIRED, LIVE, TIMELINE (sync history), drift | **T1** | read=local, sync=hub | Richest single connector — 3 lenses + the exemplar of the `diff` verb. Central to R1, R4. Application CRDs read via kubeconfig. | | **Prometheus** | RA + query-through | di, **qy**, rd (alerts) | TELEMETRY | **T1** | local-if-reachable / hub | Query-through, not retained. Central to R2, R5, R6 and R1 validation. | -| **Elasticsearch** | RA + query-through | di, **qy** | TELEMETRY (logs) | **T2** | hub (local if creds) | Log search for R3. Auth/index-mapping variance → T2. | +| **Elasticsearch** | RA + query-through | di, **qy** | TELEMETRY (logs) | **T2** | hub (local if creds) | Bounded sanitized `search/ecs-v1` cause facts now bridge into R3; HTTP/auth/index/query execution remains future. Auth/index-mapping variance → T2. | | **AWS** | RA (enum/cred) | di, rd, (qy CloudWatch later) | LIVE (nodes/infra), TIMELINE (CloudTrail later) | **T2** | cluster-enum local; deep facts hub | Enumeration + short-lived token minting (no long-lived keys). Feeds R6 (nodegroup/autoscaler). | #### Wave 2 — the desired-state / diff pipeline diff --git a/internal/brain/graph.go b/internal/brain/graph.go index 524c484..632f960 100644 --- a/internal/brain/graph.go +++ b/internal/brain/graph.go @@ -4,11 +4,15 @@ package brain import ( "bytes" + "crypto/sha256" + "encoding/hex" "encoding/json" "fmt" "io" "strings" "time" + "unicode" + "unicode/utf8" "github.com/ArdurAI/sith/internal/fleet" ) @@ -21,7 +25,15 @@ const ( githubGraphSource = "github" githubWorkflowGraphProtocolVersion = "workflow-runs/2026-03-10" maxGitHubWorkflowChangePayload = 4 << 10 - maxGraphJSONDepth = 64 + + elasticsearchGraphSource = "elasticsearch" + elasticsearchGraphProtocolVersion = "search/ecs-v1" + maxElasticsearchLogCausePayload = 4 << 10 + maxElasticsearchCauseCount = 128 + maxElasticsearchCauseWindow = 15 * time.Minute + maxElasticsearchClockSkew = 5 * time.Minute + + maxGraphJSONDepth = 64 ) type argoChangePayload struct { @@ -42,6 +54,15 @@ type githubWorkflowChangePayload struct { EventAt time.Time `json:"event_at"` } +type elasticsearchLogCausePayload struct { + Key string `json:"key"` + Value string `json:"value"` + Count int `json:"count"` + FirstEventAt time.Time `json:"first_event_at"` + LastEventAt time.Time `json:"last_event_at"` + Container string `json:"container,omitempty"` +} + // FromGraphFacts converts reviewed graph facts into normalized brain observations while preserving // caller-declared coverage. It performs no discovery, I/O, coverage inference, or mutation. func FromGraphFacts( @@ -93,9 +114,16 @@ func FromGraphFacts( func observationFromGraphFact(fact fleet.GraphFact) (Observation, bool, error) { provenance := fact.Fact.Provenance ref := fact.Fact.Ref + if provenance.ProtocolV == elasticsearchGraphProtocolVersion { + return observationFromElasticsearchLogCauseGraphFact(fact) + } if provenance.ProtocolV == githubWorkflowGraphProtocolVersion { return observationFromGitHubWorkflowGraphFact(fact) } + if (ref.SourceKind == elasticsearchGraphSource || provenance.Adapter == elasticsearchGraphSource) && + ref.Kind == "LogSignal" { + return observationFromElasticsearchLogCauseGraphFact(fact) + } if fact.Fact.Kind != fleet.FactChange || fact.Lens != fleet.LensTimeline { return Observation{}, false, nil } @@ -183,6 +211,130 @@ func observationFromArgoGraphFact(fact fleet.GraphFact) (Observation, bool, erro }, true, nil } +func observationFromElasticsearchLogCauseGraphFact(fact fleet.GraphFact) (Observation, bool, error) { + provenance := fact.Fact.Provenance + ref := fact.Fact.Ref + if fact.Fact.Kind != fleet.FactDerived || fact.Lens != fleet.LensTelemetry { + return Observation{}, false, fmt.Errorf("project Elasticsearch log-cause fact: fact must be derived TELEMETRY") + } + if ref.SourceKind != elasticsearchGraphSource || provenance.Adapter != elasticsearchGraphSource { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: source and provenance must both be %q", + elasticsearchGraphSource, + ) + } + if provenance.ProtocolV != elasticsearchGraphProtocolVersion { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: unsupported protocol version %q", + provenance.ProtocolV, + ) + } + if provenance.DeepLink != "" || provenance.Collector != "" { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: unexpected provenance metadata", + ) + } + if ref.Kind != "LogSignal" || ref.Scope == "" || ref.Namespace == "" { + return Observation{}, false, fmt.Errorf("project Elasticsearch log-cause fact: resource identity is invalid") + } + if fact.Entity == nil { + return Observation{}, false, fmt.Errorf("project Elasticsearch log-cause fact: Pod entity is required") + } + entity := *fact.Entity + if entity.Cluster != ref.Scope || entity.Namespace != ref.Namespace || entity.Pod == "" || + entity.Kind != "" || entity.Name != "" || entity.Node != "" || entity.ImageDigest != "" { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: source and entity must identify one exact Pod", + ) + } + if fact.Fact.Source != ref.Scope { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: evidence source must match the Elasticsearch scope", + ) + } + if len(ref.Attributes) != 0 || len(fact.Fact.Display) != 0 { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: unexpected attributes or display fields", + ) + } + if len(fact.Fact.Observed) == 0 || len(fact.Fact.Observed) > maxElasticsearchLogCausePayload { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: payload must be between 1 and %d bytes", + maxElasticsearchLogCausePayload, + ) + } + + payload, err := decodeElasticsearchLogCausePayload(fact.Fact.Observed) + if err != nil { + return Observation{}, false, err + } + if payload.Key != "logs.cause" { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: unsupported key %q", + payload.Key, + ) + } + switch payload.Value { + case "panic", "missing-config", "dependency-failure": + default: + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: unsupported cause %q", + payload.Value, + ) + } + if payload.Count <= 0 || payload.Count > maxElasticsearchCauseCount { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: count must be between 1 and %d", + maxElasticsearchCauseCount, + ) + } + if payload.FirstEventAt.IsZero() || payload.LastEventAt.IsZero() || + payload.LastEventAt.Before(payload.FirstEventAt) || + payload.LastEventAt.Sub(payload.FirstEventAt) > maxElasticsearchCauseWindow { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: event interval is invalid", + ) + } + if payload.LastEventAt.After(fact.Fact.ObservedAt.Add(maxElasticsearchClockSkew)) { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: last event exceeds collection clock skew", + ) + } + if payload.Container != "" && !validElasticsearchGraphText(payload.Container) { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: container identity is invalid", + ) + } + digest, valid := validElasticsearchNativeID(provenance.NativeID) + if !valid { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: native identity is inconsistent with the log signal", + ) + } + wantDigest, err := elasticsearchLogCauseDigest(fact, payload) + if err != nil || digest != wantDigest || ref.Name != "log-"+wantDigest[:32] { + return Observation{}, false, fmt.Errorf( + "project Elasticsearch log-cause fact: native identity is inconsistent with the log signal", + ) + } + + return Observation{ + Ref: fleet.ResourceRef{ + SourceKind: elasticsearchGraphSource, + Scope: entity.Cluster, + Kind: "Pod", + Namespace: entity.Namespace, + Name: entity.Pod, + }, + Lens: fleet.LensTelemetry, + Key: "logs.cause", + Value: payload.Value, + ObservedAt: payload.LastEventAt, + Source: fact.Fact.Source, + Stale: fact.Fact.Stale, + }, true, nil +} + func observationFromGitHubWorkflowGraphFact(fact fleet.GraphFact) (Observation, bool, error) { provenance := fact.Fact.Provenance ref := fact.Fact.Ref @@ -303,6 +455,119 @@ func decodeGitHubWorkflowChangePayload(raw json.RawMessage) (githubWorkflowChang return payload, nil } +func decodeElasticsearchLogCausePayload(raw json.RawMessage) (elasticsearchLogCausePayload, error) { + if !utf8.Valid(raw) { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: decode payload: invalid UTF-8", + ) + } + if err := rejectDuplicateGraphJSON(raw); err != nil { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: decode payload: %w", + err, + ) + } + var fields map[string]json.RawMessage + if err := json.Unmarshal(raw, &fields); err != nil { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: decode payload fields: %w", + err, + ) + } + for field := range fields { + switch field { + case "key", "value", "count", "first_event_at", "last_event_at", "container": + default: + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: payload field %q is unsupported", + field, + ) + } + } + if container, present := fields["container"]; present && bytes.Equal(bytes.TrimSpace(container), []byte("null")) { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: container must be a non-null string", + ) + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + var payload elasticsearchLogCausePayload + if err := decoder.Decode(&payload); err != nil { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: decode payload: %w", + err, + ) + } + var trailer json.RawMessage + if err := decoder.Decode(&trailer); err != io.EOF { + if err == nil { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: payload must contain one JSON value", + ) + } + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: decode payload trailer: %w", + err, + ) + } + if _, present := fields["container"]; present && payload.Container == "" { + return elasticsearchLogCausePayload{}, fmt.Errorf( + "project Elasticsearch log-cause fact: container must be a non-empty string when present", + ) + } + return payload, nil +} + +func validElasticsearchNativeID(nativeID string) (string, bool) { + digest, found := strings.CutPrefix(nativeID, "sha256:") + if !found || len(digest) != 64 { + return "", false + } + for _, character := range digest { + if (character < '0' || character > '9') && (character < 'a' || character > 'f') { + return "", false + } + } + return digest, true +} + +func elasticsearchLogCauseDigest(fact fleet.GraphFact, payload elasticsearchLogCausePayload) (string, error) { + identity, err := json.Marshal(struct { + Workspace string `json:"workspace"` + Scope string `json:"scope"` + Namespace string `json:"namespace"` + Pod string `json:"pod"` + Container string `json:"container,omitempty"` + Cause string `json:"cause"` + Count int `json:"count"` + FirstEventAt time.Time `json:"first_event_at"` + LastEventAt time.Time `json:"last_event_at"` + ObservedAt time.Time `json:"observed_at"` + }{ + Workspace: fact.Fact.Workspace, Scope: fact.Fact.Ref.Scope, Namespace: fact.Fact.Ref.Namespace, + Pod: fact.Entity.Pod, Container: payload.Container, Cause: payload.Value, Count: payload.Count, + FirstEventAt: payload.FirstEventAt, LastEventAt: payload.LastEventAt, + ObservedAt: fact.Fact.ObservedAt, + }) + if err != nil { + return "", err + } + digest := sha256.Sum256(identity) + return hex.EncodeToString(digest[:]), nil +} + +func validElasticsearchGraphText(value string) bool { + if value == "" || len(value) > 253 || !utf8.ValidString(value) || strings.TrimSpace(value) != value { + return false + } + for _, character := range value { + if unicode.IsControl(character) { + return false + } + } + return true +} + func rejectDuplicateGraphJSON(document []byte) error { decoder := json.NewDecoder(bytes.NewReader(document)) decoder.UseNumber() diff --git a/internal/brain/graph_elasticsearch_test.go b/internal/brain/graph_elasticsearch_test.go new file mode 100644 index 0000000..912a32f --- /dev/null +++ b/internal/brain/graph_elasticsearch_test.go @@ -0,0 +1,589 @@ +// SPDX-License-Identifier: Apache-2.0 + +package brain + +import ( + "encoding/json" + "reflect" + "strings" + "testing" + "time" + + connectorelasticsearch "github.com/ArdurAI/sith/internal/connector/elasticsearch" + "github.com/ArdurAI/sith/internal/fleet" +) + +func TestFromGraphFactsProjectsReviewedElasticsearchLogCausesIntoR3(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 123000000, time.UTC) + tests := []struct { + cause string + message string + }{ + {cause: "panic", message: "panic: discard-raw-secret"}, + {cause: "missing-config", message: "missing required environment variable DISCARD_RAW_SECRET"}, + {cause: "dependency-failure", message: "failed to connect to discard-raw-dependency"}, + } + for _, test := range tests { + test := test + t.Run(test.cause, func(t *testing.T) { + t.Parallel() + facts := projectedElasticsearchLogCause(t, test.message, eventAt) + input, err := FromGraphFacts( + fleet.LocalWorkspace, + facts, + covered(fleet.LensLive, fleet.LensTelemetry), + ) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + if len(input.Observations) != 1 { + t.Fatalf("observations = %#v, want one sanitized log cause", input.Observations) + } + observation := input.Observations[0] + wantRef := fleet.ResourceRef{ + SourceKind: "elasticsearch", Scope: "cluster-a", Kind: "Pod", + Namespace: "payments", Name: "api-0", + } + if !reflect.DeepEqual(observation.Ref, wantRef) || + observation.Lens != fleet.LensTelemetry || + observation.Key != "logs.cause" || + observation.Value != test.cause || + !observation.ObservedAt.Equal(eventAt) || + observation.Source != "cluster-a" || + observation.Stale { + t.Fatalf("observation = %#v, want exact Pod-scoped Elasticsearch cause", observation) + } + + input.Observations = append(input.Observations, crashLoopObservation(eventAt.Add(time.Minute))) + result, err := Evaluate(input) + if err != nil { + t.Fatalf("Evaluate() error = %v", err) + } + if len(result.Verdicts) != 1 { + t.Fatalf("verdicts = %#v, want one R3 verdict", result.Verdicts) + } + verdict := result.Verdicts[0] + if verdict.Rule != RuleCrashLoop || verdict.Status != StatusConfirmed || + verdict.Score != 6 || verdict.FleetWide { + t.Fatalf("verdict = %#v, want confirmed entity-local R3", verdict) + } + citations := citationsForPredicate(verdict.Citations, "logs.cause") + if len(citations) != 1 || citations[0].Observed != test.cause || + !citations[0].ObservedAt.Equal(eventAt) || + citations[0].Source != "cluster-a" || + citations[0].Stale { + t.Fatalf("log citations = %#v, want one fresh closed cause", citations) + } + + encoded, err := json.Marshal(struct { + Input Investigation `json:"input"` + Result Result `json:"result"` + }{Input: input, Result: result}) + if err != nil { + t.Fatalf("marshal projected result: %v", err) + } + for _, discarded := range []string{ + "discard-raw-secret", + "DISCARD_RAW_SECRET", + "discard-raw-dependency", + `"count"`, + `"container"`, + `"first_event_at"`, + `"last_event_at"`, + } { + if strings.Contains(string(encoded), discarded) { + t.Fatalf("brain output retained discarded Elasticsearch data %q: %s", discarded, encoded) + } + } + }) + } +} + +func TestFromGraphFactsPreservesElasticsearchStalenessAndDeclaredCoverage(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + tests := []struct { + name string + stale bool + coverage map[fleet.Lens]LensCoverage + }{ + { + name: "stale fact", + stale: true, + coverage: covered(fleet.LensLive, fleet.LensTelemetry), + }, + { + name: "telemetry coverage omitted", + coverage: covered(fleet.LensLive), + }, + { + name: "telemetry coverage unavailable", + coverage: map[fleet.Lens]LensCoverage{ + fleet.LensLive: {Available: true}, + fleet.LensTelemetry: {Reason: "log connector unavailable"}, + }, + }, + { + name: "telemetry coverage stale", + coverage: map[fleet.Lens]LensCoverage{ + fleet.LensLive: {Available: true}, + fleet.LensTelemetry: {Available: true, Stale: true, Reason: "log search stale"}, + }, + }, + } + for _, test := range tests { + test := test + t.Run(test.name, func(t *testing.T) { + t.Parallel() + facts := projectedElasticsearchLogCause(t, "panic: discard-me", eventAt) + facts[0].Fact.Stale = test.stale + input, err := FromGraphFacts(fleet.LocalWorkspace, facts, test.coverage) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + input.Observations = append(input.Observations, crashLoopObservation(eventAt.Add(time.Minute))) + result, err := Evaluate(input) + if err != nil { + t.Fatalf("Evaluate() error = %v", err) + } + if len(result.Verdicts) != 1 || result.Verdicts[0].Rule != RuleCrashLoop || + result.Verdicts[0].Status != StatusDetected || + !reflect.DeepEqual(result.Verdicts[0].MissingLenses, []fleet.Lens{fleet.LensTelemetry}) { + t.Fatalf("verdicts = %#v, want coverage-honest detected R3", result.Verdicts) + } + citations := citationsForPredicate(result.Verdicts[0].Citations, "logs.cause") + if len(citations) != 1 || citations[0].Stale != test.stale { + t.Fatalf("log citations = %#v, stale = %t", citations, test.stale) + } + }) + } + + facts := projectedElasticsearchLogCause(t, "panic: discard-me", eventAt) + input, err := FromGraphFacts(fleet.LocalWorkspace, facts, nil) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + if len(input.Coverage) != 0 { + t.Fatalf("coverage = %#v, fact presence must not infer coverage", input.Coverage) + } +} + +func TestFromGraphFactsFailsClosedOnAmbiguousElasticsearchLogCauseFacts(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + baseFacts := projectedElasticsearchLogCause(t, "panic: discard-me", eventAt) + if len(baseFacts) != 1 { + t.Fatalf("base facts = %#v, want one log-cause fact", baseFacts) + } + base := baseFacts[0] + payload := func(key, value string, count int, first, last time.Time, container, extra string) json.RawMessage { + t.Helper() + raw := `{"key":` + quotedJSON(t, key) + + `,"value":` + quotedJSON(t, value) + + `,"count":` + quotedJSON(t, count) + + `,"first_event_at":` + quotedJSON(t, first.Format(time.RFC3339Nano)) + + `,"last_event_at":` + quotedJSON(t, last.Format(time.RFC3339Nano)) + if container != "" { + raw += `,"container":` + quotedJSON(t, container) + } + return json.RawMessage(raw + extra + `}`) + } + validPayload := func(extra string) json.RawMessage { + t.Helper() + return payload("logs.cause", "panic", 1, eventAt, eventAt, "api", extra) + } + + tests := []struct { + name string + mutate func(*fleet.GraphFact) + }{ + {name: "workspace mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Workspace = "other" + }}, + {name: "both source fields mismatch with exact protocol", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.SourceKind = "other" + fact.Fact.Provenance.Adapter = "other" + }}, + {name: "source kind mismatch", mutate: func(fact *fleet.GraphFact) { fact.Fact.Ref.SourceKind = "other" }}, + {name: "provenance adapter mismatch", mutate: func(fact *fleet.GraphFact) { fact.Fact.Provenance.Adapter = "other" }}, + {name: "exact protocol on non-telemetry fact", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Kind = fleet.FactHealth + fact.Lens = fleet.LensLive + }}, + {name: "protocol mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.ProtocolV = "search/ecs-v2" + }}, + {name: "unexpected provenance deep link", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.DeepLink = "https://private.example.invalid/logs" + }}, + {name: "unexpected provenance collector", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.Collector = "unreviewed-collector" + }}, + {name: "resource kind mismatch", mutate: func(fact *fleet.GraphFact) { fact.Fact.Ref.Kind = "OtherSignal" }}, + {name: "resource scope missing", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Scope = "" + fact.Fact.Source = "" + fact.Entity.Cluster = "" + }}, + {name: "resource namespace missing", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Namespace = "" + fact.Entity.Namespace = "" + }}, + {name: "Pod entity missing", mutate: func(fact *fleet.GraphFact) { fact.Entity = nil }}, + {name: "entity cluster mismatch", mutate: func(fact *fleet.GraphFact) { fact.Entity.Cluster = "cluster-b" }}, + {name: "entity namespace mismatch", mutate: func(fact *fleet.GraphFact) { fact.Entity.Namespace = "other" }}, + {name: "entity Pod missing", mutate: func(fact *fleet.GraphFact) { fact.Entity.Pod = "" }}, + {name: "entity Pod retargeted", mutate: func(fact *fleet.GraphFact) { fact.Entity.Pod = "api-1" }}, + {name: "source scope retargeted consistently", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Scope = "cluster-b" + fact.Fact.Source = "cluster-b" + fact.Entity.Cluster = "cluster-b" + }}, + {name: "namespace retargeted consistently", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Namespace = "other" + fact.Entity.Namespace = "other" + }}, + {name: "entity carries kind", mutate: func(fact *fleet.GraphFact) { + fact.Entity.Kind = "Pod" + fact.Entity.Name = "api-0" + }}, + {name: "entity carries node", mutate: func(fact *fleet.GraphFact) { fact.Entity.Node = "worker-a" }}, + {name: "entity carries image digest", mutate: func(fact *fleet.GraphFact) { + fact.Entity.ImageDigest = "sha256:" + strings.Repeat("a", 64) + }}, + {name: "evidence source mismatch", mutate: func(fact *fleet.GraphFact) { fact.Fact.Source = "cluster-b" }}, + {name: "unexpected reference attributes", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Attributes = map[string]string{"message": "discard-me"} + }}, + {name: "unexpected display field", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Display = []fleet.DisplayField{{Name: "message", Value: "discard-me"}} + }}, + {name: "native prefix mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.NativeID = "md5:" + strings.Repeat("a", 64) + }}, + {name: "native digest length mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.NativeID = "sha256:" + strings.Repeat("a", 63) + }}, + {name: "native digest case mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Provenance.NativeID = "sha256:" + strings.Repeat("A", 64) + }}, + {name: "resource digest prefix mismatch", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Ref.Name = "log-" + strings.Repeat("b", 32) + }}, + {name: "unknown payload field", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = validPayload(`,"message":"discard-me"`) + }}, + {name: "duplicate payload field", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = validPayload(`,"value":"panic"`) + }}, + {name: "mixed-case payload alias", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = validPayload(`,"VALUE":"panic"`) + }}, + {name: "unsupported key", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.message", "panic", 1, eventAt, eventAt, "api", "") + }}, + {name: "unsupported cause", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "credential-failure", 1, eventAt, eventAt, "api", "") + }}, + {name: "supported cause retargeted without native identity", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "missing-config", 1, eventAt, eventAt, "api", "") + }}, + {name: "valid count retargeted without native identity", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 2, eventAt, eventAt, "api", "") + }}, + {name: "valid event interval retargeted without native identity", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt.Add(-time.Second), eventAt, "api", "") + }}, + {name: "valid container retargeted without native identity", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt, eventAt, "other", "") + }}, + {name: "collection time retargeted without native identity", mutate: func(fact *fleet.GraphFact) { + fact.Fact.ObservedAt = fact.Fact.ObservedAt.Add(time.Second) + }}, + {name: "missing key", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `"key":"logs.cause",`, "", 1)) + }}, + {name: "missing value", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `,"value":"panic"`, "", 1)) + }}, + {name: "missing count", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `,"count":1`, "", 1)) + }}, + {name: "missing first event", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace( + string(validPayload("")), + `,"first_event_at":"`+eventAt.Format(time.RFC3339Nano)+`"`, + "", + 1, + )) + }}, + {name: "missing last event", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace( + string(validPayload("")), + `,"last_event_at":"`+eventAt.Format(time.RFC3339Nano)+`"`, + "", + 1, + )) + }}, + {name: "zero count", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 0, eventAt, eventAt, "api", "") + }}, + {name: "negative count", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", -1, eventAt, eventAt, "api", "") + }}, + {name: "count above projector bound", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", maxElasticsearchCauseCount+1, eventAt, eventAt, "api", "") + }}, + {name: "count has wrong type", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `"count":1`, `"count":"1"`, 1)) + }}, + {name: "event order reversed", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt, eventAt.Add(-time.Second), "api", "") + }}, + {name: "event interval above projector bound", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload( + "logs.cause", "panic", 1, + eventAt.Add(-maxElasticsearchCauseWindow-time.Nanosecond), eventAt, "api", "", + ) + }}, + {name: "last event exceeds clock skew", mutate: func(fact *fleet.GraphFact) { + future := fact.Fact.ObservedAt.Add(maxElasticsearchClockSkew + time.Nanosecond) + fact.Fact.Observed = payload("logs.cause", "panic", 1, future, future, "api", "") + }}, + {name: "null event time", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace( + string(validPayload("")), + `"first_event_at":"`+eventAt.Format(time.RFC3339Nano)+`"`, + `"first_event_at":null`, + 1, + )) + }}, + {name: "container null", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `"container":"api"`, `"container":null`, 1)) + }}, + {name: "container empty", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `"container":"api"`, `"container":""`, 1)) + }}, + {name: "container invalid control", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt, eventAt, "api\nother", "") + }}, + {name: "container has surrounding whitespace", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt, eventAt, " api ", "") + }}, + {name: "container oversized", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = payload("logs.cause", "panic", 1, eventAt, eventAt, strings.Repeat("a", 254), "") + }}, + {name: "container wrong type", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Replace(string(validPayload("")), `"container":"api"`, `"container":7`, 1)) + }}, + {name: "empty payload", mutate: func(fact *fleet.GraphFact) { fact.Fact.Observed = nil }}, + {name: "oversized payload", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(strings.Repeat(" ", maxElasticsearchLogCausePayload+1)) + }}, + {name: "multiple JSON values", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = append(append(json.RawMessage(nil), fact.Fact.Observed...), []byte(` {}`)...) + }}, + {name: "malformed JSON", mutate: func(fact *fleet.GraphFact) { + fact.Fact.Observed = json.RawMessage(`{"key":`) + }}, + } + for _, test := range tests { + test := test + t.Run(test.name, func(t *testing.T) { + t.Parallel() + fact := cloneBrainGraphFact(base) + test.mutate(&fact) + if _, err := FromGraphFacts( + fleet.LocalWorkspace, + []fleet.GraphFact{fact}, + covered(fleet.LensTelemetry), + ); err == nil { + t.Fatalf("FromGraphFacts() error = nil for %#v", fact) + } + }) + } +} + +func TestDecodeElasticsearchLogCausePayloadRejectsInvalidUTF8(t *testing.T) { + t.Parallel() + raw := append([]byte(`{"key":"logs.cause","value":"panic","count":1,"first_event_at":"2026-07-18T23:30:00Z","last_event_at":"2026-07-18T23:30:00Z","container":"`), 0xff) + raw = append(raw, []byte(`"}`)...) + + if _, err := decodeElasticsearchLogCausePayload(raw); err == nil || !strings.Contains(err.Error(), "invalid UTF-8") { + t.Fatalf("decodeElasticsearchLogCausePayload() error = %v, want invalid UTF-8", err) + } +} + +func TestFromGraphFactsDoesNotCorrelateElasticsearchCauseAcrossPods(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + facts := projectedElasticsearchLogCause(t, "panic: discard-me", eventAt) + input, err := FromGraphFacts( + fleet.LocalWorkspace, + facts, + covered(fleet.LensLive, fleet.LensTelemetry), + ) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + crashLoop := crashLoopObservation(eventAt.Add(time.Minute)) + crashLoop.Ref.Name = "api-1" + input.Observations = append(input.Observations, crashLoop) + + result, err := Evaluate(input) + if err != nil { + t.Fatalf("Evaluate() error = %v", err) + } + if len(result.Verdicts) != 1 || result.Verdicts[0].Rule != RuleCrashLoop || + result.Verdicts[0].Status != StatusDetected || result.Verdicts[0].Score != 3 || + len(citationsForPredicate(result.Verdicts[0].Citations, "logs.cause")) != 0 { + t.Fatalf("verdicts = %#v, cross-Pod log evidence must not strengthen R3", result.Verdicts) + } +} + +func TestFromGraphFactsIgnoresUnrelatedDerivedTelemetryFacts(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + facts := projectedElasticsearchLogCause(t, "panic: discard-me", eventAt) + facts[0].Fact.Ref.SourceKind = "other" + facts[0].Fact.Ref.Kind = "OtherSignal" + facts[0].Fact.Provenance.Adapter = "other" + facts[0].Fact.Provenance.ProtocolV = "other/v1" + + input, err := FromGraphFacts(fleet.LocalWorkspace, facts, covered(fleet.LensTelemetry)) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + if len(input.Observations) != 0 { + t.Fatalf("observations = %#v, unrelated derived fact must be ignored", input.Observations) + } +} + +func FuzzFromGraphFactsElasticsearchLogCausePayload(f *testing.F) { + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + base := fleet.GraphFact{ + Fact: fleet.Fact{ + Evidence: fleet.Evidence{ + Ref: fleet.ResourceRef{ + SourceKind: "elasticsearch", Scope: "cluster-a", Kind: "LogSignal", + Namespace: "payments", Name: "log-" + strings.Repeat("0", 32), + }, + Kind: fleet.FactDerived, + ObservedAt: eventAt.Add(time.Minute), + Source: "cluster-a", + Provenance: fleet.Provenance{ + Adapter: "elasticsearch", ProtocolV: "search/ecs-v1", + NativeID: "sha256:" + strings.Repeat("0", 64), + }, + }, + Workspace: fleet.LocalWorkspace, + }, + Lens: fleet.LensTelemetry, + Entity: &fleet.EntityRef{ + Cluster: "cluster-a", Namespace: "payments", Pod: "api-0", + }, + } + for _, seed := range [][]byte{ + []byte(`{"key":"logs.cause","value":"panic","count":1,"first_event_at":"2026-07-18T23:30:00Z","last_event_at":"2026-07-18T23:30:00Z","container":"api"}`), + []byte(`{"key":"logs.cause","value":"dependency-failure","count":2,"first_event_at":"2026-07-18T23:29:00Z","last_event_at":"2026-07-18T23:30:00Z"}`), + []byte(`{"KEY":"logs.cause"}`), + []byte(`null`), + } { + f.Add(seed) + } + f.Fuzz(func(t *testing.T, payload []byte) { + fact := cloneBrainGraphFact(base) + fact.Fact.Observed = append(json.RawMessage(nil), payload...) + decoded, decodeErr := decodeElasticsearchLogCausePayload(fact.Fact.Observed) + if decodeErr == nil { + digest, digestErr := elasticsearchLogCauseDigest(fact, decoded) + if digestErr != nil { + t.Fatalf("elasticsearchLogCauseDigest() error = %v", digestErr) + } + fact.Fact.Provenance.NativeID = "sha256:" + digest + fact.Fact.Ref.Name = "log-" + digest[:32] + } + input, err := FromGraphFacts( + fleet.LocalWorkspace, + []fleet.GraphFact{fact}, + covered(fleet.LensTelemetry), + ) + if err != nil { + return + } + if len(input.Observations) != 1 { + t.Fatalf("successful projection observations = %#v", input.Observations) + } + observation := input.Observations[0] + if observation.Ref.SourceKind != "elasticsearch" || + observation.Ref.Scope != "cluster-a" || + observation.Ref.Kind != "Pod" || + observation.Ref.Namespace != "payments" || + observation.Ref.Name != "api-0" || + observation.Lens != fleet.LensTelemetry || + observation.Key != "logs.cause" || + (observation.Value != "panic" && + observation.Value != "missing-config" && + observation.Value != "dependency-failure") || + observation.Source != "cluster-a" { + t.Fatalf("successful projection escaped closed contract: %#v", observation) + } + }) +} + +func projectedElasticsearchLogCause(t *testing.T, message string, eventAt time.Time) []fleet.GraphFact { + t.Helper() + response, err := json.Marshal(map[string]any{ + "timed_out": false, + "_shards": map[string]any{ + "total": 1, "successful": 1, "skipped": 0, "failed": 0, + }, + "hits": map[string]any{ + "hits": []any{ + map[string]any{ + "fields": map[string]any{ + "@timestamp": []string{eventAt.Format(time.RFC3339Nano)}, + "message": []string{message}, + "orchestrator.cluster.name": []string{"cluster-a"}, + "kubernetes.namespace": []string{"payments"}, + "kubernetes.pod.name": []string{"api-0"}, + "kubernetes.container.name": []string{"api"}, + }, + }, + }, + }, + }) + if err != nil { + t.Fatalf("marshal Elasticsearch response: %v", err) + } + facts, err := connectorelasticsearch.ProjectLogCauses(connectorelasticsearch.Projection{ + Workspace: fleet.LocalWorkspace, + Scope: "cluster-a", + Namespace: "payments", + Pod: "api-0", + Container: "api", + WindowStart: eventAt.Add(-time.Minute), + WindowEnd: eventAt, + ObservedAt: eventAt.Add(time.Minute), + Response: response, + }) + if err != nil { + t.Fatalf("ProjectLogCauses() error = %v", err) + } + return facts +} + +func crashLoopObservation(observedAt time.Time) Observation { + return Observation{ + Ref: fleet.ResourceRef{ + SourceKind: "kubeconfig", Scope: "cluster-a", Kind: "Pod", + Namespace: "payments", Name: "api-0", + }, + Lens: fleet.LensLive, + Key: "pod.failure", + Value: "CrashLoopBackOff", + ObservedAt: observedAt, + Source: "cluster-a", + } +} diff --git a/internal/brain/testdata/replays/README.md b/internal/brain/testdata/replays/README.md index 8ab39f3..e60a557 100644 --- a/internal/brain/testdata/replays/README.md +++ b/internal/brain/testdata/replays/README.md @@ -5,6 +5,11 @@ they must not contain real cluster names, endpoint URLs, credentials, secrets, r customer incident data. The test-only harness rejects unknown JSON fields so every expected verdict shape remains explicit as the rule catalog evolves. +Connector-backed replay observations contain only the connector's reviewed normalized values. In +particular, the Elasticsearch R3 fixture stores the closed `logs.cause` classification and exact +Pod identity, never a raw log message, index or document identifier, query, label, URL, credential, +or user data. + Each fixture is versioned through its required `version` field. A fixture asserts the top verdict's rule, confidence, scope, cited lens/predicate/value evidence, coverage gaps, fleet-wide flag, and advisory shape. The harness evaluates every fixture twice and fails if the serialized result differs. diff --git a/internal/brain/testdata/replays/r3-elasticsearch-log-cause.json b/internal/brain/testdata/replays/r3-elasticsearch-log-cause.json new file mode 100644 index 0000000..f7528f1 --- /dev/null +++ b/internal/brain/testdata/replays/r3-elasticsearch-log-cause.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "name": "r3-elasticsearch-log-cause", + "sanitized": true, + "investigation": { + "workspace": "local", + "coverage": { + "live": {"available": true, "stale": false}, + "telemetry": {"available": true, "stale": false} + }, + "observations": [ + {"ref": {"source_kind": "kubeconfig", "scope": "alpha", "kind": "Pod", "namespace": "synthetic", "name": "payments-0"}, "lens": "live", "key": "pod.failure", "value": "CrashLoopBackOff", "observed_at": "2026-07-18T23:31:00Z", "source": "alpha", "stale": false}, + {"ref": {"source_kind": "elasticsearch", "scope": "alpha", "kind": "Pod", "namespace": "synthetic", "name": "payments-0"}, "lens": "telemetry", "key": "logs.cause", "value": "missing-config", "observed_at": "2026-07-18T23:30:00Z", "source": "alpha", "stale": false} + ] + }, + "expect": { + "top_rule": "R3", + "status": "confirmed", + "scope": "alpha", + "fleet_wide": false, + "cause_of": [], + "citation_evidence": [ + {"lens": "live", "predicate": "pod.failure", "observed": "CrashLoopBackOff"}, + {"lens": "telemetry", "predicate": "logs.cause", "observed": "missing-config"} + ], + "missing_lenses": [], + "advisory": {"command": true, "pr_diff": false, "sensitive": false} + } +} diff --git a/internal/cli/investigate_test.go b/internal/cli/investigate_test.go index 4deec4b..b9a561b 100644 --- a/internal/cli/investigate_test.go +++ b/internal/cli/investigate_test.go @@ -13,6 +13,7 @@ import ( "github.com/spf13/cobra" "github.com/ArdurAI/sith/internal/brain" + connectorelasticsearch "github.com/ArdurAI/sith/internal/connector/elasticsearch" "github.com/ArdurAI/sith/internal/fleet" ) @@ -42,6 +43,93 @@ func TestInvestigateJSONUsesStableVerdictSchema(t *testing.T) { } } +func TestInvestigationSurfaceRendersSanitizedElasticsearchR3Evidence(t *testing.T) { + t.Parallel() + eventAt := time.Date(2026, 7, 18, 23, 30, 0, 0, time.UTC) + rawMarker := "DISCARD_RAW_PASSWORD=https://user:pass@example.invalid" + containerMarker := "discard-container-metadata" + response, err := json.Marshal(map[string]any{ + "timed_out": false, + "_shards": map[string]any{ + "total": 1, "successful": 1, "skipped": 0, "failed": 0, + }, + "hits": map[string]any{ + "hits": []any{map[string]any{ + "fields": map[string]any{ + "@timestamp": []string{eventAt.Format(time.RFC3339Nano)}, + "message": []string{"missing required environment variable " + rawMarker}, + "orchestrator.cluster.name": []string{"alpha"}, + "kubernetes.namespace": []string{"synthetic"}, + "kubernetes.pod.name": []string{"payments-0"}, + "kubernetes.container.name": []string{containerMarker}, + }, + }}, + }, + }) + if err != nil { + t.Fatalf("marshal Elasticsearch response: %v", err) + } + facts, err := connectorelasticsearch.ProjectLogCauses(connectorelasticsearch.Projection{ + Workspace: fleet.LocalWorkspace, Scope: "alpha", Namespace: "synthetic", Pod: "payments-0", + Container: containerMarker, WindowStart: eventAt.Add(-time.Minute), WindowEnd: eventAt, + ObservedAt: eventAt.Add(time.Minute), Response: response, + }) + if err != nil { + t.Fatalf("ProjectLogCauses() error = %v", err) + } + input, err := brain.FromGraphFacts( + fleet.LocalWorkspace, + facts, + map[fleet.Lens]brain.LensCoverage{ + fleet.LensLive: {Available: true}, + fleet.LensTelemetry: {Available: true}, + }, + ) + if err != nil { + t.Fatalf("FromGraphFacts() error = %v", err) + } + input.Observations = append(input.Observations, brain.Observation{ + Ref: fleet.ResourceRef{ + SourceKind: "kubeconfig", Scope: "alpha", Kind: "Pod", + Namespace: "synthetic", Name: "payments-0", + }, + Lens: fleet.LensLive, Key: "pod.failure", Value: "CrashLoopBackOff", + ObservedAt: eventAt.Add(time.Minute), Source: "alpha", + }) + result, err := brain.Evaluate(input) + if err != nil { + t.Fatalf("Evaluate() error = %v", err) + } + if len(result.Verdicts) != 1 || result.Verdicts[0].Rule != brain.RuleCrashLoop || + result.Verdicts[0].Status != brain.StatusConfirmed || result.Verdicts[0].FleetWide { + t.Fatalf("result = %#v, want confirmed entity-local R3", result) + } + + for _, format := range []string{"text", "json"} { + format := format + t.Run(format, func(t *testing.T) { + t.Parallel() + var output bytes.Buffer + command := &cobra.Command{} + command.SetOut(&output) + if err := writeInvestigation(command, format, result); err != nil { + t.Fatalf("writeInvestigation(%s) error = %v", format, err) + } + encoded := output.String() + for _, expected := range []string{"R3", "logs.cause", "missing-config"} { + if !strings.Contains(encoded, expected) { + t.Errorf("output = %q, want %q", encoded, expected) + } + } + for _, discarded := range []string{rawMarker, containerMarker, "example.invalid", `"count"`, `"first_event_at"`, `"last_event_at"`} { + if strings.Contains(encoded, discarded) { + t.Errorf("output retained discarded Elasticsearch data %q: %s", discarded, encoded) + } + } + }) + } +} + func TestInvestigateUnknownContextFailsClosed(t *testing.T) { stdout, stderr, exitCode := runCLIWithReader(t, []string{"investigate", "--context", "missing", "-o", "json"}, &cacheReader{}) if exitCode == 0 || !strings.Contains(stderr, "reached 0/1 contexts") { diff --git a/internal/connector/elasticsearch/boundary_test.go b/internal/connector/elasticsearch/boundary_test.go index 82fc0b3..2315839 100644 --- a/internal/connector/elasticsearch/boundary_test.go +++ b/internal/connector/elasticsearch/boundary_test.go @@ -33,7 +33,7 @@ var allowedProductionImports = map[string]bool{ } var allowedProductionFiles = map[string]string{ - "project.go": "8084eb3558bd7055c4524e3e2fa9c08504bd99662c75b2596bb256b8c2adaa56", + "project.go": "965301c6aff18bb9dd8fd5415f8c7781b3cb3557c21d1564eb268860ad2340c1", } var allowedProductionDeclarations = map[string]bool{ diff --git a/internal/connector/elasticsearch/project.go b/internal/connector/elasticsearch/project.go index 998e499..5a3dc63 100644 --- a/internal/connector/elasticsearch/project.go +++ b/internal/connector/elasticsearch/project.go @@ -520,17 +520,24 @@ func buildFact(input Projection, cause string, aggregate causeAggregate) (fleet. return fleet.GraphFact{}, fmt.Errorf("elasticsearch log-cause fact exceeds %d encoded bytes", maxFactPayloadBytes) } + // Bind the source identity only to fields that survive sanitization so downstream graph + // consumers can independently revalidate the exact workspace, Pod, aggregate, and collection. identity, err := json.Marshal(struct { - Scope string `json:"scope"` - Namespace string `json:"namespace"` - Pod string `json:"pod"` - Container string `json:"container,omitempty"` - Cause string `json:"cause"` - WindowStart time.Time `json:"window_start"` - WindowEnd time.Time `json:"window_end"` + Workspace string `json:"workspace"` + Scope string `json:"scope"` + Namespace string `json:"namespace"` + Pod string `json:"pod"` + Container string `json:"container,omitempty"` + Cause string `json:"cause"` + Count int `json:"count"` + FirstEventAt time.Time `json:"first_event_at"` + LastEventAt time.Time `json:"last_event_at"` + ObservedAt time.Time `json:"observed_at"` }{ - Scope: input.Scope, Namespace: input.Namespace, Pod: input.Pod, Container: input.Container, - Cause: cause, WindowStart: input.WindowStart.UTC(), WindowEnd: input.WindowEnd.UTC(), + Workspace: input.Workspace, Scope: input.Scope, Namespace: input.Namespace, Pod: input.Pod, + Container: input.Container, Cause: cause, Count: aggregate.Count, + FirstEventAt: aggregate.First.UTC(), LastEventAt: aggregate.Last.UTC(), + ObservedAt: input.ObservedAt.UTC(), }) if err != nil { return fleet.GraphFact{}, fmt.Errorf("encode Elasticsearch log-cause identity: %w", err) diff --git a/internal/connector/elasticsearch/project_test.go b/internal/connector/elasticsearch/project_test.go index db61889..2397cb6 100644 --- a/internal/connector/elasticsearch/project_test.go +++ b/internal/connector/elasticsearch/project_test.go @@ -98,6 +98,63 @@ func TestProjectLogCausesIsDeterministicAcrossHitOrder(t *testing.T) { } } +func TestLogCauseNativeIdentityBindsRetainedAggregateAndPod(t *testing.T) { + t.Parallel() + input := testProjection(t) + aggregate := causeAggregate{ + Count: 2, + First: testObservedAt.Add(-8 * time.Minute), + Last: testObservedAt.Add(-2 * time.Minute), + } + base, err := buildFact(input, "panic", aggregate) + if err != nil { + t.Fatalf("buildFact() error = %v", err) + } + workspaceChanged := input + workspaceChanged.Workspace = "workspace-b" + scopeChanged := input + scopeChanged.Scope = "beta" + namespaceChanged := input + namespaceChanged.Namespace = "other" + podChanged := input + podChanged.Pod = "api-other" + containerChanged := input + containerChanged.Container = "worker" + observedAtChanged := input + observedAtChanged.ObservedAt = observedAtChanged.ObservedAt.Add(time.Second) + tests := []struct { + name string + input Projection + cause string + aggregate causeAggregate + }{ + {name: "workspace", input: workspaceChanged, cause: "panic", aggregate: aggregate}, + {name: "scope", input: scopeChanged, cause: "panic", aggregate: aggregate}, + {name: "namespace", input: namespaceChanged, cause: "panic", aggregate: aggregate}, + {name: "Pod", input: podChanged, cause: "panic", aggregate: aggregate}, + {name: "container", input: containerChanged, cause: "panic", aggregate: aggregate}, + {name: "cause", input: input, cause: "missing-config", aggregate: aggregate}, + {name: "count", input: input, cause: "panic", aggregate: causeAggregate{Count: 3, First: aggregate.First, Last: aggregate.Last}}, + {name: "first event", input: input, cause: "panic", aggregate: causeAggregate{Count: aggregate.Count, First: aggregate.First.Add(time.Second), Last: aggregate.Last}}, + {name: "last event", input: input, cause: "panic", aggregate: causeAggregate{Count: aggregate.Count, First: aggregate.First, Last: aggregate.Last.Add(time.Second)}}, + {name: "observed at", input: observedAtChanged, cause: "panic", aggregate: aggregate}, + } + for _, test := range tests { + test := test + t.Run(test.name, func(t *testing.T) { + t.Parallel() + changed, err := buildFact(test.input, test.cause, test.aggregate) + if err != nil { + t.Fatalf("buildFact() error = %v", err) + } + if changed.Fact.Provenance.NativeID == base.Fact.Provenance.NativeID || + changed.Fact.Ref.Name == base.Fact.Ref.Name { + t.Fatalf("identity did not bind %s: base=%#v changed=%#v", test.name, base.Fact, changed.Fact) + } + }) + } +} + func TestProjectLogCausesAbstainsForUnclassifiedAndEmptySuccess(t *testing.T) { t.Parallel() for _, hits := range [][]map[string]any{ diff --git a/sessions/2026-07-18-e14-elasticsearch-r3-bridge.md b/sessions/2026-07-18-e14-elasticsearch-r3-bridge.md new file mode 100644 index 0000000..14af441 --- /dev/null +++ b/sessions/2026-07-18-e14-elasticsearch-r3-bridge.md @@ -0,0 +1,121 @@ +# Session — 2026-07-18 — E14 Elasticsearch R3 graph bridge + +**Builder:** Gnani Rahul · **Branch:** `gnanirahulnutakki/feat/e14-elasticsearch-r3-bridge` +**Slice:** [#280](https://github.com/ArdurAI/sith/issues/280), E14 +[#46](https://github.com/ArdurAI/sith/issues/46) · **Status:** complete local proof; ready for signed commit + +## [G] Goal + +Connect the reviewed, bounded Elasticsearch `search/ecs-v1` log-cause facts to the existing R3 +CrashLoop rule without widening the source contract, retaining raw logs, inferring coverage, +correlating Pods, or adding any read, credential, storage, mutation, or execution capability. + +## [S] Scope + +- Route only exact Elasticsearch `LogSignal` TELEMETRY `FactDerived` values through a fail-closed + graph bridge. +- Revalidate source/provenance, exact Pod attachment, native/resource identity, closed payload, + count, event interval, clock skew, and optional container text. +- Normalize only the exact `logs.cause` value and last classified event time. Preserve source and + staleness; copy caller-declared coverage without inference. +- Exercise the actual projector-to-graph-to-R3 path, deterministic replay, and text/JSON rendering. +- Keep live Elasticsearch querying, TLS/auth, endpoint/index configuration, mappings, pagination, + retention, negative evidence, dependency identity, new rules, alerts, typed intents, mutation, + dispatch, and execution out of scope. + +## [A] Design and implementation + +- `FromGraphFacts` recognizes the exact `elasticsearch` / `search/ecs-v1` protocol before generic + derived-fact filtering, so malformed exact claims cannot bypass validation. +- Accepted facts must be derived TELEMETRY, have exact source and provenance, carry no attributes + or display fields, identify one Pod in the same scope and namespace, and use a lowercase + `sha256:` native ID whose prefix exactly names the `LogSignal` resource. The projector now binds + that digest only to retained sanitized workspace, Pod, aggregate, and collection fields, so the + bridge can recompute it and fail closed on scope, namespace, Pod, or payload retargeting. +- The JSON object is duplicate-safe, exact-case, single-valued, size-bounded, and closed to `key`, + `value`, `count`, `first_event_at`, `last_event_at`, and optional non-null `container`. +- Only `logs.cause` values `panic`, `missing-config`, and `dependency-failure` survive. Count, + container, source payload, and all raw source material are discarded after validation. +- The emitted observation uses the attached Pod identity, the last event time, evidence source, + and fact stale flag. Different Pods remain different evaluator entities; no fleet correlation is + introduced. + +## [T] Focused proof + +- The brain and CLI packages pass after formatting. +- All three cause classes pass through the real Elasticsearch projector and become one exact + Pod-scoped R3 observation and citation. +- Missing, unavailable, stale, omitted, and observation-stale TELEMETRY coverage remain honest; + fact presence never creates coverage. +- Adversarial cases cover workspace/source/scope/namespace/entity mismatch, ambiguous entity + dimensions, wrong kind/lens/protocol, malformed native/resource identity, unexpected metadata, + missing/duplicate/mixed-case/unknown/trailing/malformed JSON, invalid count/time/window, and + invalid optional container values. +- A cross-Pod regression proves Elasticsearch evidence for one Pod cannot strengthen another. +- The deterministic replay corpus now includes a sanitized Elasticsearch R3 case. +- Text and JSON renderer tests start with a raw-message-bearing Search response, pass through the + source projector and graph bridge, and prove the raw marker plus count/container/time metadata + never appears in output. +- Native Go fuzzing completed 653,689 post-hardening executions in fifteen seconds without a panic + or closed contract escape. +- Focused race coverage on the original feature-head local run is green: Elasticsearch 95.7%, + brain 88.6%, and CLI 62.7%. +- CodeRabbit's first complete uncommitted review found one valid defense-in-depth improvement: make + the non-empty source scope requirement explicit at the Elasticsearch bridge even though graph + validation already enforces it. The explicit check and regression are present, focused race tests + remain green, and the repeated full review reports zero findings across all changed files. +- `make ci` passes formatting, vet, lint with zero findings, `govulncheck` with no reachable + vulnerabilities, the complete race suite, policy and alert-rule checks, performance, subprocess + E2E, and production build. Brain coverage is 88.6% and Elasticsearch remains 95.7%. +- `make e2e-isolation` passes PostgreSQL 18.4 forced-RLS tests and both 50,000-execution + cross-workspace fuzzers. +- `make release-check` verifies modules, two byte-identical four-platform release builds, archive + contents, SPDX SBOMs, Homebrew output, and the release-derived amd64/arm64 distroless Hub OCI + layout. +- `make e2e-kind` passes the pinned Kubernetes 1.36.1 two-cluster fan-out, OCI, and Argo suite in + 234.972 seconds. Teardown leaves no Kind cluster or matching Kind/Sith container. +- `README.md` was reviewed in full and updated because this slice adds a public graph-fact behavior + even though the cache-backed CLI still performs no Elasticsearch fetch. A final high-signal + credential scan finds only the deliberately fake `.invalid` raw-message marker used by the CLI + non-retention test; no real credential or private-key material is present. + +## [S] Security, reliability, and cost + +Raw logs are classified and discarded by the existing source projector before the bridge executes. +The bridge accepts only the closed sanitized fact and fails closed on ambiguous provenance, +identity, or payload. It creates one bounded in-memory observation per accepted fact and performs no +I/O. There is no new cloud resource, API request, egress, retention, credential, privilege, write +path, or recurring cost. A future live reader remains responsible for TLS, least-privilege index +`read`, an allowlisted target, finite request/query budgets, and complete-shard responses. + +## [P] Primary sources + +- [Elasticsearch Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) +- [Elasticsearch selected fields](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields) +- [Elastic Common Schema orchestrator fields](https://www.elastic.co/docs/reference/ecs/ecs-orchestrator) +- [Filebeat Kubernetes processor fields](https://www.elastic.co/docs/reference/beats/filebeat/exported-fields-kubernetes-processor) + +## [N] Next + +Create one signed DCO/GSTACK commit, publish and verify the exact PR head, require hosted CI, +CodeQL, and CodeRabbit, merge without rewriting the signed head, prove exact post-merge `dev`, close +issue 280, update #46 without claiming E14 complete, recheck all GitHub security queues, and synchronize +Notion and Obsidian. + +## [C] Checkpoint #1 + +Pending signed implementation commit — issue, design, production bridge, identity hardening, +adversarial tests, replay, CLI privacy proof, documentation, full local gate matrix, repeated +zero-finding review, and clean Kind teardown are frozen in the EXTENDED worktree. + +## [C] Checkpoint #2 + +The explicit hosted review identified two valid fail-closed gaps and one Markdown lint defect. +The bridge now rejects unreviewed provenance metadata and malformed UTF-8 before decoding; dedicated +regressions cover both paths. The session note no longer starts a line with a malformed issue token. +The complete post-remediation local matrix is green: `make ci` with brain coverage at 88.7% and no +lint or vulnerability findings; 296,650 native fuzz executions; PostgreSQL forced-RLS and both +50,000-execution workspace-isolation fuzzers; reproducible release, SPDX SBOM, Homebrew, and +multi-platform distroless OCI proof; and pinned Kubernetes 1.36.1 two-cluster Kind in 246.857 +seconds with clean teardown. Two consecutive complete local CodeRabbit reviews report zero findings +across all 13 changed files. A new signed commit and fresh exact-head hosted proof remain required.