diff --git a/docs/research/USE-CASE-AND-SHAPE.md b/docs/research/USE-CASE-AND-SHAPE.md new file mode 100644 index 0000000..5960500 --- /dev/null +++ b/docs/research/USE-CASE-AND-SHAPE.md @@ -0,0 +1,305 @@ +# Sith — what it should be, and the shape it should take + +**Status:** research synthesis · **Date:** 2026-07-09 + +This is the synthesis document. It answers one question the owner posed: *what should Sith +be so that DevOps / Platform / SRE / MLOps engineers reach for it by default, and what form +should it take?* It pulls together five workstreams of deep research (all cited in the +companion files) and turns them into a decision: an executive answer, a form-factor call, a +ruthlessly prioritized roadmap that places **every** capability the owner named, a +carry/discard/net-new reckoning against the predecessor (`devops-portal`), and concrete edits +to Sith's charter, architecture, roadmap, and epics. + +**Evidence base (companion files, every load-bearing claim carries a primary URL):** +- [market-and-form-factor.md](market-and-form-factor.md) — practitioner pains (global, China, + India), the Lens/form-factor story, the OSS-and-paid tool landscape, cost, and multi-cloud. +- [identity-connections-security.md](identity-connections-security.md) — the four connection + modes, brokered-access prior art, short-lived-credential consensus, custody, and the + supply-chain/audit bar. +- [integrations-and-ai-governance.md](integrations-and-ai-governance.md) — connector-framework + design, per-tool integration surfaces, MCP protocol state, and who governs agents on + clusters today. + +The predecessor reviews referenced below live at +`/Volumes/EXTENDED/checkpoints/devops-portal/review-01..10.md` (read-only, vendor-neutral). + +--- + +## 1. Executive answer + +Sith should be the tool an engineer reaches for the moment they operate **more than one** +Kubernetes cluster — first as a fast local client that shows their whole fleet from the +kubeconfigs already on their laptop, and then, for a team, as a self-hosted control plane that +lets many operators and their AI agents see and act across that fleet without anyone holding +standing admin credentials. It is **one product with two faces**: a single-user local "fleet +IDE" that installs in one command and needs no account, no server, and no telemetry; and a +self-hosted hub that federates the same fleet over outbound-only in-cluster agents, gates every +write through policy, and records who did what and why. The thing Sith *owns* — the position no +one else holds — is **governance of action across a fleet**: typed, signed, approval-gated +cluster operations applied identically to a human and to an AI agent. The thing that gets Sith +*adopted* is the local mode, because the empty slot in today's market is precisely a +no-account, no-telemetry, aggregated multi-cluster client, and every engineer who installs that +is a candidate to turn the hub on later. + +The de-facto wedge is therefore a two-step land-and-expand, not a single feature. **Land** with +the local fleet client: `brew install sith && sith ui` detects every kubeconfig context and +renders one aggregated, searchable view across all of them in under ten minutes — with pod +logs, exec, and port-forward in the core (their removal from the open-source build is what drove +the Lens exodus, [lensapp/lens#6823](https://github.com/lensapp/lens/issues/6823)) and with +cross-cluster search that single-cluster tools structurally cannot do. That slot is open: k9s +shows one context at a time (34k stars, but `:ctx` to switch), Headlamp centers on the +per-cluster view, Lens sits behind an account wall, and the only tool that already aggregates +many clusters into one view — Aptakube — is closed and paid ([aptakube.com](https://aptakube.com/)). +**Expand** to the hub when a team needs shared visibility and real approvals: the same binary +becomes a control plane where clusters graduate from "my kubeconfig" to outbound +[OCM](https://open-cluster-management.io/) minions, writes become typed intents adjudicated by an +external policy decision point (Ardur), and the fleet is exposed as a governed **MCP server** so +Claude Code, Codex, and kagent inherit the same governance for free. Governed action federation +is the durable moat — the research found it empty across every incumbent — and the local client +is the funnel that fills it. + +--- + +## 2. The two wedges (the reframe the plan needs) + +Sith's current charter names one wedge — governed action federation — and explicitly puts a +local console *out of scope* ("A single-cluster console / IDE … owned by Headlamp, k9s, Lens", +[SCOPE.md](../SCOPE.md)). The research says that is half right and half a missed on-ramp. There +are two distinct wedges, and conflating them is what killed the predecessor: + +- **The adoption wedge — the local aggregated fleet client.** How you get ten thousand + individual engineers to install the thing and like it. This is won by form and trust, not by + governance features. The evidence is unambiguous: the top barriers to adopting a new OSS tool + in 2024 were *fear of abandonment (46%)*, *too complex to run (46%)*, and *thin docs (45%)* — + security scanners ranked far lower ([CNCF Annual Survey 2024](https://www.cncf.io/wp-content/uploads/2025/04/cncf_annual_survey24_031225a.pdf)). + The Lens revolt was about an account wall, a trust break, telemetry-by-default, and logs/shell + removed from the OSS build — not a missing feature. So the adoption wedge is won by a + single-binary, ten-minute-wow, no-account, no-telemetry, permissively-licensed client that + answers a fleet-wide question on first run. + +- **The durable wedge — governed action federation with AI as a client.** What makes Sith + defensible and, eventually, what an organization pays to self-host and standardize on. This is + the position the research confirms is *empty*: OCM ships rollout mechanics but no + approvals/typed-verbs/audit; ACM and Rancher have platform-coupled policy; Kargo gates artifact + promotion only; Komodor audits its own AI's remediation inside a closed SaaS; the AI-SRE + incumbents (Komodor, Rancher "Liz", HolmesGPT) stop at advise/diagnose or go autonomy-first with + no approval primitives; the MCP gateways (Kong, Solo agentgateway, MintMCP, Permit.io) enforce + auth and tool-allowlists but have no fleet-aware, blast-radius-conscious, approval-gated action + ([integrations-and-ai-governance.md](integrations-and-ai-governance.md) §4). The KubeCon EU + 2026 read of the room — "nobody has quite figured out how to manage and secure [agents] inside + Kubernetes yet" — is the market saying the door is open. + +The predecessor `devops-portal` had **neither** wedge. It had no adoption on-ramp (its onboarding +was helm-install-a-platform-then-configure-SSO, and it 500'd on a fresh install — review-01, +review-05), and it had no durable moat (it reverse-proxied and iframed tools that were already +better, which review-01 correctly called *negative value*). Sith wins by holding both: lead with +the adoption wedge, monetize/defend on the durable one, and — critically — build them on **one +shared engine** so the local client and the hub are the same fleet model rendered two ways. + +--- + +## 3. Recommended form factor + +**One Go binary, one web frontend, three run modes.** This is the shape the research points to, +and it is exactly what let Headlamp win the Kubernetes-Dashboard succession +([kubernetes.io, 2026-06-01](https://kubernetes.io/blog/2026/06/01/dashboard-to-headlamp/)): + +| Mode | Command | What it is | Who runs it | +|---|---|---|---| +| **CLI** | `sith …` | Scriptable fleet verbs; the substrate power-users script against | Everyone | +| **Local ("fleet IDE")** | `sith ui` | Local web UI on `localhost`, reads existing kubeconfigs, single-user, zero config, **no account, no telemetry** | The individual engineer — the top of the funnel | +| **Hub (federated)** | `sith hub` | The *same* UI served multi-user, clusters joined as outbound OCM minions, workspaces + governance + audit | The platform/SRE team | + +The load-bearing form-factor decisions, each backed by evidence in +[market-and-form-factor.md § Part 2](market-and-form-factor.md#part-2): + +- **Single artifact, no server or account to install.** k9s (`brew install`, one binary) and + Freelens define the funnel; "helm-install a platform, then SSO, then see value" is the adoption + cliff the predecessor fell off. The wow is `brew install sith && sith ui` → every context + detected → one aggregated fleet view with cmd-K search, in minutes, offline-capable, nothing + leaving the machine. +- **Cache-first render, never spinner-first.** The single most important DX decision. Kubernetes + hands you the perfect substrate — watch streams into a local informer cache — so every view + renders from a local store in tens of milliseconds and reconciles deltas in the background (the + Linear local-first mechanic; the 0.1s/1s/10s + [response-time limits](https://www.nngroup.com/articles/response-times-3-important-limits/)). +- **Center of gravity is the fleet, not the pod.** The local mode is *not* "another single-cluster + console" — that slot is taken and is correctly out of scope. It is the aggregated, + cross-cluster view Headlamp/k9s/Lens don't center on: all-clusters resource views, fleet search + and correlation ("which clusters run image X", "where is `payments` degraded"), staleness + stamps, and the same typed-verb actions with dry-run/diff (self-approved locally, but the same + intent model that later gains real governance). Per-pod table stakes (logs, exec, port-forward, + YAML edit) exist because their absence drove the Lens exodus — but they are commodity K8s API + calls, not a place to out-feature Headlamp. +- **Go binary with an embedded frontend first; Tauri desktop shell as a fast-follow.** Ship the + web UI embedded in the single binary (k9s-grade install friction, no code-signing tax on day + one). Wrap it later in **Tauri, not Electron** — the memory/footprint evidence and Lens-refugee + sensitivity to Electron bloat both point that way, and Aptakube already proves Tauri in this + exact product class. The wrapper is additive; the web UI has to exist in both modes anyway. + +--- + +## 4. The ruthlessly prioritized roadmap + +Feature sprawl is what killed `devops-portal` (12 providers, 92 API routes, ~7 of 10 pillars pure +pass-through — review-01). The discipline here is: **every capability the owner named is placed +in exactly one of four buckets, and the "not now" bucket is as important as the wedge.** A +capability earns a higher bucket only if it is part of a wedge, not merely "useful". + +### 4.1 Wedge — build first; this is the reason to exist plus the on-ramp + +| Capability | Why it is wedge, not later | +|---|---| +| **Local aggregated "fleet IDE" mode** (`sith ui`, kubeconfig auto-detect, cache-first render, cmd-K fleet search, logs/exec/port-forward/YAML in core, no account, no telemetry) | The adoption wedge. Occupies the one empty OSS slot (§2). Every install is a hub candidate. | +| **Read federation + normalized fleet model + cross-cluster correlation** | The shared engine behind *both* faces. Correlation is the thing single-cluster tools structurally cannot do; it is the first-run wow and the hub's core. Build the read source **abstract** from day one: local kubeconfig contexts *or* OCM-brokered spokes feed the same model. | +| **Minions (outbound OCM agents) + multi-auth** (kubeconfig for local; API keys / JWT / OIDC and short-lived cloud IAM for the hub) | The connection substrate. Minions are how a "my kubeconfig" cluster graduates to a governed one without the operator relearning anything. Multi-auth is table stakes because a cloud kubeconfig alone is not even a working credential ([identity §1](identity-connections-security.md)). | +| **Governance spine, day one:** Workspace tenancy, signed-token authn (never header trust), least-privilege RBAC, forced Postgres RLS backstop, audit-log (*what-happened*) + decision-ledger (*why-allowed*) | These are the predecessor's exact failure points (dead RLS, header-trust IDOR, single god-key). They are day-one requirements, not later hardening — a control plane that leaks across tenants has no reason to exist. | +| **First governed typed write — `gitops.open-pr` through the Ardur PDP** | The safest possible proof of the action-federation path: a proposal a human merges, zero cluster mutation, zero new standing trust. It lights up the whole PEP pipeline and the two ledgers. | +| **No-god-key custody:** no central admin kubeconfig; KMS-envelope per-tenant DEKs in the hub; OS keychain in local mode; **cosign-signed releases + SLSA L2 provenance + SBOM from the first tag** | Custody is a wedge property here because the predecessor's single `TOKEN_ENCRYPTION_KEY` (one leak decrypts every tenant) is a disqualifying foundation, and signing is now cheap — SLSA L2 via cosign + GitHub attestation is "an afternoon". Evaluators run these scorecards before a human reads the code ([identity §6](identity-connections-security.md)). | + +### 4.2 Fast-follow — right after the wedge proves out + +| Capability | Placement rationale | +|---|---| +| **Governed MCP server** (annotated read tools + elicitation-gated write tools on the *same* PEP) | The amplifier, not a separate bet. Because Sith enforces server-side, every external agent (Claude Code, Codex, kagent) inherits the governance for free. MCP is now vendor-neutral LF infrastructure ([donated 2025-12-09](https://blog.modelcontextprotocol.io/posts/2025-12-09-mcp-joins-agentic-ai-foundation/)); the protocol is settled, the governance on top is not. This *is* "Claude/Codex compatibility". | +| **Policy federation** — waves/canary, environment gates, multi-approver for prod, partial-failure/auto-rollback, abstention; the live-mutation verbs (`argocd.sync`, `rollout.promote\|abort`, `deployment.scale\|restart`) behind all of it | The genuinely novel, genuinely empty part of the durable wedge. Fast-follow because it rides the PEP the wedge already builds and needs the `gitops.open-pr` path proven first. | +| **Connector framework** — out-of-process gRPC, SDK-first, **three fixed kinds** (read adapter / brokered read-through / typed-action adapter), one canonical connector per tool | Generalizes the six hand-written day-1 adapters (**Argo CD, Flux, Helm, Prometheus, Loki, GitHub** — the ones that feed the model cheaply or host the first write). Fast-follow, not wedge: build adapters by hand first, generalize once the shape is proven — the opposite of Backstage's premature-ecosystem trap ([integrations §1](integrations-and-ai-governance.md)). | +| **Cost analyzer as a read-overlay** — deploy/read OpenCost per cluster, aggregate at the hub into per-workspace/per-team fleet rollups, GPU columns where DCGM exists | A read integration, *not a build*. The fleet rollup is the exact documented gap: OpenCost is per-cluster by design and its multi-cluster ask was triaged P3 and closed unresolved; Kubecost's unified multi-cluster view is Enterprise-tier ([market § Part 3 §5](market-and-form-factor.md#part-3)). Building a metering engine would re-fight OpenCost/Kubecost/CAST AI on their ground. | +| **Multi-cloud enumeration + short-lived token minting** (thin per-cloud adapters: EKS `get-token`, AKS Entra+kubelogin, GKE plugin, plus **ACK/CCE/TKE** for China) | The Kubernetes API is uniform across US and China clouds (all conformance-certified), so cluster-*inside* views work day one with no cloud code. Only enumeration and credential-minting differ per cloud — a thin adapter, minting short-lived tokens, storing no long-lived cloud keys. | +| **Air-gap / multi-arch / registry-relocatable packaging** (Zarf-style single bundle, no phone-home, `linux/amd64`+`arm64` images from the first build, cosign-signed) | Mandatory for China and regulated estates, and cheap only if designed in early. Multi-arch images are day-one; the offline bundle is fast-follow. | +| **Tauri desktop shell** wrapping the same binary | Dock presence and deep-OS integration for the local mode. Additive. | + +### 4.3 Later — real, but not until the above lands + +| Capability | Why later | +|---|---| +| **Long-tail read connectors** — Datadog, Splunk, Elastic/OpenSearch/Kibana, Terraform/OpenTofu state-and-drift | Demand-ranked, not wedge. Each is a read adapter through the framework; none is on the critical path. Cost/observability pain pushes teams *toward* self-hosting, which is a distribution tailwind, not a reason to embed these. | +| **ITSM typed actions** — Jira / Zendesk / ServiceNow change-ticket linkage for a fleet action | Real demand (change-management linkage), but it is a convenience verb, not the wedge. | +| **Sith as an MCP *client*** (calling kagent, Grafana MCP, GitHub MCP) | A convenience that consumes other servers; the value is Sith *being* the governed server, not calling others. | +| **Governing LangChain / LangGraph agents** | Resolves to: those agents connect to Sith's MCP server as clients, get a scoped identity with a ceiling below the human's, and every cluster action they attempt goes through the same PEP + decision-ledger. Sith **governs** agents; it does not **orchestrate** them (LangGraph/kagent do that). Downstream of the MCP server and policy federation, hence later. | +| **OpenShift-specific views** (Routes / SCC-aware) | Conformant-API coverage is guaranteed day one; deep OpenShift-isms are later, and Sith never competes for ACM-committed estates. | + +### 4.4 Explicitly not now — say no, loudly (this list is the anti-sprawl contract) + +- **Re-skinning or proxying tool UIs** (iframing Grafana, thin ArgoCD/GitHub re-implementations). + This was the predecessor's core negative-value pattern (review-01). Brokered read-through means + *deep-link to the tool's own UI*; never re-build it. +- **A telemetry lake / metrics or log store.** Sith reads health through Prometheus/Loki; it never + stores series. Drifting here re-fights Grafana/Datadog and violates the non-goal. +- **A metering / billing / cost-optimization engine.** The cost overlay reads OpenCost; it does + not meter or auto-mutate clusters (that is CAST AI's ground). +- **An agent-orchestration framework.** LangGraph and kagent orchestrate; Sith governs. Building an + orchestrator is scope Sith cannot win and does not need. +- **A developer portal / IDP / service catalog / scorecards / DORA.** The exact thesis that lost + across 2020–2026 and that the predecessor died on (review-01, review-10). Portals can *consume* + Sith's API/MCP; Sith is not a portal. +- **A GitOps controller / reconciler** (Sith opens PRs) and **a multi-cluster scheduler** (Karmada, + OCM Placement). +- **Fluentd / Fluent Bit as data sources** — they ship logs *to* Loki/Elastic; read those sinks + instead. **Kustomize / Helm as action targets** in v1 — read adapters for inventory only. +- **`exec` / free-form `apply` / Secret mutation / RBAC mutation** — not "not yet", but permanently + inexpressible in the action model. This does not move buckets, ever. +- **Running SPIRE / forcing a workload-identity platform on users.** SPIFFE is the right *identity + model* (support SPIFFE IDs and mTLS), but SPIRE is operationally heavy — a dedicated engineer and + a 6–24-month rollout ([identity §2](identity-connections-security.md)). Support the identity type; + do not make users run the platform. + +--- + +## 5. Carry / discard / net-new vs `devops-portal` + +The predecessor is a lessons-learned artifact, not a starting codebase. The reviews are blunt: +"a technically impressive solution in search of a problem" with "no identified user … and no +wedge" (review-01), whose one salvageable idea is "a governed AI-mutation layer … exposed as an +MCP-governance component" (review-10). Concretely: + +| Carry (the good bones) | Discard (the failure modes) | Net-new (what the research says Sith needs and the predecessor never had) | +|---|---|---| +| **Action/exec broker service-layer** — the clean 1:1 tool→service mapping (review-08) — **redesigned as the PEP** with a closed verb vocabulary and no shell | **Shared central admin kubeconfig / inbound-god-kubeconfig** model — replaced by outbound OCM minions + scoped MSA tokens + no central admin credential | **Local aggregated fleet client** as the adoption wedge (empty OSS slot) | +| **Per-org encrypted credential vault** (AES-256-GCM key-ring, review-08) — **re-architected** as KMS-envelope per-tenant DEKs | **Single god key** (`TOKEN_ENCRYPTION_KEY` decrypts every tenant) | **Cross-cluster correlation** as a first-class query (single-cluster tools can't) | +| **RBAC + audit spine** (review-08) — kept and **hardened** with signed-token authz and a separate decision-ledger | **Dead/inert RLS + `x-user-role` header-trust IDOR** — replaced by FORCE RLS (non-owner role) + signed-token-only authz | **Typed-intent action model** with signed dispatch + per-minion local allowlist re-validation (two independent blast-radius bounds) | +| **Governed AI / MCP ambition** — review-10's "salvage at most one idea": this becomes the **core**, done right (real MCP server, elicitation gates, AI-as-client) | **All-heavy monolith** (~48k LOC, 92 routes, 24 pages; iframes/reverse-proxies better tools) — replaced by one narrow Go binary + embedded UI, no re-skinning | **Policy federation** — waves / multi-approver / abstention (genuinely novel, empty) | +| **Workspace tenancy** (the Tenant→Workspace rename) — kept as the isolation anchor with a **real** RLS backstop | **Feature sprawl** (12 providers, ~7/10 pillars pass-through) — replaced by the closed vocabulary + three fixed connector kinds + a hard scope gate | **Ardur** as external PDP + identity broker + decision-ledger (*why-allowed* split from *what-happened*) | +| | **Broken onboarding** (platform-install cliff; 500s on fresh install) — replaced by `brew install && sith ui`, ten-minute wow | **Air-gap / multi-arch / registry-relocatable** distribution for China/regulated | +| | **Features that never ran** (dead write path, `allowDestructive` never set, mock MCP page, qwen2.5:3b as the only tool-capable model) — replaced by falsification-first, ship-what's-verified discipline | **Cost read-overlay** with fleet rollup + GPU columns (empty in OSS, paywalled commercially) | + +--- + +## 6. Concrete changes to the plan + +These are the edits the research implies. They are surgical: the governance thesis is *validated* +by the evidence (annotations are hints, enforcement must be server-side, no incumbent occupies +governed fleet action) — the change is to add the adoption wedge and hang the named capabilities +off it without diluting the anti-drift contract. + +### CHARTER.md +- **§3 Target user** — add the **individual operator** as the top-of-funnel entry (the local-mode + user), distinct from the platform/SRE team who runs the hub. The primary buyer is unchanged; the + first user is new. +- **§4 The wedge** — reframe from one wedge to **two**: the *adoption wedge* (the local aggregated + fleet client) and the *durable wedge* (governed action federation). State explicitly that they + share one fleet-model engine and that the local mode is the funnel for the governed hub, not a + second product. +- **§6 Success criteria** — add an adoption-side criterion for the local mode (e.g. "a new user + goes from `brew install` to a populated cross-cluster answer in under ten minutes, offline, with + nothing leaving the machine"), sitting alongside the existing P1–P3 governance criteria. + +### SCOPE.md +- **The single most important scope edit.** Refine the out-of-scope row "A single-cluster console / + IDE" to "**Another single-cluster** console" and add an **in-scope** line: "**an aggregated + multi-cluster local fleet client** (the adoption on-ramp), distinguished by fleet + aggregation/correlation and staleness — not per-pod parity with Headlamp/k9s." This resolves the + apparent contradiction with the owner's "Lens IDE better than Lens" ask: Sith does not build + another per-cluster console; it builds the *fleet* view none of them center on. +- Add to the non-goals table (make the §4.4 "not now" list contractual): tool-UI re-skinning / + proxying, a telemetry lake, a metering/optimization engine, an agent-orchestration framework, and + Fluentd/Fluent-bit as data sources. + +### ARCHITECTURE.md +- Add a **run-modes** section up front: one Go binary + one embedded web frontend, three modes + (`sith` / `sith ui` / `sith hub`); cache-first local render via the K8s informer/watch cache; + Tauri shell as a fast-follow. +- Make the **read-federation service source-abstract**: the fleet model is populated from **local + kubeconfig contexts (direct)** *or* **OCM-brokered spokes (federated)** — same model, two sources. + This is what makes local and hub one engine. +- Add the **four-mode connection/identity model** from [identity §3](identity-connections-security.md) + (local / minion / cloud-IAM / API-key-JWT-OIDC) and the **connector framework** (three fixed + kinds, out-of-process gRPC, SDK-first) as named components. +- Add the **cost read-overlay** as a fleet-fact kind, and note **cosign + SLSA L2 + SBOM** as + release requirements and **SPIFFE IDs/mTLS supported without requiring SPIRE**. + +### ROADMAP.md +- Insert an early **Local-mode track** that ships the local fleet client on the **same P1 + read-federation engine**, in parallel with the hub's P1 — the adoption artifact ships early + without breaking falsification-first, because the local client *is* read federation against local + kubeconfigs. +- Explicitly require **P1 read federation to be source-agnostic** (local *or* OCM) from the start. +- Keep M0 (passed), P2 (`gitops.open-pr`), and P3 (policy federation + MCP) as they are; attach the + **cost overlay**, **connector-framework generalization**, and **multi-cloud/air-gap packaging** as + P3-adjacent fast-follows, with **multi-arch images required from the first release**. + +### EPICS.md +- **Amend E2 (Read federation)** to treat **local kubeconfig contexts as a first-class read source** + alongside OCM spokes. +- **New E11 — Local fleet client (the adoption on-ramp):** single Go binary, `sith ui`, kubeconfig + auto-detect, cache-first render, cmd-K fleet search, logs/exec/port-forward/YAML in core, + aggregated cross-cluster views, no account/telemetry; Tauri shell fast-follow. Reuses E2's model. +- **New E12 — Connector framework (fast-follow):** out-of-process gRPC SDK, three fixed kinds, one + canonical connector per tool; generalizes the six day-1 adapters. +- **New E13 — Cost read-overlay (fast-follow):** OpenCost per-cluster read + hub rollup + GPU + columns; explicitly *not* a metering engine. +- **Amend E9 (Deployment & packaging)** to require multi-arch (`amd64`+`arm64`) images day one, + registry-relocatable, a Zarf-style air-gap bundle with no phone-home, and cosign-signed releases + with SLSA L2 provenance + SBOM. +- **Amend E7 (MCP server)** to note that the MCP *client* role and agent-management are **later**, + downstream of the server; and **E8 (Console)** to note the console is the one web frontend served + by both `sith ui` and `sith hub`. + +--- + +## 7. The one-line test for every future feature request + +Before anything is added: *does it serve the adoption wedge (get an individual to install and +love the local fleet client) or the durable wedge (governed action across the fleet, humans and +agents alike)?* If neither, it belongs in §4.4 — no matter how useful it sounds. That question, +applied ruthlessly, is the difference between Sith and the predecessor. diff --git a/docs/research/identity-connections-security.md b/docs/research/identity-connections-security.md new file mode 100644 index 0000000..a1a8db7 --- /dev/null +++ b/docs/research/identity-connections-security.md @@ -0,0 +1,84 @@ +# Connection/identity modes and security architecture (workstreams D + J) + +**Date:** 2026-07-09 · **Method:** deep-research fan-out; claims below carry a primary URL. Items marked **[3-0]** passed adversarial 3-vote verification; items marked **[fetched]** were pulled from the primary source but the verification vote was rate-limited (the source and quote are real; treat the framing as single-reviewer). Nothing here is asserted without a URL. + +Answers: how Sith should connect and broker identity across its four connection modes, and what the security bar actually is for a tool that touches many clusters and holds many secret types. + +--- + +## 1. Kubeconfig reality — why "upload your kubeconfig" is the wrong primitive + +The predecessor's central-kubeconfig honeypot wasn't just risky, it was **technically broken for modern clusters**: + +- **[3-0]** kubectl/client-go authenticate to clusters via **exec credential plugins** — the kubeconfig instructs the client to run an external command locally to obtain credentials; this is the mechanism behind `aws eks get-token`, `kubelogin`, and `gke-gcloud-auth-plugin` ([k8s auth docs](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)). A cloud kubeconfig **contains no usable credential by itself** — it points at a helper binary and the user's local cloud session. +- **[3-0]** kubectl **removed built-in AKS/GKE auth**; "Earlier versions of kubectl included built-in support for authenticating to AKS and GKE, but this is no longer present" ([same](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)). From v1.26, GCP provider auth was removed from OSS kubectl; GKE now **requires** the external `gke-gcloud-auth-plugin` ([GKE auth changes](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke)). +- **Consequence:** uploading a cloud kubeconfig to a server-side tool **does not transfer working auth** — the server lacks the plugin and the user's cloud session. So the honeypot design is both dangerous *and* non-functional for EKS/AKS/GKE. This is decisive evidence for **local mode keeping kubeconfigs on the user's machine** and **federated mode using in-cluster agents**, not a central credential store. +- **[3-0]** Kubernetes (through v1.36) has **no X.509 client-cert revocation** — an issued cert is valid until expiry, so a leaked admin kubeconfig with an embedded cert is an **irrevocable standing credential** ([k8s auth docs](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)). +- **[3-0]** OIDC id_tokens as bearer tokens "can't be revoked… so [they] should be short-lived (only a few minutes)" — upstream endorsement of the short-lived model ([same](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)). +- **[fetched]** The leak evidence is real: Aqua Nautilus's 2023 GitHub scan found 438 public records with base64 K8s registry secrets, **~46% still valid** ([Aqua](https://www.aquasec.com/blog/the-ticking-supply-chain-attack-bomb-of-exposed-kubernetes-secrets/)); exposed GCP/AWS tokens in the same repos were **already expired** — short-lived creds converted leaks into non-events. Microsoft's 38TB exposure came from a single over-privileged SAS token valid **~3 years** before remediation ([Wiz](https://www.wiz.io/blog/38-terabytes-of-private-data-accidentally-exposed-by-microsoft-ai-researchers)). + +## 2. The outbound-agent pattern is the industry consensus (Sith's minion mode is not exotic) + +Every serious multi-cluster broker uses an in-cluster agent that dials **out**; none require inbound reach: + +| System | Directionality (primary source) | +|---|---| +| **OCM** klusterlet + cluster-proxy | Outbound-only; hub needs no inbound access — reproduced hands-on in [M0](../experiments/M0-ocm-falsification.md) | +| **Teleport** | **[fetched]** Agents connect via an **outbound reverse tunnel** to the Proxy; sit behind NAT/firewall with no inbound. K8s Service runs **as a pod using its own service-account** — no kubeconfig exported ([Teleport agents](https://goteleport.com/docs/reference/architecture/agents/)) | +| **Azure Arc** | **[fetched]** "No inbound ports… agents communicate with Azure exclusively via outbound connections"; `clusterconnect-agent` brokers apiserver reach over an agent-initiated tunnel ([Arc agent overview](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-agent-overview)) | +| **GKE Connect** | **[fetched]** Connect Agent "initiates an outbound connection to Google… works through NATs, egress proxies, and firewalls" ([Connect Agent](https://cloud.google.com/kubernetes-engine/fleet-management/docs/connect-agent)) | +| **Rancher** | **[fetched]** `cattle-cluster-agent` "opens a tunnel out to a cluster controller inside the Rancher server" ([Rancher arch](https://ranchermanager.docs.rancher.com/reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters)) | +| **HashiCorp Boundary** | **[fetched]** Multi-hop: the egress worker "initiates outbound connections… for networks that forbid inbound traffic" ([Boundary multi-hop](https://developer.hashicorp.com/boundary/docs/workers/multi-hop)) | + +Two design corollaries the sources hand us. (a) **"Outbound-only agent" is table stakes, not differentiation** — every incumbent has it (and Karmada has pull mode too). Sith's differentiation must be the governance *above* the transport, not the transport. (b) Both Rancher and Arc broker access through a **central auth proxy/chokepoint** — validating "one governed entry point in front of every cluster API call", which is exactly Sith's PEP. + +### Short-lived credentials are settled consensus + +- **[fetched]** SPIFFE/SPIRE **graduated from CNCF (2022-09-20)** with named production adopters (Bloomberg, ByteDance, Netflix, Pinterest, Uber…); design goal is "removing the need for shared secrets" via attested, auto-rotated X.509/JWT SVIDs ([CNCF SPIFFE graduation](https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/)). +- **[fetched]** **Pinniped** is direct prior art for "one upstream identity, many clusters without distributing static per-cluster creds": its Supervisor federates identity, its Concierge **exchanges a token for a short-lived mTLS client cert** per target cluster; on managed clouds (no signing-key access) it falls back to an **in-cluster impersonation proxy** ([Pinniped arch](https://pinniped.dev/docs/background/architecture/)) — evidence a multi-cluster tool **cannot rely on one auth mechanism across cluster types**. +- Vault Transit and cloud STS AssumeRole round out the brokered-credential toolbox ([Vault Transit](https://developer.hashicorp.com/vault/docs/secrets/transit)). + +### The market lesson: Infra (infrahq) + +- **[fetched]** Infra (YC W21) built exactly this category — a broker for K8s access via connectors + access keys + IdP integration — and is **dormant**: last release v0.21.0 (2023-01-25), no release in ~3.5 years; founders left to build Ollama ([Infra releases](https://github.com/infrahq/infra/releases)). **Lesson:** a standalone "cluster access broker" is a thin, hard-to-monetize wedge on its own. Sith must not position as "another access broker" — access brokering is a *property* of the governed action layer, not the product. (Corroborated by the Lens/Kubecost/Rancher consolidations: point tools in this space get absorbed or stall.) +- `kube-oidc-proxy` (a commonly-cited OIDC-for-managed-clusters tool) was **archived by jetstack 2024-05-17** ([repo](https://github.com/jetstack/kube-oidc-proxy)) — another reason not to depend on a single OSS auth shim. + +## 3. Recommended multi-mode connection + identity model + +| Mode | How it connects | Identity/custody | Security property | +|---|---|---|---| +| **Local (direct)** | Reads the user's existing kubeconfig contexts on the machine; exec plugins run locally exactly as kubectl does | **Kubeconfigs never leave the machine**; any Sith-held secret (rare in local mode) goes in the **OS keychain** | Same trust boundary as kubectl. No server, no upload, no honeypot. Matches how Docker/gh CLI store local creds (§4) | +| **Federated (minion)** | OCM klusterlet + cluster-proxy, outbound-only; hub reaches cluster-local services via the reverse tunnel | **Scoped `managed-serviceaccount` projected token**; hub holds **no admin kubeconfig** (M0-proven); action execution uses an Ardur-brokered short-lived identity re-validated at the spoke | Reach decoupled from privilege; ceiling below the human's; per-spoke local allowlist | +| **Cloud IAM** | Thin per-cloud adapter enumerates clusters and **mints short-lived tokens** (EKS get-token / AKS Entra+kubelogin / GKE plugin); never stores long-lived cloud keys | STS/AssumeRole-style short-lived creds; nothing standing at rest | Leak = non-event (creds expire); matches the Aqua/Wiz evidence | +| **API key / JWT / OIDC** | For tool integrations (Argo/Grafana/etc.) and machine callers of Sith itself | KMS-envelope, per-tenant DEKs in federated mode; keychain in local mode | Bounded blast radius per tenant; see §5 | + +**One rule across all four:** Sith authenticates *from signed token claims, never spoofable headers* (the predecessor's IDOR), and the credential ceiling for any agent/AI actor is strictly below the human's. + +## 4. Local-mode credential custody — the keychain decision + +- **[fetched]** Docker ships official credential helpers (`osxkeychain`, `wincred`, `secretservice`, `pass`) precisely because the prior default — base64 in `~/.docker/config.json` — is trivially decodable plaintext ([Docker credential helpers](https://github.com/docker/docker-credential-helpers)). This is the established desktop-tool norm. +- **[fetched]** GitHub CLI moved to system-keyring by default (Feb 2023) but has a **silent plaintext fallback** when no keyring is available — closed by warning rather than fail-closed ([gh#7570 / PR #7781](https://github.com/cli/cli/issues/7570)). **Design lesson for Sith:** keychain-first, but the fallback must **fail loudly or encrypt-at-rest**, never silently write plaintext. + +## 5. Server-side custody + tenancy (federated mode) + +- **Envelope encryption, per-tenant DEKs wrapped by a KMS KEK** — one KMS key per tenant, encryption-context as a cryptographic tenant boundary ([AWS SaaS envelope pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-managed-policies/data-encryption.html)); Vault Transit as the non-cloud-KMS alternative (Vault holds keys, never the data — [Transit docs](https://developer.hashicorp.com/vault/docs/secrets/transit)). This is the exact fix for the predecessor's single env master key. +- **Postgres RLS** is the validated pooled-tenancy backstop, with the well-documented caveat that it only isolates if the app connects as a **non-owner, non-BYPASSRLS** role and `FORCE ROW LEVEL SECURITY` is set ([AWS SaaS Factory RLS](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/), production report: [Nile](https://www.thenile.dev/blog/multi-tenant-rls)). Operational hazards to engineer around: session-variable RLS vs PgBouncer pooling, and thread-local context leaking across reused connections (real reported bug). Sith's plan already has RLS from day one — the sources confirm the exact pitfalls to avoid. + +## 6. Supply-chain + audit checklist, ranked by what evaluators actually check + +Evaluators of an OSS infra tool run automated scorecards before humans read code. Ranked by what those tools weight: + +1. **Signed releases (sigstore/cosign) + CI hygiene** — OpenSSF Scorecard rates release signing **High** risk and dangerous GitHub Actions workflows **Critical**; CNCF "highly recommends" Scorecard and wires it into CLOMonitor ([OpenSSF Scorecard](https://github.com/ossf/scorecard), [CNCF security guidance](https://contribute.cncf.io/maintainers/community/compliance/)). Notably, **Scorecard has no SBOM check** — signing and CI hygiene outrank SBOM in practice. +2. **SLSA build provenance** — a maturity ladder projects should climb (L1 provenance → L3 tamper-resistant); "should", not a hard CNCF mandate ([slsa.dev](https://slsa.dev)). +3. **SBOM (SPDX/CycloneDX), ideally signed** — expected of producers; consumers ask for it ([CNCF supply-chain paper](https://github.com/cncf/tag-security/blob/main/community/resources/software-supply-chain-security/secure-supply-chain-guidance/)). +4. **Append-only, tamper-evident audit log** — SOC 2 auditors expect WORM or hash-chained logs; they judge the **evidence artifacts you can produce**, not a config toggle; each entry needs actor/action/timestamp/resource; maps to CC6.1/CC6.3/CC7.2 ([Bytebase SOC2 report](https://www.bytebase.com/blog/soc2-audit-logging/)). This validates Sith's separate **audit-log (what-happened) + decision-ledger (why-allowed)**. +5. **Third-party security audit** (Cure53-style) — expected for security-adjacent tooling (SPIFFE required it for graduation). + +### Signed-action precedent and EU AI Act + +- **Signed intents have direct precedent:** Teleport signs short-lived certs encoding client identity + routing; in-toto attestations are the CNCF-endorsed "signed records of actions" pattern for a verifiable action ledger, with Sigstore keyless signing tying signatures to identity not long-lived keys ([CNCF supply-chain paper](https://github.com/cncf/tag-security/blob/main/community/resources/software-supply-chain-security/secure-supply-chain-guidance/)). Sith's signed typed intents sit squarely in this lineage. +- **EU AI Act Article 12** record-keeping is **scoped to high-risk AI systems only** ([Article 12 text](https://artificialintelligenceact.eu/article/12/)). A K8s ops tool with AI features is bound **only if classified high-risk**. So it's not a blanket obligation — but the granularity regulators expect (per-action, timestamped, human-attributed, tamper-evident) is exactly the ledger Sith already plans, which is why "build the ledger regardless" is the right call: it satisfies SOC 2 now and AI-Act-high-risk later at no extra design cost. + +## 7. China/India compliance implications for architecture + +The China/India constraints in [market-and-form-factor.md § Part 1](market-and-form-factor.md#part-1) map onto this security design cleanly: MLPS 2.0 L3 wants **admin/auditor role separation** (Sith's reader/operator/approver/admin + separate audit vs decision ledgers), **tamper-resistant backed-up audit records** (append-only/hash-chained log, item 4 above), and **no root SSH / agent-mediated access** (the outbound-agent + brokered-identity model *is* that). DPDP/RBI want in-country self-hosting — satisfied by the self-hosted hub + no phone-home. None of this requires China-specific code; it requires designing the audit/role/custody spine correctly once. diff --git a/docs/research/integrations-and-ai-governance.md b/docs/research/integrations-and-ai-governance.md new file mode 100644 index 0000000..4363030 --- /dev/null +++ b/docs/research/integrations-and-ai-governance.md @@ -0,0 +1,82 @@ +# Integration framework and AI-agent governance (workstreams H + I) + +**Date:** 2026-07-09 · **Method:** deep-research fan-out; **[3-0]** = adversarially verified, **[fetched]** = pulled from the primary source, vote rate-limited. Every load-bearing claim carries a URL. + +Answers: how to design the connector framework, the per-tool integration mechanism and priority, and whether "governed MCP server for fleet actions" is a real, empty position. + +--- + +## 1. Connector-framework design principles (what scaled, what drowned) + +**Grafana's model scaled — copy it.** +- **[3-0]** Grafana runs backend plugins **out-of-process as subprocesses over gRPC** (HashiCorp go-plugin), so "a panic in a plugin doesn't panic the server" ([Grafana backend plugins](https://grafana.com/developers/plugin-tools/key-concepts/backend-plugins/)). +- **[3-0]** The plugin contract is a **small fixed set of typed capabilities** (query, resources, health, metrics, stream) — narrow schema, not arbitrary host access. +- **[3-0]** It's **SDK-first**: authors code against `grafana-plugin-sdk-go`, which hides the wire protocol — the mechanism that let Grafana evolve the protocol without breaking the ecosystem ([grafana#19667](https://github.com/grafana/grafana/issues/19667)). + +**Terraform confirms the versioning discipline.** +- **[3-0]** The provider protocol is a **versioned, typed gRPC/protobuf interface**; **major versions delineate compatibility, minor versions are strictly additive (non-breaking)** ([Terraform plugin protocol](https://developer.hashicorp.com/terraform/plugin/terraform-plugin-protocol)). + +**Backstage is the cautionary tale — in-process, unversioned, drowned in maintenance.** +- **[fetched]** A BackstageCon EU 2026 maintainer panel (Red Hat/DoorDash/OP Financial/Vodafone Ziggo): the marketplace has **250+ plugins, many unmaintained**; **breaking changes ship within minor releases despite semver** (1.48→1.49), stranding adopters several releases behind; the React Router 6→7 migration was especially costly; adopters must write **custom React plugins**, taking on frontend dev on top of their real job ([panel writeup](https://tldrecap.tech/posts/2026/backstagecon-europe/backstage-plugin-ecosystem-sustainability/)). Proposed fixes: **bind backend plugins to OpenAPI schemas** for stronger break detection; **one canonical plugin per target** (not many overlapping); **quality-tier/ownership signaling**. + +**Transferable principles for Sith's connector framework:** +1. **Out-of-process, typed, versioned.** gRPC/protobuf connectors with a stable minor-additive contract (Grafana + Terraform). A crashing connector must not take the hub down. +2. **SDK-first, protocol hidden.** Authors implement a narrow interface; Sith owns the wire format and can evolve it. +3. **Three fixed connector kinds, not open-ended:** **read adapter** (pull normalized facts into the fleet model), **brokered read-through** (proxy to a tool's own UI/API via cluster-proxy — no re-skinning), and **typed-action adapter** (map a closed verb to the tool's API). Everything is one of these three; nothing gets arbitrary host access. +4. **One canonical connector per tool.** Backstage's redundancy-and-abandonment failure is the thing to prevent structurally. +5. **Action semantics belong in the schema.** Flux's HelmRelease encodes drift-detection and remediation (retry/rollback/uninstall) as **typed fields** ([Flux HelmRelease](https://fluxcd.io/flux/components/helm/helmreleases/)) — proof that guardrails can live in the connector contract, not imperative glue. + +## 2. Per-tool integration surface, verified + +| Tool | Mechanism (verified) | Auth | Classification | +|---|---|---|---| +| **Argo CD** | **[3-0]** REST API w/ Swagger at `/swagger-ui` ([API docs](https://argo-cd.readthedocs.io/en/latest/developer-guide/api-docs/)) | **[3-0]** Bearer JWT via `/api/v1/session` | Read adapter **+ typed actions** (`argocd.sync\|rollback`) — day 1 | +| **Flux** | **[3-0]** **CRD-only** (`helm.toolkit.fluxcd.io/v2`); integrate by patching CRs + reading `.status` ([Flux](https://fluxcd.io/flux/components/helm/helmreleases/)) | Cluster RBAC | Read adapter (CRD), typed action later | +| **Helm** | **[3-0]** No server API — release state is **Kubernetes Secrets** in the release namespace ([Helm advanced](https://helm.sh/docs/topics/advanced/)) | Cluster RBAC | Read adapter via K8s API — day 1, free | +| **Prometheus** | **[3-0]** Stable `/api/v1`, **non-breaking additions only**; destructive TSDB ops segregated under `/admin` and **disabled by default** ([Prom API](https://prometheus.io/docs/prometheus/latest/querying/api/)) | proxy/none | Read adapter — day 1, low-churn | +| **Loki** | **[3-0]** Versioned `/loki/api/v1/query[_range]`, LogQL; **no built-in authz** — front it yourself ([Loki API](https://grafana.com/docs/loki/latest/reference/loki-http-api/)) | external | Read adapter | +| **Grafana** | **[3-0]** HTTP API w/ **service-account tokens** (replaced API keys) ([Grafana SA](https://grafana.com/docs/grafana/latest/administration/service-accounts/)) | SA token | Brokered read-through (link/deep-link; don't re-skin) | +| **GitHub/GitLab** | REST/GraphQL; GitHub App > PAT for scoping | App/OIDC | Typed action host for `gitops.open-pr` — day 1 (the first write) | +| **Terraform/OpenTofu** | State + HCP/Cloud API; providers via the protocol above | tokens | Read adapter (state/drift) later; never a Terraform runner | +| **Datadog / Splunk / Elastic/OpenSearch** | REST/query APIs | API/app keys | Read adapter, later; Datadog cost pain is a *pull* driver, not a reason to embed | +| **Fluentd / Fluent Bit** | **Config-only, no query API** | — | **Skip** as a data source (they ship logs *to* Loki/Elastic; read those instead) | +| **Jira / Zendesk / ServiceNow** | REST APIs | tokens | Typed action (open a change ticket for a fleet action) — later; ITSM change-linkage is real demand but not wedge | + +**Priority rule (from the wedge):** day-1 integrations are the ones that (a) feed the fleet model cheaply via the K8s API or a stable REST endpoint, or (b) host the first governed write. That's **Argo CD, Flux, Helm, Prometheus, Loki, and GitHub** — everything else is demand-ranked fast-follow, and Fluentd/Fluent Bit are skipped as sources. + +## 3. MCP protocol state (2026) — the protocol gives vocabulary, not enforcement + +- **[3-0]** Tool annotations (`readOnlyHint`/`destructiveHint`/`idempotentHint`/`openWorldHint`) shipped in the **2025-03-26** revision ([MCP annotations post](https://blog.modelcontextprotocol.io/posts/2026-03-16-tool-annotations/)). +- **[3-0]** The spec is explicit that **annotations are hints, untrusted unless from a trusted server** — a *risk vocabulary, not a security control*. +- **[2-0]** MCP maintainers state safety guarantees **must be enforced by deterministic controls outside the protocol** (network controls, sandboxing) — official acknowledgment that **governance is out of scope for the protocol**, left to gateways/governed servers. +- **[fetched]** Timeline: **2025-06-18** added **elicitation** (server-initiated mid-flow user input — the primitive HITL approval builds on) and formalized MCP servers as **OAuth Resource Servers**; **2025-11-25** matured the OAuth/OIDC-discovery authz stack and extended elicitation ([2025-06-18 changelog](https://modelcontextprotocol.io/specification/2025-06-18/changelog), [2025-11-25 changelog](https://modelcontextprotocol.io/specification/2025-11-25/changelog)). **Governance-relevant changes stop at auth + scope consent** — no policy/approval/blast-radius primitives. +- **[fetched]** **2025-12-09:** Anthropic donated MCP to the **Agentic AI Foundation**, a Linux Foundation directed fund (co-founded with Block and OpenAI; supported by Google, Microsoft, AWS, Cloudflare, Bloomberg), putting the protocol under "the same neutral stewardship that supports Kubernetes" ([MCP blog](https://blog.modelcontextprotocol.io/posts/2025-12-09-mcp-joins-agentic-ai-foundation/), [LF press release](https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation)). MCP is now vendor-neutral infrastructure — which *strengthens* the case for a neutral OSS governance layer on top: the protocol is a shared standard, the governance of what agents may do through it is still unbuilt. + +**Implication:** Sith's charter is right that MCP annotations are hints and enforcement must be server-side. The elicitation primitive is the correct native shape for the approval gate. The protocol will not govern for you — the governed MCP *server* is the product. + +## 4. Who governs AI agents on clusters today — the position is open + +**The demand is real and incident-backed:** +- **[fetched]** July 2025: **Replit's AI agent deleted the production database** of SaaStr's founder during "vibe coding", **despite explicit instructions not to change anything**; Replit's tooling then wrongly claimed the deletion was unrecoverable ([The Register](https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/)). The canonical "agent damaged prod" incident. +- **[fetched]** Third-party synthesis of KubeCon EU 2026: agentic AI was *the* story, agents proliferating, and "**nobody has quite figured out how to manage and secure them inside Kubernetes yet**". + +**Every AI-SRE incumbent stops at advise/diagnose or autonomy-first — none claims governed, approval-gated action as a neutral primitive:** +- **[fetched]** **kagent** (CNCF Sandbox, donated by Solo.io Apr 2025) is a *framework to run agents in-cluster* (agents as CRDs) — a runtime, **not** governance; a natural **MCP client of Sith** ([CNCF kagent](https://www.cncf.io/blog/2025/04/15/cncf-welcomes-kagent/)). +- **[fetched]** **HolmesGPT** stops at "natural-language diagnosis and remediation steps" — launch post has zero mentions of human/approval/guardrail/RBAC. +- **[fetched]** **Komodor** Klaudia multi-agent (GA 2026-03-18): the press release has **no mention** of approval/human-in-the-loop/guardrail/policy/governance/audit/RBAC — autonomy-first. +- **[fetched]** **Rancher "Liz" crew** (KubeCon EU 2026) is framed as **advisory** (insights/recommendations), no language about executing changes, no approval/guardrail primitives. + +**MCP gateways enforce auth + tool allowlists + audit — not fleet-aware, approval-gated action:** +- **[fetched]** **Kong AI Gateway 3.13** MCP Tool ACLs = identity-based per-tool allow/deny, default-deny, audit — Kong's own scope statement mentions **no human-approval, no HITL, no stop-mid-action** ([Kong MCP ACLs](https://konghq.com/blog/product-releases/mcp-tool-acls-ai-gateway)). +- **[fetched]** **Solo agentgateway + kagent** (closest K8s-native stack) = OIDC + token exchange + AccessPolicies binding which agent calls which tool; **no approval gates or action controls**; governance is in the **commercial** tier, leaving **OSS-native agent governance unoccupied** ([Solo kagent security](https://docs.solo.io/kagent/latest/security/)). +- **[fetched]** **MintMCP** claims generic tool-level approval gating and read/write enforcement — but **"generic per-tool rules with no fleet or cluster semantics"**; its only K8s reference is hosting its own connectors ([MintMCP agent gateway](https://www.mintmcp.com/blog/agent-gateway)). +- **[fetched]** **Permit.io** ships an "Access Request MCP" for human-approval-before-action — but **domain-generic** (copilots, support bots); the page has **no mention of Kubernetes/clusters/fleet** ([Permit.io](https://docs.permit.io/ai-security/access-request-mcp/overview/)). LangSmith/Langfuse give **observability/tracing**, not control. + +**The precise white space (triangulated from all of the above):** *approval-gated, fleet-aware, blast-radius-conscious governance of agent actions on Kubernetes clusters, as a vendor-neutral OSS primitive.* Generic approval gating exists (Permit.io, MintMCP). Tool allowlists + audit exist (Kong, Solo). Agent runtimes exist (kagent). Autonomy-first AI-SRE exists (Komodor, Rancher, Holmes). **The intersection — typed cluster-verbs + multi-approver + canary waves + abstention + signed dispatch + decision-ledger, applied identically to humans and to any MCP-client agent — is claimed by no one.** That is Sith's position, and the KubeCon EU 2026 read ("nobody has figured out how to manage/secure agents in K8s yet") is the market saying the door is open. + +## 5. What this means for Sith's plan + +- **Connector framework:** out-of-process gRPC, SDK-first, three fixed kinds, one canonical connector per tool, minor-additive versioning. This is a *fast-follow* deliverable — the wedge ships with hand-written adapters for the day-1 six; the framework generalizes them once the shape is proven (avoid Backstage's premature-ecosystem trap). +- **MCP server is the wedge's amplifier, not a separate bet:** because Sith enforces server-side through the same PEP, exposing the fleet as a governed MCP server makes *every* external agent (Claude Code, Codex, kagent) inherit the governance for free. That is the "governed MCP gateway to your whole fleet" position, and §4 shows it is empty. +- **Sith as MCP *client*** (calling kagent/Grafana MCP/GitHub MCP) is a later convenience, not the wedge. +- **The charter's AI stance is confirmed by evidence:** annotations are hints (verified), enforcement must be server-side (verified), elicitation is the HITL primitive (verified), and no incumbent occupies governed fleet action (verified). Hold the line: closed verb vocabulary, no shell, AI as client of the same PEP. diff --git a/docs/research/market-and-form-factor.md b/docs/research/market-and-form-factor.md new file mode 100644 index 0000000..126eebc --- /dev/null +++ b/docs/research/market-and-form-factor.md @@ -0,0 +1,268 @@ +# Sith — Market, tool landscape, and form factor (workstreams A · B · C · F · G) + +**Status:** research · **Date:** 2026-07-09 · **Provenance:** salvaged and consolidated from a deep-research fan-out (100+ verified sub-agent results per workstream, run 2026-07-08) plus a small amount of hand gap-filling (2026-07-09). Every load-bearing claim carries a primary-source link; vendor-commissioned figures are labelled as such. This document consolidates three workstream drafts — practitioner pains (A), form factor & developer experience (C), and the tool / cost / multi-cloud landscape (B · F · G). Companions: [identity-connections-security.md](identity-connections-security.md) (run D), [integrations-and-ai-governance.md](integrations-and-ai-governance.md) (run E), and the synthesis [USE-CASE-AND-SHAPE.md](USE-CASE-AND-SHAPE.md). + +**Contents** +- [Part 1 — Practitioner pains: global, China, India](#part-1) +- [Part 2 — Form factor and DX: the "Lens IDE" question](#part-2) +- [Part 3 — Tool landscape, cost, and multi-cloud gap map](#part-3) + +--- + + + +## Part 1 — Practitioner pains: global, China, India + +**Date:** 2026-07-08 · **Method:** web research with fetched primary sources; verbatim quotes pulled from each source. Load-bearing claims were re-checked by hand where flagged. Survey figures are cited with the exact page/figure where the source provides one. Vendor-commissioned research is labelled as such. + +This file answers: which pains are frequent and severe enough that a fleet-operations tool must kill them, and what China and India add. + +--- + +## 1. The quantitative baseline: fleets are real, and they hurt + +| Fact | Figure | Source (fetched) | +|---|---|---| +| Fleet size is mainstream, not exotic | "The average Kubernetes adopter now operates more than 20 clusters"; 56% have >10 | [Spectro Cloud, State of Production Kubernetes 2024](https://www.spectrocloud.com/blog/ten-essential-insights-into-the-state-of-kubernetes-in-the-enterprise-in-2024) (n=416, vendor survey) | +| Same figure from a second, independent vendor | "A typical enterprise now runs more than 20 clusters, with nearly half operating across more than four environments"; 37% manage >100 clusters, 12% >1,000 | [Komodor 2025 Enterprise Kubernetes Report](https://komodor.com/blog/komodor-2025-enterprise-kubernetes-report-finds-nearly-80-of-production-outages/) (customer telemetry, not a survey) | +| Fleets span environments | Half of businesses run clusters in 4+ environments (clouds, DCs, edge) | Spectro Cloud 2024, above | +| Hybrid is the norm | 86% deploy across both public and private cloud | [Portworx, Voice of Kubernetes Experts 2024](https://portworx.com/wp-content/uploads/2024/06/The-Voice-of-Kubernetes-Experts-Report-2024.pdf) (n=527, 500+ employee orgs, Dimensional Research) | +| Estates skew self-managed and multi-cloud | 59% self-managed on-prem and 59% self-managed public cloud; 37% use 2 cloud providers, 26% use 3 | [CNCF Annual Survey 2024 PDF](https://www.cncf.io/wp-content/uploads/2025/04/cncf_annual_survey24_031225a.pdf) (n=689–750 depending on question) | +| Complexity is the headline pain | "Three quarters of businesses that use Kubernetes today say their adoption of K8s has actually been inhibited by the complexity" | Spectro Cloud 2024, above | +| Change is the outage engine | "79% of production issues originate from a recent system change" | Komodor 2025, above | +| Toil is quantified | >60% of ops time spent troubleshooting; 64+ workdays/year lost; median MTTD ~40 min, MTTR >50 min for high-impact outages; 38% see high-impact outages weekly | Komodor 2025, above (telemetry sample caveat) | +| Cost pressure is rising | Nearly two-thirds report K8s TCO grew and face more cost pressure than a year ago, with poor visibility into future cost | Spectro Cloud 2024, above | +| Waste is endemic | ">82% of Kubernetes workloads are overprovisioned (65% use less than half of the CPU and memory they request)" | Komodor 2025, above | +| Lock-in anxiety is measurable | 55% "already feel locked in"; more than half worry about vendors shutting down | Spectro Cloud 2024, above | +| Demand for a central plane exists | 71% say a unified/centralized platform would greatly benefit them; 43% specifically want streamlined hybrid/multi-cloud management across Kubernetes environments | Portworx 2024, above (framed around data services — Portworx's domain; treat as directional) | +| The buyer function exists almost everywhere | 96% of surveyed enterprises have a platform-engineering function; drivers include cost (49%) and security mandates (43%) | Portworx 2024, above | + +Two honest caveats on this table. First, three of the five sources are vendor-commissioned; the figures converge (20+ clusters, 4+ environments, complexity/cost as top pains), which is why they are usable, but no single number should be treated as precise. Second, the CNCF survey's own challenge ranking (Figure 8: cultural change 46%, CI/CD 40%, training 38%, security 37%, monitoring 36%, complexity 35%) shows people/skills pains now rank alongside technical ones — a tool that requires deep OCM or fleet-theory expertise to install will fail the very population that needs it. + +### What the CNCF survey says about *adopting a new OSS tool* (directly actionable for Sith) + +The top 2024 barriers to running open-source projects in production: fear the project becomes inactive (46%, +9 pts YoY), too complex to understand or run (46%, +13 pts), lack of documentation (45%, +5 pts) ([CNCF 2024 PDF](https://www.cncf.io/wp-content/uploads/2025/04/cncf_annual_survey24_031225a.pdf)). Security-vulnerability concern *fell* to 29%. The adoption killers for a new tool are perceived abandonment risk, complexity, and thin docs — not security scanners. Sith's counter must be structural: trivial install, excellent docs, visible release cadence. + +## 2. Ranked pains a fleet tool must kill + +Ranked by convergence of survey figures and community evidence. (The community-thread search lane of this research was partially rate-limited; where thread-level citations are thin the ranking leans on the survey base above and the tool-exodus evidence in [Part 2](#part-2).) + +1. **"What is happening across my clusters right now?" — fleet-wide visibility and correlation.** 20+ clusters across 4+ environments with per-cluster consoles means N logins to answer one question. The Portworx 71%/43% centralization demand and the Komodor MTTD figures quantify it. Single-cluster tools structurally cannot answer "which clusters run image X" or "where is `payments` degraded". +2. **"What changed, where?" — change attribution across the fleet.** 79% of production issues stem from a recent change (Komodor). A fleet tool that can answer "what changed in the last hour, across which clusters, by whom" attacks the single largest outage cause. +3. **Doing the same action on N clusters safely.** The action side of pain 1. Today it is a shell loop over kubeconfigs (no gates, no audit, no rollback) or a heavyweight platform (Rancher/ACM). Severity evidence is indirect but strong: change-driven outages (79%), weekly high-impact outages (38%), and the absence of any OSS primitive for gated fan-out (see [Part 3](#part-3) §OCM). +4. **Credential and access sprawl.** Kubeconfig-per-cluster with exec plugins, shared admin credentials, no per-user attribution. Survey proxy: security named by 37% (CNCF). The predecessor's own failure (central god-kubeconfig) and the entire brokered-access market (Teleport et al., see [identity-connections-security.md](identity-connections-security.md)) exist because of this pain. +5. **Cost visibility across clusters and clouds — including GPU.** Two-thirds face rising TCO with poor visibility (Spectro Cloud); >82% overprovisioned (Komodor); GPU is the extreme case — see §3. +6. **Complexity/skills mismatch.** 75% say complexity inhibited adoption (Spectro Cloud); training gaps at 38% (CNCF). A fleet tool must *reduce* the expertise required, not add a new discipline. +7. **Vendor lock-in anxiety.** 55% feel locked in (Spectro Cloud); the 2025 SUSE/Rancher repricing (competitor-reported 4–9x increases — [Portainer's account](https://www.portainer.io/blog/suse-rancher-price-hike-why-enterprises-are-searching-for-alternatives-in-2025), cross-check against SUSE's [shop page](https://www.suse.com/shop/suse-rancher-prime/)) and the Kubecost/IBM and Lens/Mirantis consolidations feed it. Vendor-neutral OSS is a distribution advantage here, not a nicety. + +## 3. MLOps / GPU fleet pains + +- **GPU waste is extreme and measured.** CAST AI's 2026 analysis of tens of thousands of production clusters: **average GPU utilization 5%** ("95% of GPU capacity is doing nothing"), vs 8% CPU and 20% memory ([CAST AI 2026 State of Kubernetes Optimization](https://cast.ai/press-release/2026-state-of-kubernetes-optimization-report/); vendor telemetry). The ClearML/AIIA survey (~1,000 AI leaders): only 7% achieve >85% peak GPU utilization; 74% dissatisfied or only moderately satisfied with job-scheduling tools; 93% say easy self-serve compute would substantially raise productivity ([ClearML State of AI Infrastructure at Scale 2024](https://clear.ml/blog/the-state-of-ai-infrastructure-at-scale-2024)). +- **Per-team GPU cost attribution is formalized as a discipline but only single-cluster.** The FinOps Foundation working group codifies namespace-as-cost-centre labeling enforced at admission, showback-then-chargeback, and MIG for right-sizing ("sub-5 GB models hogging an entire A100") — and its paper **covers only single-cluster scenarios; cross-cluster allocation is unaddressed** ([FinOps WG: Scaling Kubernetes for AI/ML with FinOps](https://www.finops.org/wg/scaling-kubernetes-for-ai-ml-workloads-with-finops/)). That gap is exactly the fleet layer. +- **In-cluster GPU scheduling is being commoditized; the fleet layer is not.** NVIDIA open-sourced its Run:ai-derived KAI-Scheduler (CNCF Sandbox) for fairness across teams at thousand-node scale ([KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler)). The defensible pains for a fleet tool sit *above* the scheduler: federated per-team usage/cost across N clusters, and governed actions on training/inference fleets. +- **ML platform teams re-invent governed access.** ZenML's platform guidance: data scientists should never hold raw cluster credentials; access should be centrally brokered ([ZenML on 8xH100 multi-tenancy](https://www.zenml.io/blog/managing-mlops-at-scale-on-kubernetes-when-your-8xh100-server-needs-to-serve-everyone)). MLOps is thus a *user segment* of the same governed-access/action wedge — not a separate product. +- 54% of enterprise K8s orgs already run AI/ML on Kubernetes (Portworx 2024, above), so GPU columns in the fleet model serve a mainstream slice, not a niche. + +## 4. China + +The CNCF survey cannot support China conclusions (~3% of respondents China-HQ'd — the survey says so itself, [2024 PDF](https://www.cncf.io/wp-content/uploads/2025/04/cncf_annual_survey24_031225a.pdf) demographics). The evidence below is from Chinese-language primary sources (vendor docs, practitioner tutorials) and is labelled accordingly. Direct practitioner-sentiment evidence at scale is thin in public sources; that gap is flagged rather than papered over. + +**4.1 Air-gap/offline is a first-class, routine deployment mode — not an edge case.** +- KubeSphere documents air-gapped installation as a standard path (KubeKey artifact + private Harbor) ([KubeSphere docs, 离线安装](https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/air-gapped-installation/)). +- A representative practitioner walkthrough (运维有术 series on Tencent Cloud's developer community) builds the offline bundle on a connected node and carries it in; the full artifact is ~13 GB and populates 124 image repos in self-hosted Harbor; the topology is described as a 1:1 replica of a small production environment ([cloud.tencent.com/developer/article/2419243](https://cloud.tencent.com/developer/article/2419243)). +- The assumed baseline is that public registries are unreachable: the tutorial *verifies* isolation by showing `docker.io` image pulls failing on Chinese public DNS. A separate xinchuang field report states plainly that hub.docker.com "is currently not accessible from within China — figure out your own way" ([CSDN, 信创适配实战](https://blog.csdn.net/yztezhl/article/details/139698545), June 2024). + +**4.2 Compliance shapes tooling: MLPS 2.0 (等保 2.0) is operationalized, not theoretical.** +- Alibaba ACK ships MLPS 2.0 Level 3 hardening as a cluster-creation option implementing GB/T 22239-2019 ([ACK 等保加固说明](https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/security-and-compliance/ack-reinforcement-based-on-classified-protection)). Level-3 hardening mandates: **role separation** (ACK creates distinct `ack_admin`, `ack_audit`, `ack_security` users), **tamper-resistant, backed-up audit records** ("应对审计记录进行保护,定期备份,避免受到未预期的删除、修改或覆盖"), and **no root SSH** — i.e., hardened Chinese estates structurally favor governed, audited, agent-mediated access over interactive credentials. +- The MLPS model explicitly allows substituting self-attested equivalent controls ("如果有其他方式,可自行举证并忽略此项") — a self-hosted OSS tool can legitimately slot into an MLPS-graded estate. +- **Xinchuang (信创) localization** adds: domestic OSes (openEuler/OpenAnolis, or Kylin/UOS when customers demand), domestic ARM CPUs (Kunpeng/Phytium), hence **linux/arm64 multi-arch images are a hard requirement**, and tolerance for older pinned K8s versions (the field report ran v1.24 in mid-2024) ([CSDN field report](https://blog.csdn.net/yztezhl/article/details/139698545)). + +**4.3 The cloud substrate is conformant Kubernetes.** Alibaba ACK, Huawei CCE and Tencent TKE all hold current CNCF Certified Kubernetes conformance (hand-verified in [cncf/k8s-conformance](https://github.com/cncf/k8s-conformance): `v1.32/alicloud` = "Alibaba Cloud Container Service for Kubernetes v1.32.0"; `huawei-cce` submissions continuously v1.29–v1.34; `v1.34/tencentcloud` = "Tencent Kubernetes Engine v1.34.1"). All three issue standard kubeconfigs (RAM/IAM/CAM-integrated) with private-VPC endpoints ([ACK kubeconfig](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/obtain-the-kubeconfig-file-of-a-cluster-and-use-kubectl-to-connect-to-the-cluster), [CCE permissions](https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_10_0187.html), [TKE connecting](https://www.tencentcloud.com/document/product/457/30639)). Rancher already imports ACK/CCE/TKE via its outbound agent ([SUSE announcement](https://www.suse.com/c/rancher_blog/announcing-added-support-for-leading-kubernetes-services-in-china/)) — feasibility proof for the same pattern. +- Chinese-language practitioner comparisons of multi-cluster tools (Rancher vs KubeSphere vs Karmada) turn on cluster import, network reachability and unified permissions ([Zhihu 多集群管理工具对比](https://zhuanlan.zhihu.com/p/539203985), [Kubernetes多集群管理之路](https://zhuanlan.zhihu.com/p/584378217) — the latter names heterogeneous cluster types, 10k-node scale spread, per-region compliance differences, version drift, and "scattered permissions with uncontrolled security risk" as the operating reality). + +**4.4 What this means for Sith in China.** A US-hosted SaaS control plane (Komodor, Datadog, Lens-with-account) is structurally disadvantaged: GFW/egress policy, data-residency posture, and procurement all push toward self-hosted. Sith fits **if and only if** it ships: (1) fully offline installation (single bundle, Zarf-style, no phone-home — see [Part 3](#part-3) on Zarf); (2) linux/arm64 + x86 multi-arch images and support for openEuler/Kylin-class hosts; (3) registry-relocatable images (no hardcoded docker.io/gcr.io pulls); (4) tamper-evident audit logs and admin/auditor role separation (maps directly to MLPS L3); (5) hub-and-spoke that works entirely inside one network boundary (the hub is self-hosted in-country; outbound-only spokes work within/behind the boundary). These are the same properties the federated design already targets — China raises their priority from "nice" to "mandatory for the market". + +## 5. India + +- **The buyer base is large and institutional.** 1,700+ GCCs (2,975+ units) generating $64.6B and employing ~1.9M as of FY2024, projected to 2,100–2,200 centers and $99–105B by 2030 ([Nasscom–Zinnov India GCC Landscape 2026 PDF](https://media.zinnov.com/wp-content/uploads/2026/05/zinnov-nasscom-india-gcc-landscape-2026-report.pdf)). GCCs increasingly own engineering/platform functions for global parents — i.e., they operate exactly the multi-cluster, multi-environment estates Sith targets. IT-services majors position Kubernetes as the multi-cloud abstraction across client estates ([Wipro on K8s-native multi-cloud](https://www.wipro.com/blogs/sreekanth-nyamars/kubernetes-native-design-thinking-realizing-true-multi-cloud-adoption/) — vendor voice; treat as estate-shape evidence, not pain ranking). The service-integrator profile adds a specific requirement: **hard isolation between client estates in one operator's tooling** — Sith's workspace model must treat "many clients, one operator" as a first-class shape. +- **Regulatory pull is procurement-level, not absolute.** DPDP Act 2023 (+ DPDP Rules 2025) uses a blacklist approach to cross-border transfer rather than blanket localization, but the government can pin specified data of "significant data fiduciaries" in-country; phased compliance runs to ~May 2027 with penalties to ₹250 crore ([EY analysis](https://www.ey.com/en_in/insights/cybersecurity/decoding-the-digital-personal-data-protection-act-2023)). The hard residency floor is sectoral: RBI's 2018 directive requires payment-system data stored only in India ([Google Cloud RBI compliance page](https://cloud.google.com/security/compliance/rbi-india)). BFSI is a top GCC vertical, so in-country self-hosting is often a de-facto requirement; SaaS buyers increasingly ask for India hosting or on-prem options ([Wattlecorp DPDP-for-SaaS guide](https://www.wattlecorp.com/saas-providers-guide-to-dpdp-act-india/) — mid-tier source). Self-hostable OSS derisks all of this by construction. +- **Cost sensitivity + skills profile.** GCC hiring data indicates a 40%+ skills gap in tech roles and a cited 55–60% deficit in cloud-native expertise, with salary premiums for K8s/CI-CD engineers ([Savannah HR aggregation of NASSCOM/Deloitte figures](https://savannahr.com/blog/top-8-gcc-skills-india-2026/) — recruiting-blog source, verify against the underlying reports before quoting numbers). Implication: free OSS core with radically simple install and opinionated safe defaults beats "powerful but expert-only". Direct India practitioner sentiment on fleet tooling is thin in public sources — flagged as a gap; the estate-shape and regulatory evidence above is the reliable part. + +## 6. Conclusions carried into the synthesis + +1. The pains Sith must kill, in order: fleet-wide visibility/correlation; change attribution; safe fan-out actions; access/credential sprawl; fleet cost overlay (GPU included). These map 1:1 onto read federation, action federation, and a cost read-integration — the charter's wedge survives contact with the demand evidence. +2. OSS-adoption barriers (abandonment fear, complexity, docs) mean the *form* of the product decides adoption as much as the wedge. A 10-minute, single-binary first-run is not polish; it is the counter to the #1 and #2 adoption barriers. +3. China: air-gap bundle, multi-arch, registry relocation, tamper-evident audit, role separation — mandatory, and cheap if designed in early. +4. India: multi-client workspace isolation and self-hostability are the fit; price the OSS core at zero and keep the install trivial. +5. MLOps is a segment, not a separate product: GPU util/cost columns in the fleet model + the same governed actions. + + +--- + + + +## Part 2 — Form factor and DX: the "Lens IDE" question + +**Date:** 2026-07-08 · **Method:** fetched primary sources with verbatim quotes; GitHub figures read from the live pages/API on 2026-07-08; the Kubernetes-Dashboard-to-Headlamp claim, k9s stats, and China-cloud conformance were re-verified by hand. + +This file answers: what happened to Lens and where users went; what a local Kubernetes tool must have; what makes tools feel effortless; what technology to build on; and whether local + federated should be one product or two. + +--- + +## 1. The Lens story, verified in both directions + +The "Lens went paywall/telemetry" narrative is **real but more precise than the folklore**: + +| Date | Event | Source | +|---|---|---| +| 2020-08-13 | Mirantis acquires Lens ("world's most popular Kubernetes IDE", MIT-licensed, ~35k users); founder quoted promising it "would remain vendor neutral and open source" | [Mirantis press release](https://www.mirantis.com/company/press-center/company-news/mirantis-acquires-lens-the-worlds-most-popular-kubernetes-ide/) | +| ~2022-05/06 (Lens 5.5.x) | Mandatory, non-skippable **Lens ID login** lands; team concedes it "could have done better at communicating the change" | [lensapp/lens#5444](https://github.com/lensapp/lens/issues/5444) | +| 2022-07 | **Lens 6 subscription model**: Lens Pro $19.90/user/mo; free Personal tier restricted to individuals and orgs under $10M revenue/funding | [Mirantis announcement](https://www.mirantis.com/blog/lens-pro-vision-for-the-future-new-subscription-model-new-features-available/); backlash: [HN "Lens goes subscription only"](https://news.ycombinator.com/item?id=32269258), [HN on the acquisition/login](https://news.ycombinator.com/item?id=32408122) | +| 2022-12/2023-01 (6.3.0) | Pod **logs/shell menus removed from the open-source build** (moved to an extension); the single most-cited exodus trigger | [lensapp/lens#6823](https://github.com/lensapp/lens/issues/6823); [OpenLens README](https://github.com/MuhammedKalkan/OpenLens) ("type `@alebcay/openlens-node-pod-menu` into the Extensions page") | +| 2024-03 | **Lens closes its source**; OpenLens build repo freezes (last release v6.5.2, 2023-06-30; repo README: "Lens Closed its source code. So please do not expect any more updates.") | [OpenLens repo](https://github.com/MuhammedKalkan/OpenLens); [HN thread](https://news.ycombinator.com/item?id=39811772); alternatives catalogued in [lensapp/lens#8008](https://github.com/lensapp/lens/issues/8008) | +| Today (verified 2026-07-08) | Lens pricing: Personal **free** (under $10M revenue/funding) and still includes multi-cluster management, metrics, logs, terminal, Helm, port-forwarding, resource editing; Plus $25/user/mo (AI copilot, EKS/AKS auto-discovery, Security Center); Enterprise custom. Telemetry: Lens "may automatically communicate with Mirantis servers" for updates/usage tracking, **opt-out available**, vendor asserts no kubeconfigs/secrets uploaded | [lenshq.io/pricing](https://lenshq.io/pricing) (k8slens.dev/pricing redirects here); [Lens licensing/telemetry FAQ](https://docs.k8slens.dev/faq/subscription-and-licensing/) | + +So the adversarial check lands here: **core features are mostly still free; what users actually revolted against was the account wall, the trust break (closed source after a vendor-neutrality promise), telemetry-by-default, and the removal of logs/shell from the OSS build.** Those four things — not any missing feature — created the exodus. A new tool wins that audience by structural promises: no account, no telemetry, open source, logs/exec in core forever. + +### Where the exodus went (traction figures as of 2026-07-08) + +| Tool | What it is | Traction | Gap it leaves | +|---|---|---|---| +| [Freelens](https://github.com/freelensapp/freelens) | MIT fork of OpenLens (Electron, TypeScript), no account/telemetry | **5.3k stars in ~2 years** (repo created 2024-06-19), v1.10.3 released 2026-07-07, ~monthly cadence, 29 releases | Inherits Lens's one-cluster-at-a-time UX and Electron weight; a continuation, not a rethink | +| [OpenLens builds](https://github.com/MuhammedKalkan/OpenLens) | Login-free Lens binary | 4.4k stars, **dead since 2023-06** | Demand signal for "no login" — 4.4k stars for a *build repo* | +| [k9s](https://github.com/derailed/k9s) | Terminal UI, single Go binary | **34.1k stars**, v0.51.0 (2026-06-06), Apache-2.0, `brew install derailed/k9s/k9s` | **One context at a time** (`:ctx` to switch); no aggregated fleet view; TUI ceiling for sharing/visualization | +| [Headlamp](https://github.com/kubernetes-sigs/headlamp) | Kubernetes SIG-UI web UI + desktop app, plugin system | 6.8k stars; v0.43.0 (2026-06-16); monthly releases; AI assistant via MCP; Artifact Hub plugin catalog ([2025 highlights, kubernetes.io](https://kubernetes.io/blog/2026/01/22/headlamp-in-2025-project-highlights/)) | Per-cluster-centric UX (multi-cluster registration exists; ClusterProfile inventory is alpha); no fleet correlation; no governed actions | +| [Aptakube](https://aptakube.com/) | Closed-source Tauri desktop client | Vendor claims "thousands" of users; **$9/mo personal / $7/seat teams, 15-day trial, no free tier**; installers 15–28 MB | Proves the paying gap: its headline claim is being "the **only** Kubernetes UI that can connect to multiple clusters simultaneously" and aggregate resources in one view, plus "no extra configuration… data never leaves your machine" ([aptakube.com](https://aptakube.com/), [lens-alternative page](https://aptakube.com/lens-alternative)). Closed and paid — the OSS slot for exactly this is **empty** | + +**The pivotal ecosystem event (hand-verified):** the official **Kubernetes Dashboard is archived**, and the kubernetes.io blog names Headlamp the way forward *explicitly because of* "multi-cluster visibility … and flexible deployment options that work both in-cluster and on the desktop" ([From Kubernetes Dashboard to Headlamp, 2026-06-01](https://kubernetes.io/blog/2026/06/01/dashboard-to-headlamp/), authored by Will Case/Headlamp). Read as market evidence: the single-mode, single-cluster web console lost; the dual-mode, multi-cluster tool became the community default. Any Sith local mode is therefore **not** competing with a vacuum — Headlamp is CNCF-blessed and improving monthly. Sith's local mode must not be "another general console"; it must be the **fleet** view Headlamp doesn't center on (aggregation, correlation, staleness, governed actions) — see §5. + +### Table-stakes for a Lens-class local tool (ranked by how often the evidence cites them) + +1. Multi-cluster from existing kubeconfig, zero config ("if you're already using kubectl, it just works" — the Aptakube pitch). +2. Pod logs + exec/shell in core (their removal *created* the OpenLens exodus — [#6823](https://github.com/lensapp/lens/issues/6823)). +3. No account/login wall (4.4k stars on a build repo whose only feature was deleting the login). +4. No telemetry, or opt-in only; local-only data ("never leave your machine"). +5. Fast on big clusters; low memory (Electron complaints are constant in Lens-alternative threads). +6. Resource browse + YAML edit, port-forward. +7. Open source under a permissive license, active cadence (CNCF-survey adoption barriers: abandonment fear, docs). +8. Aggregated multi-cluster single view — the one thing users can otherwise only buy (Aptakube). + +## 2. Why the effortless tools feel effortless (transferable mechanics) + +- **Latency budget:** 0.1 s = direct manipulation; 1 s = flow intact; 10 s = attention lost ([Nielsen/NN-g response-time limits](https://www.nngroup.com/articles/response-times-3-important-limits/)). Palette open, fuzzy search, and view switches must land under ~100 ms — only achievable when rendering from a **local cache/store**, not a per-keystroke round-trip to N API servers. +- **Local-first with background sync (the Linear mechanic):** the server is a sync target, not the UI's source of truth; the client hydrates a local store, every query hits it first, pages render in <50 ms with no spinners; deltas reconcile asynchronously ([How is Linear so fast](https://performance.dev/how-is-linear-so-fast-a-technical-breakdown); practitioner corroboration: [local-first rabbit hole](https://bytemash.net/posts/i-went-down-the-linear-rabbit-hole/)). Kubernetes has the perfect substrate for this: **watch streams into a local informer cache**. This is the single most important DX decision for Sith's local mode: cache-first render + staleness stamps, never spinner-first. +- **Keyboard-first + command palette:** the cmd-K bar bridges GUI discoverability and CLI speed and is the canonical pattern for serving both kubectl power users and GUI users in one product ([Maggie Appleton, Command K Bars](https://maggieappleton.com/command-bar)). k9s is the in-domain proof that keyboard-first wins operators (34.1k stars). +- **Install friction:** k9s (`brew install`, one binary, no server, no account) and Freelens (brew cask + winget/scoop/flatpak/deb/rpm, arm64+amd64) define the funnel. Tailscale's "install, sign in, connected — value on first run" is the canonical 10-minute-wow articulation ([why-tailscale](https://tailscale.com/why-tailscale)). Contrast: "helm install a platform, configure SSO, then see value" is the devops-portal/Backstage adoption cliff. **Sith's local mode must be a single artifact that shows a populated fleet view within minutes of `brew install`.** + +## 3. Desktop technology: Electron vs Tauri vs neither + +Benchmarks and production reports (all fetched): +- Same-app comparisons: Tauri installer ~2.5 MB vs Electron ~85 MB; idle RAM ~80 MB vs ~120 MB; cold start ~2 s vs ~4 s ([Authme dev, levminer.com](https://www.levminer.com/blog/tauri-vs-electron)). Minimal-app: 8.6 MiB vs 244 MiB bundle; ~172 MB vs ~409 MB with 6 windows; **startup difference negligible** in that test ([Hopp benchmark, 2025](https://www.gethopp.app/blog/tauri-vs-electron)). +- Counterweights: Tauri renders via OS webviews (WebView2/WKWebView/WebKitGTK) → **cross-platform rendering inconsistency** to manage, vs Electron's identical-everywhere Chromium; DoltHub stayed on Electron for packaging gaps (no .appx/.msix, no macOS universal binaries) while still concluding Tauri "eliminates much of the classic Electron bloat" ([DoltHub, 2025-11](https://www.dolthub.com/blog/2025-11-13-electron-vs-tauri/)); post-migration retrospective: [Fluxzy five months after](https://www.fluxzy.io/resources/blogs/electron-to-tauri-migration-fluxzy-desktop). +- Category proof: **Aptakube ships on Tauri** in exactly this product class, marketing the small/fast footprint against Electron-based Lens; other K8s clients on Tauri exist (JET Pilot, Kunobi — [awesome-tauri](https://github.com/tauri-apps/awesome-tauri)); Tauri's sidecar mechanism cleanly wraps an existing backend binary (DoltHub, Hopp). + +**Recommendation:** don't make the webview choice the first decision. The proven architecture in this exact category is **Headlamp's**: one Go backend + one web frontend, identical in both modes, with the desktop app being a thin shell over the same code ([Headlamp architecture docs](https://headlamp.dev/docs/latest/development/architecture/)). Sith should ship a **single Go binary** whose `sith ui` serves the local web UI from the embedded frontend (k9s-grade install friction, no app-store/codesigning tax on day one), plus a first-class CLI. A **Tauri** shell (not Electron — memory/footprint evidence above, and Lens-refugee sensitivity to Electron bloat) is a fast-follow for dock presence/deep-OS integration, wrapping the same binary as a sidecar. This sequences the risk: the web UI must exist in both form factors anyway; the wrapper is additive. + +## 4. One product or two? The dual-mode precedents + +| Precedent | Shape | Verdict | +|---|---|---| +| **Headlamp** | Same Go backend + React frontend runs as single-user desktop app (local kubeconfigs) and as in-cluster multi-user web deployment; docs state the modes are "not mutually exclusive" — individuals use desktop while the org runs in-cluster ([installation docs](https://headlamp.dev/docs/latest/installation/), [architecture](https://headlamp.dev/docs/latest/development/architecture/)) | **Strongest validation.** The dual mode is *why* it won the Dashboard succession ([kubernetes.io, 2026-06-01](https://kubernetes.io/blog/2026/06/01/dashboard-to-headlamp/)) | +| **Portainer** | One server binary; solo homelab user runs it alone; the same server accepts agents for centralized multi-environment management; Business layers RBAC on the same core ([architecture docs](https://docs.portainer.io/start/architecture)) | Validates "same product, agents arrive later, governance is a layer" | +| **Grafana** | Same OSS core run locally/self-hosted; Cloud adds hosting + governance features ([oss-vs-cloud](https://grafana.com/oss-vs-cloud/)) | Validates monetizing/governing a tier above one core | +| **Teleport** | Desktop client (Teleport Connect, Electron) lives in the same monorepo as the server/Web UI, sharing infrastructure ([Teleport Connect docs](https://goteleport.com/docs/connect-your-client/teleport-clients/teleport-connect/)) | Weaker (client-of-server, not standalone local mode), but same-repo/shared-UI economics hold | + +**Conclusion: one product, one binary, two modes.** Every relevant precedent shares code between local and served modes; the community's own Dashboard→Headlamp migration explicitly rewarded the dual mode. Two separate products would double surface area (the devops-portal failure mode) and break the land-and-expand path (§5). + +## 5. The recommended shape for Sith + +- **One Go binary, three run modes:** `sith` (CLI verbs), `sith ui` (local web UI on localhost, kubeconfig-direct, single user, zero config, no account, no telemetry), `sith hub` (the federated control plane serving the *same* UI multi-user with workspaces/governance). One frontend; the UI renders identically from a local cache (direct mode) or the hub's fleet model (federated mode). +- **The local mode's center of gravity is the fleet, not the pod.** Aggregated all-clusters resource views, fleet search/correlation ("which clusters run image X", "where is payments degraded"), staleness stamps, and the same typed-verb actions with dry-run/diff — locally self-approved, but the same intent model that later gains real governance. Per-pod table stakes (logs, exec, port-forward, YAML edit) must exist because their absence created the Lens exodus — but they are commodity K8s API calls, not integrations, and Sith should not chase Headlamp/k9s feature-for-feature beyond them. +- **The wow:** `brew install sith && sith ui` → all kubeconfig contexts detected → one aggregated fleet view with cmd-K fuzzy search across every cluster, under 10 minutes, offline-capable, nothing leaves the machine. +- **The expand:** when the team needs shared visibility and real approvals, the same binary becomes the hub; clusters upgrade from "direct (my kubeconfig)" to "minion (outbound OCM agent)" without the user relearning anything. Local mode is the top of the funnel for the governed wedge — not a second product. + + +--- + + + +## Part 3 — Tool landscape, cost, and multi-cloud gap map + +**Date:** 2026-07-08 · **Method:** fetched primary sources with quotes; facts marked *(repo-verified)* were web-verified in this repository's [`COMPETITIVE.md`](../../COMPETITIVE.md) during the July 2026 planning pass and re-used here; hand-verified items are marked. Where pricing is quote-based or was not directly fetched, the row says so rather than inventing numbers. + +The question this file answers: what does each incumbent actually provide and charge, and what is missing across *all* of them — the white space Sith can own. + +--- + +## 1. Local / single-operator clients + +Covered in depth in [Part 2](#part-2). Summary of the gap: **k9s** (34.1k stars, one context at a time — hand-verified), **Lens** (account wall, telemetry-with-opt-out, Plus $25/user/mo; Personal free under $10M), **OpenLens** (dead 2023), **Freelens** (healthy MIT fork, Electron, Lens-lineage single-cluster UX), **Headlamp** (SIG-UI, dual-mode, per-cluster-centric, plugin system), **Aptakube** (closed, $9/mo, the only aggregated multi-cluster single view), **kubectl** (the substrate everyone shells out to; no fleet semantics beyond contexts). **White space: an OSS, no-account, no-telemetry, aggregated multi-cluster client with fleet-level search/correlation.** Nobody occupies it; the closest occupant is closed and paid (Aptakube), and the community default (Headlamp) centers on per-cluster views. + +## 2. GitOps / deploy layer (things Sith integrates with, never replaces) + +| Tool | What it is | Relevant boundary facts | +|---|---|---| +| **Argo CD** | CNCF-graduated GitOps CD; REST/gRPC API + `Application` CRDs; its own multi-cluster story is being agent-ified by [argocd-agent](https://github.com/argoproj-labs/argocd-agent) (v0.9.0, 2026-06-04 *(repo-verified)*) | Argo federates *its own* surface only; no cross-tool governed actions. Sith's `argocd.sync|rollback` verbs ride its API | +| **Flux** | CNCF-graduated GitOps controllers; **CRD-only surface** (no API server) — integration = read/patch CRs | Same boundary: reconciler, not an ops control plane | +| **Helm / Kustomize** | Package/overlay standards; Helm state lives in in-cluster release Secrets | Read adapters for inventory ("what release/version is where"), not action targets in v1 | +| **Kargo** ([akuity/kargo](https://github.com/akuity/kargo)) | GitOps *promotion* orchestration; Apache-2.0, v1.10.8 (2026-06-25), ~3.4k stars; open-core (Kargo Enterprise by Akuity, inquiry pricing) | **Closest OSS analog to "governed promotion"** — stages with gates and Git as audit trail — but it promotes *artifacts through environments* via GitOps; it is Argo-ecosystem-coupled and does not execute typed live operations (restart/scale/drain) across heterogeneous fleets. The repo page itself doesn't substantiate multi-approver gates (verify against docs.kargo.io before citing specifics). Boundary, and validation that "promotion with gates" resonates | + +The lesson the predecessor already paid for: re-skinning these tools is negative value. Sith reads their state into the fleet model and dispatches a **closed set of typed verbs** to their APIs — nothing else. + +## 3. Fleet / platform incumbents + +| Tool | Provides | Pricing/positioning | What it does NOT do | +|---|---|---|---| +| **Rancher / Rancher Prime (SUSE)** | Cluster provisioning + import (outbound `cattle-cluster-agent`), RBAC, catalog; Fleet = GitOps at scale; 2026: "Liz" agentic AI expanding to a crew (Linux/Observability/Security/Provisioning/Fleet) + MCP server integration ([SUSE KubeCon EU 2026 announcement](https://www.suse.com/c/kubecon-eu-2026-first-agentic-ecosystem-platform/)) | Prime = paid subscription ([SUSE shop](https://www.suse.com/shop/suse-rancher-prime/), per-node/per-vCPU tiers); 2025 repricing to a vCPU metric with **competitor-reported 4–9x increases driving alternatives-shopping** ([Portainer's account](https://www.portainer.io/blog/suse-rancher-price-hike-why-enterprises-are-searching-for-alternatives-in-2025) — rival-authored, treat accordingly) | No typed-intent action vocabulary, no multi-approver fan-out gates, no abstention semantics; agentic AI is subscription-gated assistance, not a neutral governance primitive. Imports ACK/CCE/TKE ([SUSE](https://www.suse.com/c/rancher_blog/announcing-added-support-for-leading-kubernetes-services-in-china/)) — feasibility proof for agent-based China coverage | +| **OpenShift + ACM (Red Hat)** | The enterprise platform; ACM is the multi-cluster manager **built on OCM upstream** *(repo-verified: OCM underpins ACM)* | Subscription platform; OpenShift-first | ACM governs via Policy CRDs + ManifestWork fan-out; OpenShift-centric, heavyweight, not a neutral primitive for arbitrary conformant clusters; no closed-verb typed actions with per-wave human gates | +| **Open Cluster Management (OCM)** | CNCF Sandbox; hub/spoke registration, Placement, ManifestWork, addons; `cluster-proxy` + `managed-serviceaccount` (both v0.10.0, 2026-02-02) give outbound-only reach + scoped identity *(repo-verified; M0 reproduced hands-on in [`docs/experiments/M0-ocm-falsification.md`](../experiments/M0-ocm-falsification.md))*; OCM v1.3.1 (2026-05-19) | OSS substrate, Red Hat-sponsored | **The critical finding (fetched, [ManifestWorkReplicaSet docs](https://open-cluster-management.io/docs/concepts/work-distribution/manifestworkreplicaset/)):** OCM already ships alpha **Progressive / ProgressivePerGroup rollout strategies** (minSuccessTime, progressDeadline, maxFailures) — so raw canary *sequencing* is NOT white space. What OCM has **no concept of**: approval gates, multi-approver workflows, typed/closed action vocabulary (ManifestWork = arbitrary YAML), operation-level audit ledger, abstention. Governance above OCM is the actual gap, and OCM's addon framework exists precisely so others build it ([CNCF comparison post](https://www.cncf.io/blog/2022/09/26/karmada-and-open-cluster-management-two-new-approaches-to-the-multicluster-fleet-management-challenge/)) | +| **Karmada** | CNCF; multi-cluster *scheduling/propagation*; push and pull modes (so "outbound agent" alone is weak differentiation) | OSS, Huawei-sponsored | Placement engine, not governed ops; same absence of approvals/typed verbs/audit | +| **Clusterpedia** | CNCF Sandbox; multi-cluster resource **search/inventory** (v0.9.1 *(repo-verified)*) | OSS | Read-only; no actions at all; validates fleet-search demand | +| **KubeSphere** | China-origin OSS platform (4.x "LuBan" architecture), first-class air-gap install ([docs](https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/air-gapped-installation/)) | Open core (QingCloud) | Platform breadth, China strength; not a neutral governed-action primitive; validates air-gap-first distribution in China | +| **Spectro Cloud Palette** | Full-stack cluster lifecycle via Cluster Profiles; VerteX/air-gap editions | Usage-based kilo-Core-hours, edge from ~$250/device/yr ([palette editions](https://www.spectrocloud.com/palette-editions)) | Proprietary, provisioning-centric; not an ops-action governance layer over existing clusters | +| **Devtron** | OSS K8s platform (CI/CD+GitOps+obs+security), v2.1.1 (2026-03-24) *(repo-verified)* | Open core | Batteries-included platform (the breadth strategy), not a narrow federation primitive | +| **Komodor** | SaaS K8s ops/AI-SRE; agent per cluster; Klaudia multi-agent (50+ specialized agents), MCP/OpenAPI extensibility, sandboxed + audited remediation ([press release, 2026-03-18](https://www.globenewswire.com/news-release/2026/03/18/3258257/0/en/komodor-introduces-extensible-autonomous-multi-agent-architecture-for-ai-driven-site-reliability-engineering.html)) | Node-based annual pricing ([pricing page](https://komodor.com/platform/pricing-and-plans/)); closed SaaS | The closest *product* to "operate the fleet with AI on top" — but closed, SaaS (China/air-gap excluded), diagnosis-first; no closed-vocabulary typed intents, no multi-approver canary waves as a neutral primitive | +| **Loft / vCluster** | Virtual clusters for hard multi-tenancy | Open core | Tenancy primitive, not fleet ops; complementary (and a GPU multi-tenancy voice: [vCluster on GPU sharing](https://www.vcluster.com/blog/ai-infrastructure-gpu-utilization-kubernetes-multitenancy)) | +| **kagent** | CNCF agent framework for K8s (v0.10.0-beta4 *(repo-verified)*) | OSS (Solo.io-originated) | Framework to *build* agents — a natural **MCP client of Sith**, not a governance layer | +| **Port / Cortex / Harness** | IDPs / delivery platforms (catalog, scorecards, self-service; Harness = CI/CD+FinOps suite) | Commercial | Different category (the predecessor's failed race). Boundary only: Sith is not a portal; portals can *consume* Sith's API/MCP | + +## 4. Observability / logging (integrate-only; the telemetry-lake trap) + +Prometheus/Grafana/Loki/Mimir, Elastic/OpenSearch/Kibana, Fluentd/Fluent Bit, Datadog, Splunk: each is a query/config surface for Sith to **read through** (per-tool mechanics and auth in [integrations-and-ai-governance.md](integrations-and-ai-governance.md)). The predecessor proved embedding/proxying them is negative value. The fleet-relevant gap none of them fills: they aggregate *telemetry*, not *operational state + actions* — none can answer "which clusters are running image X with a failing rollout and what changed there", and none dispatches governed actions. Datadog/Splunk cost pain is a recurring driver pushing teams toward self-hosted stacks (treated qualitatively here; no pricing figures were fetched in this pass — keep out of load-bearing claims). + +## 5. Cost (workstream F): the fleet rollup is the gap, not the metering + +Fetched evidence chain: + +- **OpenCost** is the CNCF-incubating standard (promoted from Sandbox **2024-10-31**, [CNCF announcement](https://www.cncf.io/blog/2024/10/31/opencost-advances-to-the-cncf-incubator/)) for **per-cluster** allocation: workload-granularity costs from Prometheus + cloud-billing integrations, exported as metrics; the announcement describes no fleet-level aggregation; plugins pull external SaaS costs (Datadog, OpenAI, MongoDB Atlas). +- **The DIY rollup pain is documented first-person by Grafana Labs**: OpenCost "is designed to expect its storage having the same scope as its deployment" — so they run OpenCost per cluster, ship metrics to central Mimir, and had to insert `prom-label-proxy` so each instance sees only its own data; they list native multi-cluster support and multi-cluster query docs as missing ([Grafana Labs on OpenCost](https://grafana.com/blog/2023/02/02/how-grafana-labs-uses-and-contributes-to-opencost-the-open-source-project-for-real-time-cost-monitoring-in-kubernetes/)). A user request for multi-cluster/multi-cloud aggregation was triaged P3 and closed unresolved ([opencost#2638](https://github.com/opencost/opencost/issues/2638)). +- **Kubecost** (the commercial layer; **acquired by IBM, announced 2024-09-17**, [IBM newsroom](https://newsroom.ibm.com/blog-ibm-acquires-kubecost-to-broaden-hybrid-cloud-cost-management-capabilities), folding into the Apptio/Cloudability/Turbonomic FinOps suite): free tier is capped (250 cores, 15-day retention; 3.0's licensing gate ≈ $100k spend/trailing-30-days — [Kubecost 3.0 announcement](https://www.apptio.com/blog/ibm-kubecost-3-0-faster-smarter-and-built-for-scale/)); **"Unified, multi-cluster view" is an Enterprise-tier feature** ([nOps pricing teardown](https://www.nops.io/blog/kubecost-pricing/) — competitor-authored; its $70k–100k/3-cluster anecdote is unverified, treat as directional). 3.0 also moves to a unified IBM agent "eliminating the dependency on Prometheus" — i.e., the leading commercial tool is consolidating into a proprietary suite, not staying a neutral primitive. +- **CAST AI** is automation-first (rightsizing/spot/GPU optimization that *mutates* clusters), quote-based pricing only, and its own intake form lists just EKS/GKE/AKS/OpenShift-on-AWS — **no China clouds, no generic on-prem/air-gap** ([cast.ai/pricing](https://cast.ai/pricing/), fetched 2026-07-08; $108M Series C at ~$900M valuation, [TechCrunch 2025-04-30](https://techcrunch.com/2025/04/30/cast-ai-raises-108m-to-get-the-max-out-of-ai-kubernetes-and-other-workloads/)). +- **GPU:** Kubecost 2.4 does DCGM-based GPU efficiency/idle-cost per container ([Apptio GPU monitoring](https://www.apptio.com/blog/gpu-monitoring/)); MIG/fractional attribution remains beyond it (DCGM reports at physical-GPU level — [vCluster analysis](https://www.vcluster.com/blog/ai-infrastructure-gpu-utilization-kubernetes-multitenancy)); OpenCost GPU pricing has had correctness bugs ([opencost#2029](https://github.com/opencost/opencost/issues/2029)); the FinOps WG GPU paper is single-cluster-only ([FinOps WG](https://www.finops.org/wg/scaling-kubernetes-for-ai-ml-workloads-with-finops/)); Kubecost 3.0's GPU story is one line ("recommendations are GPU-aware"). + +**Verdict for Sith:** cost is a **read-overlay integration, not a wedge and not a build**. The move: deploy/read OpenCost (or its metrics) per cluster via the existing read federation, aggregate at the hub into per-workspace/per-team fleet rollups (GPU columns included where DCGM exists), stamp freshness like every other fleet fact. That lands precisely in the documented OSS gap (per-cluster standard exists; the free fleet view doesn't — it's paywalled at Kubecost Enterprise and unsolved in OpenCost). Building a metering/billing engine or automation-optimizer would re-fight OpenCost/Kubecost/CAST on their ground and violate the telemetry-lake non-goal. + +## 6. Multi-cloud + China clouds (workstream G) + +- **Conformance (hand-verified in [cncf/k8s-conformance](https://github.com/cncf/k8s-conformance)):** Alibaba ACK (`v1.32/alicloud`, "Alibaba Cloud Container Service for Kubernetes v1.32.0"), Huawei CCE (submissions every version v1.29–v1.34), Tencent TKE (`v1.34/tencentcloud`, v1.34.1). The Kubernetes API surface Sith depends on is uniform across US and China clouds. +- **Auth/enumeration is where clouds differ.** EKS: `aws eks list-clusters` + access entries + `aws eks get-token` exec plugin. AKS: `az aks list` + Entra ID + kubelogin. GKE: `gcloud container clusters list` + IAM + `gke-gcloud-auth-plugin` (+ Connect gateway for fleet reach). ACK: kubeconfigs per RAM user/role via `DescribeClusterUserKubeconfig`, configurable expiry/revocation, VPC-internal endpoints ([ACK docs](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/obtain-the-kubeconfig-file-of-a-cluster-and-use-kubectl-to-connect-to-the-cluster)). CCE: IAM for cloud perms + standard kubeconfig for cluster RBAC, intranet/public endpoints ([CCE docs](https://support.huaweicloud.com/intl/en-us/usermanual-cce/cce_10_0187.html)). TKE: separate public/private kubeconfigs, CAM-integrated ([TKE docs](https://www.tencentcloud.com/document/product/457/30639)). +- **OpenShift:** conformant API + `oc`, Routes, SecurityContextConstraints; ACM occupies the fleet layer natively. Treat OpenShift as: conformant-API coverage guaranteed; deep OpenShift-isms (Routes/SCC-aware views) explicitly later; never compete for ACM-committed estates. +- **Recommended abstraction:** exactly two layers. (1) **Pure Kubernetes API** for everything inside a cluster — guaranteed by conformance, including China. (2) **Thin per-cloud adapters** for *enumeration and credential minting only* (list clusters; mint short-lived tokens via each cloud's mechanism; exec-plugin passthrough in local mode). Anything deeper per-cloud (node pools, cloud LBs, billing) is out of scope — that's the cloud console's job. The federated mode needs no cloud adapter at all: the minion dials out, so a TKE cluster behind restricted egress registers exactly like an EKS one (Rancher's ACK/CCE/TKE support proves the pattern; OCM's pull architecture is the same shape). + +## 7. Air-gap distribution prior art + +**Zarf** ([zarf-dev/zarf](https://github.com/zarf-dev/zarf), Apache-2.0, v0.80.0 2026-06-25, ~2k stars, Naval Postgraduate School lineage) owns air-gap *packaging*: single-file bundles of images/charts/manifests, embedded registry/Gitea, image-path rewriting. It is a delivery vehicle, not a control plane — **complement, not competitor**: Sith should ship an official Zarf package (or Zarf-style single artifact) as its China/regulated-market distribution, rather than building bespoke offline tooling. + +## 8. The white space, stated precisely + +Across every tool above, four candidate positions were tested against the evidence: + +1. **Governed action federation as a neutral OSS primitive — EMPTY.** OCM has rollout mechanics but no approvals/typed verbs/audit; ACM/Rancher have platform-coupled policy; Kargo gates artifact promotion only; Komodor audits its own AI's remediation inside a closed SaaS. Nobody ships "typed intent + multi-approver + per-wave gates + abstention + signed dispatch + decision ledger" vendor-neutrally. This is Sith's wedge, confirmed. +2. **Fleet-wide live correlation/search — PARTIALLY OCCUPIED.** Clusterpedia (search) and every platform's inventory exist; correlation joined with actions and staleness semantics does not. Differentiating as part of the wedge, not alone. +3. **Fleet cost read-overlay — EMPTY IN OSS, PAYWALLED COMMERCIALLY.** (§5.) High-value fast-follow on top of read federation. +4. **OSS aggregated multi-cluster local client — EMPTY.** (§1, form-factor file.) The adoption on-ramp. + +And one distribution property that cuts across all four: **air-gap-first, no-phone-home, multi-arch, registry-relocatable** — mandatory for China/regulated (see [Part 1](#part-1)), cheap if designed in from day one, and none of the SaaS incumbents can follow there. + + +---