Skip to content

Latest commit

 

History

History
311 lines (228 loc) · 19.2 KB

architecture.md

File metadata and controls

311 lines (228 loc) · 19.2 KB

Architecture

This document explains how cainjekt works at a technical level — the three-stage injection pipeline, the processor system, and the design decisions behind them.

The Problem

Enterprise Kubernetes clusters often use custom Certificate Authorities (CAs) — for TLS inspection proxies, internal PKI, or self-signed services. Every container that makes HTTPS calls needs to trust these CAs. Without cainjekt, teams have two bad options:

  1. Bake CAs into every container image — breaks the separation between application and infrastructure, requires rebuilding images when CAs rotate, and doesn't work for third-party images.
  2. Mount CA bundles via volumes — requires every pod spec to include volume mounts, which is error-prone and doesn't update the OS trust store (only puts a file on disk).

cainjekt solves this by injecting CAs transparently at the container runtime level — no image changes, no pod spec changes beyond a single annotation.

How It Works

cainjekt uses containerd's Node Resource Interface (NRI) to intercept container lifecycle events. It runs as a DaemonSet on every node and operates in three phases:

┌──────────────────────────────────────────────────────────────────┐
│ Node                                                             │
│                                                                  │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────────┐     │
│  │ NRI Plugin  │────▶│ OCI Hook    │────▶│ Wrapper         │     │
│  │ (DaemonSet) │     │ (host ctx)  │     │ (container PID1)│     │
│  └─────────────┘     └─────────────┘     └─────────────────┘     │
│                                                                  │
│  CreateContainer      CreateRuntime       Container Start        │
│  event from           hook callback       entrypoint exec        │
│  containerd           before container    replaces wrapper       │
│                       process starts      with real command      │
└──────────────────────────────────────────────────────────────────┘

Phase 1: NRI Plugin (CreateContainer)

When containerd creates a container, the NRI plugin receives a CreateContainer event. If the pod has the opt-in annotation (cainjekt.natron.io/enabled: "true"), the plugin:

  1. Reads the CA bundle from the ConfigMap-mounted file (/etc/cainjekt/ca-bundle.pem)
  2. Validates the PEM — rejects bundles with no valid certificates
  3. Stages a per-container copy at /run/cainjekt/containers/<container-id>/ca-bundle.pem (so each container gets its own copy, isolated from others)
  4. Injects an OCI CreateRuntime hook pointing to the cainjekt binary with env vars for hook mode
  5. Prepends the wrapper binary to the container's entrypoint args ([/cainjekt-entrypoint, original-cmd, ...])
  6. Bind-mounts the cainjekt binary into the container as /cainjekt-entrypoint
  7. Sets env vars in the container: CAINJEKT_WRAPPER_MODE=1, CAINJEKT_HOOK_CONTEXT_FILE=/etc/cainjekt/hook-context.json

All of this happens via NRI container adjustments — the container spec is modified before the container starts.

Phase 2: OCI Hook (CreateRuntime)

The CreateRuntime hook runs on the host after the container's root filesystem is mounted but before the container process starts. This is the phase that modifies the container's filesystem:

  1. Reads OCI state from stdin (container ID, bundle path, annotations)
  2. Resolves the container rootfs from the OCI spec
  3. Runs the processor pipeline in priority order:
    • OS store processors (priority 275-300): Detect the distro via /etc/os-release, merge the CA into the system trust store (e.g., /etc/ssl/certs/ca-certificates.crt on Debian)
    • Language processors (priority 100): Detect language runtimes, but do nothing in the hook phase (they act in the wrapper phase)
    • Fallback processor (priority -100): Tries common trust store paths if no distro matched
  4. Detects read-only rootfs — if the filesystem is not writable, skips trust store modification and sets a fact so language processors can use the dynamic CA path instead
  5. Writes individual CA file to the container's anchor directory (e.g., /usr/local/share/ca-certificates/cainjekt.crt)
  6. Persists hook context to /etc/cainjekt/hook-context.json — which processors were detected, which facts were collected (distro, trust store path, individual CA path)
  7. Writes status file to /etc/cainjekt/status.json — human-readable injection status for operator inspection

Phase 3: Wrapper (Container Entrypoint)

The wrapper runs as the container's first process (PID 1). It:

  1. Reads the persisted hook context from /etc/cainjekt/hook-context.json
  2. Runs wrapper processors for each detected language runtime:
    • lang-dotnet: sets SSL_CERT_FILE
    • lang-go: sets SSL_CERT_FILE
    • lang-java: nothing, unless the hook could not patch cacerts in place — then JAVA_TOOL_OPTIONS points at the staged keystore
    • lang-nodejs: sets NODE_EXTRA_CA_CERTS
    • lang-python: sets SSL_CERT_FILE and REQUESTS_CA_BUNDLE
    • lang-ruby: sets SSL_CERT_FILE
  3. Applies environment variables to the process
  4. Execs the original command via syscall.Exec() — the wrapper process is replaced by the real entrypoint, maintaining PID 1

The wrapper is fail-open: if any step fails (reading context, running processors, resolving the command), it logs a warning and execs the original command anyway. The container always starts.

Processor System

Processors are the extensible core of cainjekt. Each processor implements a simple interface:

type Processor interface {
    Name() string
    Category() string
    Detect(*Context) DetectResult
    Apply(*Context) error
}

type WrapperProcessor interface {
    Processor
    ApplyWrapper(*Context) error
}

OS Store Processors

These run in the hook phase and modify the container's filesystem:

Processor Distro Match Trust Store Path Priority
os-debian debian, ubuntu /etc/ssl/certs/ca-certificates.crt 300
os-rhel rhel, fedora, centos, rocky, almalinux, ol, amzn /etc/pki/tls/certs/ca-bundle.crt 290
os-opensuse opensuse, sles, suse /etc/ssl/ca-bundle.pem 285
os-alpine alpine /etc/ssl/certs/ca-certificates.crt 280
os-arch arch /etc/ssl/certs/ca-certificates.crt 275
os-fallback (any) tries common paths -100

Detection works by reading /etc/os-release in the container rootfs and matching the ID and ID_LIKE fields.

Language Processors

These run in the wrapper phase and set environment variables:

Processor Detection Env Vars Set
lang-dotnet /usr/share/dotnet/dotnet SSL_CERT_FILE
lang-go /usr/local/go/bin/go SSL_CERT_FILE
lang-java /usr/bin/java, $JAVA_HOME, */lib/security/cacerts none normally; JAVA_TOOL_OPTIONS (-Djavax.net.ssl.trustStore, -Djavax.net.ssl.trustStoreType) when cacerts is not writable
lang-nodejs /usr/bin/node NODE_EXTRA_CA_CERTS
lang-python /usr/bin/python3 SSL_CERT_FILE, REQUESTS_CA_BUNDLE
lang-ruby /usr/bin/ruby SSL_CERT_FILE

Detection checks for the binary at multiple common paths in the container rootfs.

Processor Filtering

Pods can control which processors run via annotations:

annotations:
  cainjekt.natron.io/processors.include: "os-debian,lang-python"
  cainjekt.natron.io/processors.exclude: "os-fallback"

Security Model

Threat Model

This section is the definition of "secure" for cainjekt. A security review checks a change against it; a finding it does not cover means the model is updated first, then the code.

Attacker. Anyone who can create a pod on an injected node. They control the image completely (every file, symlink, FIFO and device node in the rootfs), the pod annotations, and the container's command and environment. They cannot change the plugin configuration, the CA bundle ConfigMap or the node.

Where untrusted input meets privilege.

Component Runs as Untrusted input
NRI plugin uid 0, no capabilities, DaemonSet pod pod annotations and labels, namespace labels
OCI hook root on the host, outside the container's device cgroup the container rootfs, OCI state annotations, spec env (JAVA_HOME)
Wrapper the container's user, inside the container hook-context.json (the container can write it)

The OCI hook is the high-risk part: root on the node, reading a filesystem the attacker built.

Invariants. Each has one choke point and a test, so it cannot be broken by accident in a new code path.

Invariant Enforced by Tested by
The hook never reads or writes outside the rootfs, except the per-container staging dir containerfs resolves every path with links re-anchored inside the rootfs; Context.StageFile for the staging dir. The forbidigo lint bans raw os/filepath/fsx file calls in engine/processors, runtime/hook and runtime/hookctx TestHostileRootfs (every registered processor), FuzzResolveSymlinks
The hook reads only regular files of at most 16 MiB from the rootfs (no FIFOs, device nodes, unbounded reads) containerfs.ReadRegularFile TestHostileRootfs, TestReadRegularFile*
cainjekt never blocks a container from starting fail-open processors and wrapper; hook timeout TestHostileRootfs (no hang), E2E
Annotations only select processors and containers; they never become paths, commands or env values review —
The plugin pod has no privilege beyond uid 0 for the NRI socket and /run/cainjekt chart securityContext, RBAC limited to get namespaces E2E Helm test runs the defaults

Assumptions. No process runs in the container while the CreateRuntime hook runs, so the rootfs cannot change between resolving a path and using it (which is why the hook does not need openat-style resolution everywhere). containerd, the node and the CA bundle ConfigMap are trusted.

When a change needs a security review. When it adds or changes a hook-phase file access or a processor, reads a new annotation or env var, adds a host path, or changes the chart's securityContext, volumes or RBAC. Other changes do not.

Privileges

The plugin container runs as root (uid 0) but is not privileged: all capabilities are dropped, the root filesystem is read-only, privilege escalation is disabled and the default seccomp profile applies. Root is needed only because:

  • NRI socket access: The NRI socket at /var/run/nri is owned by root
  • Per-container CA staging: The plugin writes /run/cainjekt/containers/<id>/ on the host, which is root-owned

The part that modifies container filesystems — the OCI hook — is not run in this pod. containerd runs it on the host as root, as it does all OCI hooks, so the pod needs no capabilities and no access to the host root filesystem. The plugin does not call the Kubernetes nodes API; RBAC grants only get on namespaces (for the namespace-label opt-in).

The pod runs in the SELinux type spc_t. Under SELinux enforcing (RKE2 with selinux: true, SLE Micro, RHEL), the default container_t may not connect to the NRI socket or write /run/cainjekt and /opt/cainjekt, which carry host labels. spc_t lifts only the SELinux confinement; capabilities stay dropped. Nodes without SELinux ignore it. If a node still denies access, set securityContext: {privileged: true} in the Helm values.

hostNetwork and hostPID are not required — the plugin communicates via Unix socket (hostPath volume), not the network. They are disabled by default in the Helm chart but can be enabled if needed.

Nodes Without NRI

Some managed Kubernetes nodes don't have NRI enabled in their containerd configuration — for example, AKS GPU nodes use a custom containerd config for the NVIDIA runtime that may not include the NRI plugin. When the NRI socket (/var/run/nri/nri.sock) is not available, the plugin exits with a clear error message. The DaemonSet will crash-loop on that node, which is intentional — CrashLoopBackOff is the most visible signal to platform teams and existing monitoring will catch it. The fix is to either enable NRI on the node or exclude it via nodeSelector/affinity.

When the connection to containerd drops (containerd restart, RKE2 upgrade), the plugin reconnects in-process with a backoff of 100 ms to 2 s and is not Ready while disconnected. It exits only after 2 minutes without a connection, so a node without NRI still ends up in CrashLoopBackOff.

The plugin reaches the API server through KUBERNETES_SERVICE_HOST/KUBERNETES_SERVICE_PORT, not cluster DNS, so namespace-label opt-in also works with hostNetwork: true.

Safety Guarantees

  • Opt-in only: Containers are never modified unless the pod has cainjekt.natron.io/enabled: "true"
  • Atomic writes: All file operations use temp file + rename for crash safety
  • Symlink protection: Refuses to overwrite symlinks in trust stores (prevents symlink attacks)
  • Regular files only: The hook runs as root on the node, outside the container's device cgroup, so it reads image files (trust stores, os-release, cacerts) only if they are regular files of at most 16 MiB. A FIFO or device node placed by the image is refused rather than read
  • Fail-open: Hook and wrapper failures never block container startup
  • Per-container isolation: Each container gets its own staged CA file; containers can't interfere with each other
  • No network calls: The plugin operates entirely on local filesystem; no external communication

CA Bundle Validation

The CA bundle is validated at staging time: certs.ValidatePEM() checks that the file contains at least one parseable X.509 certificate in PEM format. Invalid bundles are rejected with a clear error.

Read-Only Root Filesystem Handling

When a container has a read-only root filesystem:

  1. The OS store processor detects the read-only rootfs by probing writability
  2. Trust store modification is skipped (no error)
  3. The FactRootfsReadOnly fact is set
  4. The FactIndividualCAPath is set to the dynamic CA file path (host-mounted, always readable)
  5. Language processors use this path for env vars — they work regardless of rootfs writability
  6. A warning is logged in the hook output

This means: OS-level tools (curl, wget) won't trust the CA on read-only rootfs, but language runtimes (Java, Node.js, Python, Ruby, Go) will, because they use the env var pointing to the dynamic CA file.

Observability

Prometheus Metrics

The NRI plugin exposes metrics on :9443/metrics:

Metric Type Description
cainjekt_injections_total Counter Total injection attempts
cainjekt_injections_errors_total Counter Injection errors
cainjekt_skipped_total Counter Containers without opt-in annotation
cainjekt_active_containers Gauge Currently tracked injected containers
cainjekt_cleanups_total Counter Dynamic CA cleanups on removal
cainjekt_cleanups_errors_total Counter Cleanup errors
cainjekt_orphans_cleaned_total Counter Orphaned CA dirs cleaned
cainjekt_processor_detected_total{processor} Counter Per-processor detection count
cainjekt_processor_applied_total{processor} Counter Per-processor application count
cainjekt_hook_incomplete_total Counter OCI hooks that did not finish (SIGKILLed on timeout). Alert on sustained increase: rate(cainjekt_hook_incomplete_total[5m]) > 0 — usually means CAINJEKT_HOOK_TIMEOUT_SEC is too low.
cainjekt_ns_lookup_errors_total Counter Namespace label lookups that failed (expired service account token, API unreachable). Alert on any increase: rate(cainjekt_ns_lookup_errors_total[5m]) > 0 — namespace-label opt-in pods are being silently skipped.
cainjekt_ns_lookup_skipped_total Counter Containers skipped because the namespace lookup failed, so opt-in could not be determined. Unlike cainjekt_skipped_total these may well have been opted in and got no CA.
cainjekt_missed_containers Gauge Opted-in containers already running when the plugin connected that carry no hook.done breadcrumb — created during a gap in coverage and never injected. Set by the startup scan and decremented as each one is removed, so it reads as outstanding work, not a historical total. Cannot be fixed in place: the pods need a restart.

Plus standard Go runtime and process metrics via prometheus/client_golang.

Namespace-label opt-in depends on the service account token. The plugin re-reads the projected token from disk (the kubelet rotates it before expiry), so it survives the short-lived bound tokens that some clusters (e.g. AKS) issue. If cainjekt_ns_lookup_errors_total climbs, the API lookup is failing and namespace-label opt-in stops working until it recovers — pod-annotation and pod-label opt-in are unaffected because they need no API call.

Status File

Every injected container has /etc/cainjekt/status.json:

{
  "injected": true,
  "timestamp": "2026-04-15T08:27:15Z",
  "distro": "debian",
  "trust_store": "/etc/ssl/certs/ca-certificates.crt",
  "ca_file": "/run/cainjekt/containers/.../ca-bundle.pem",
  "processors": [
    { "name": "os-debian", "category": "os", "applicable": true },
    { "name": "lang-python", "category": "language", "applicable": true },
    ...
  ]
}

Logging

Structured logging via log/slog. Configurable via CAINJEKT_LOG_LEVEL (debug/info/warn/error).

Key log points:

  • Hook start: processor count, rootfs path, CA file
  • Hook end: applied/skipped/failed counts, distro, trust store path
  • Hook warning: read-only rootfs detected
  • Wrapper: each applied processor, exec command
  • Plugin: container create/remove events, injection decisions

Lifecycle Management

Orphan Cleanup

If the plugin crashes or a container is removed without triggering RemoveContainer, the per-container CA directory at /run/cainjekt/containers/<id>/ becomes orphaned. A background goroutine sweeps every 5 minutes and removes directories that:

  • Are not tracked by any currently active container
  • Are older than 10 minutes

Graceful Shutdown

On SIGTERM/SIGINT:

  1. Stop accepting new NRI events
  2. Stop the orphan cleaner
  3. Gracefully shut down the HTTP server
  4. Exit cleanly

ConfigMap Hot-Reload

The CA bundle ConfigMap is read fresh on every CreateContainer call. When the ConfigMap is updated, Kubernetes refreshes the mounted file (via symlink swap). New containers automatically get the updated CA — no DaemonSet restart needed. Existing containers keep their staged CA.