This document explains how cainjekt works at a technical level — the three-stage injection pipeline, the processor system, and the design decisions behind them.
Enterprise Kubernetes clusters often use custom Certificate Authorities (CAs) — for TLS inspection proxies, internal PKI, or self-signed services. Every container that makes HTTPS calls needs to trust these CAs. Without cainjekt, teams have two bad options:
- Bake CAs into every container image — breaks the separation between application and infrastructure, requires rebuilding images when CAs rotate, and doesn't work for third-party images.
- Mount CA bundles via volumes — requires every pod spec to include volume mounts, which is error-prone and doesn't update the OS trust store (only puts a file on disk).
cainjekt solves this by injecting CAs transparently at the container runtime level — no image changes, no pod spec changes beyond a single annotation.
cainjekt uses containerd's Node Resource Interface (NRI) to intercept container lifecycle events. It runs as a DaemonSet on every node and operates in three phases:
┌──────────────────────────────────────────────────────────────────┐
│ Node │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
│ │ NRI Plugin │────▶│ OCI Hook │────▶│ Wrapper │ │
│ │ (DaemonSet) │ │ (host ctx) │ │ (container PID1)│ │
│ └─────────────┘ └─────────────┘ └─────────────────┘ │
│ │
│ CreateContainer CreateRuntime Container Start │
│ event from hook callback entrypoint exec │
│ containerd before container replaces wrapper │
│ process starts with real command │
└──────────────────────────────────────────────────────────────────┘
When containerd creates a container, the NRI plugin receives a CreateContainer event. If the pod has the opt-in annotation (cainjekt.natron.io/enabled: "true"), the plugin:
- Reads the CA bundle from the ConfigMap-mounted file (
/etc/cainjekt/ca-bundle.pem) - Validates the PEM — rejects bundles with no valid certificates
- Stages a per-container copy at
/run/cainjekt/containers/<container-id>/ca-bundle.pem(so each container gets its own copy, isolated from others) - Injects an OCI CreateRuntime hook pointing to the cainjekt binary with env vars for hook mode
- Prepends the wrapper binary to the container's entrypoint args (
[/cainjekt-entrypoint, original-cmd, ...]) - Bind-mounts the cainjekt binary into the container as
/cainjekt-entrypoint - Sets env vars in the container:
CAINJEKT_WRAPPER_MODE=1,CAINJEKT_HOOK_CONTEXT_FILE=/etc/cainjekt/hook-context.json
All of this happens via NRI container adjustments — the container spec is modified before the container starts.
The CreateRuntime hook runs on the host after the container's root filesystem is mounted but before the container process starts. This is the phase that modifies the container's filesystem:
- Reads OCI state from stdin (container ID, bundle path, annotations)
- Resolves the container rootfs from the OCI spec
- Runs the processor pipeline in priority order:
- OS store processors (priority 275-300): Detect the distro via
/etc/os-release, merge the CA into the system trust store (e.g.,/etc/ssl/certs/ca-certificates.crton Debian) - Language processors (priority 100): Detect language runtimes, but do nothing in the hook phase (they act in the wrapper phase)
- Fallback processor (priority -100): Tries common trust store paths if no distro matched
- OS store processors (priority 275-300): Detect the distro via
- Detects read-only rootfs — if the filesystem is not writable, skips trust store modification and sets a fact so language processors can use the dynamic CA path instead
- Writes individual CA file to the container's anchor directory (e.g.,
/usr/local/share/ca-certificates/cainjekt.crt) - Persists hook context to
/etc/cainjekt/hook-context.json— which processors were detected, which facts were collected (distro, trust store path, individual CA path) - Writes status file to
/etc/cainjekt/status.json— human-readable injection status for operator inspection
The wrapper runs as the container's first process (PID 1). It:
- Reads the persisted hook context from
/etc/cainjekt/hook-context.json - Runs wrapper processors for each detected language runtime:
lang-dotnet: setsSSL_CERT_FILElang-go: setsSSL_CERT_FILElang-java: nothing, unless the hook could not patchcacertsin place — thenJAVA_TOOL_OPTIONSpoints at the staged keystorelang-nodejs: setsNODE_EXTRA_CA_CERTSlang-python: setsSSL_CERT_FILEandREQUESTS_CA_BUNDLElang-ruby: setsSSL_CERT_FILE
- Applies environment variables to the process
- Execs the original command via
syscall.Exec()— the wrapper process is replaced by the real entrypoint, maintaining PID 1
The wrapper is fail-open: if any step fails (reading context, running processors, resolving the command), it logs a warning and execs the original command anyway. The container always starts.
Processors are the extensible core of cainjekt. Each processor implements a simple interface:
type Processor interface {
Name() string
Category() string
Detect(*Context) DetectResult
Apply(*Context) error
}
type WrapperProcessor interface {
Processor
ApplyWrapper(*Context) error
}These run in the hook phase and modify the container's filesystem:
| Processor | Distro Match | Trust Store Path | Priority |
|---|---|---|---|
os-debian |
debian, ubuntu | /etc/ssl/certs/ca-certificates.crt |
300 |
os-rhel |
rhel, fedora, centos, rocky, almalinux, ol, amzn | /etc/pki/tls/certs/ca-bundle.crt |
290 |
os-opensuse |
opensuse, sles, suse | /etc/ssl/ca-bundle.pem |
285 |
os-alpine |
alpine | /etc/ssl/certs/ca-certificates.crt |
280 |
os-arch |
arch | /etc/ssl/certs/ca-certificates.crt |
275 |
os-fallback |
(any) | tries common paths | -100 |
Detection works by reading /etc/os-release in the container rootfs and matching the ID and ID_LIKE fields.
These run in the wrapper phase and set environment variables:
| Processor | Detection | Env Vars Set |
|---|---|---|
lang-dotnet |
/usr/share/dotnet/dotnet |
SSL_CERT_FILE |
lang-go |
/usr/local/go/bin/go |
SSL_CERT_FILE |
lang-java |
/usr/bin/java, $JAVA_HOME, */lib/security/cacerts |
none normally; JAVA_TOOL_OPTIONS (-Djavax.net.ssl.trustStore, -Djavax.net.ssl.trustStoreType) when cacerts is not writable |
lang-nodejs |
/usr/bin/node |
NODE_EXTRA_CA_CERTS |
lang-python |
/usr/bin/python3 |
SSL_CERT_FILE, REQUESTS_CA_BUNDLE |
lang-ruby |
/usr/bin/ruby |
SSL_CERT_FILE |
Detection checks for the binary at multiple common paths in the container rootfs.
Pods can control which processors run via annotations:
annotations:
cainjekt.natron.io/processors.include: "os-debian,lang-python"
cainjekt.natron.io/processors.exclude: "os-fallback"This section is the definition of "secure" for cainjekt. A security review checks a change against it; a finding it does not cover means the model is updated first, then the code.
Attacker. Anyone who can create a pod on an injected node. They control the image completely (every file, symlink, FIFO and device node in the rootfs), the pod annotations, and the container's command and environment. They cannot change the plugin configuration, the CA bundle ConfigMap or the node.
Where untrusted input meets privilege.
| Component | Runs as | Untrusted input |
|---|---|---|
| NRI plugin | uid 0, no capabilities, DaemonSet pod | pod annotations and labels, namespace labels |
| OCI hook | root on the host, outside the container's device cgroup | the container rootfs, OCI state annotations, spec env (JAVA_HOME) |
| Wrapper | the container's user, inside the container | hook-context.json (the container can write it) |
The OCI hook is the high-risk part: root on the node, reading a filesystem the attacker built.
Invariants. Each has one choke point and a test, so it cannot be broken by accident in a new code path.
| Invariant | Enforced by | Tested by |
|---|---|---|
| The hook never reads or writes outside the rootfs, except the per-container staging dir | containerfs resolves every path with links re-anchored inside the rootfs; Context.StageFile for the staging dir. The forbidigo lint bans raw os/filepath/fsx file calls in engine/processors, runtime/hook and runtime/hookctx |
TestHostileRootfs (every registered processor), FuzzResolveSymlinks |
| The hook reads only regular files of at most 16 MiB from the rootfs (no FIFOs, device nodes, unbounded reads) | containerfs.ReadRegularFile |
TestHostileRootfs, TestReadRegularFile* |
| cainjekt never blocks a container from starting | fail-open processors and wrapper; hook timeout | TestHostileRootfs (no hang), E2E |
| Annotations only select processors and containers; they never become paths, commands or env values | review | — |
The plugin pod has no privilege beyond uid 0 for the NRI socket and /run/cainjekt |
chart securityContext, RBAC limited to get namespaces |
E2E Helm test runs the defaults |
Assumptions. No process runs in the container while the CreateRuntime hook
runs, so the rootfs cannot change between resolving a path and using it (which
is why the hook does not need openat-style resolution everywhere). containerd,
the node and the CA bundle ConfigMap are trusted.
When a change needs a security review. When it adds or changes a hook-phase
file access or a processor, reads a new annotation or env var, adds a host
path, or changes the chart's securityContext, volumes or RBAC. Other changes
do not.
The plugin container runs as root (uid 0) but is not privileged: all capabilities are dropped, the root filesystem is read-only, privilege escalation is disabled and the default seccomp profile applies. Root is needed only because:
- NRI socket access: The NRI socket at
/var/run/nriis owned by root - Per-container CA staging: The plugin writes
/run/cainjekt/containers/<id>/on the host, which is root-owned
The part that modifies container filesystems — the OCI hook — is not run in this pod. containerd runs it on the host as root, as it does all OCI hooks, so the pod needs no capabilities and no access to the host root filesystem. The plugin does not call the Kubernetes nodes API; RBAC grants only get on namespaces (for the namespace-label opt-in).
The pod runs in the SELinux type spc_t. Under SELinux enforcing (RKE2 with selinux: true, SLE Micro, RHEL), the default container_t may not connect to the NRI socket or write /run/cainjekt and /opt/cainjekt, which carry host labels. spc_t lifts only the SELinux confinement; capabilities stay dropped. Nodes without SELinux ignore it. If a node still denies access, set securityContext: {privileged: true} in the Helm values.
hostNetwork and hostPID are not required — the plugin communicates via Unix socket (hostPath volume), not the network. They are disabled by default in the Helm chart but can be enabled if needed.
Some managed Kubernetes nodes don't have NRI enabled in their containerd configuration — for example, AKS GPU nodes use a custom containerd config for the NVIDIA runtime that may not include the NRI plugin. When the NRI socket (/var/run/nri/nri.sock) is not available, the plugin exits with a clear error message. The DaemonSet will crash-loop on that node, which is intentional — CrashLoopBackOff is the most visible signal to platform teams and existing monitoring will catch it. The fix is to either enable NRI on the node or exclude it via nodeSelector/affinity.
When the connection to containerd drops (containerd restart, RKE2 upgrade), the plugin reconnects in-process with a backoff of 100 ms to 2 s and is not Ready while disconnected. It exits only after 2 minutes without a connection, so a node without NRI still ends up in CrashLoopBackOff.
The plugin reaches the API server through KUBERNETES_SERVICE_HOST/KUBERNETES_SERVICE_PORT, not cluster DNS, so namespace-label opt-in also works with hostNetwork: true.
- Opt-in only: Containers are never modified unless the pod has
cainjekt.natron.io/enabled: "true" - Atomic writes: All file operations use temp file + rename for crash safety
- Symlink protection: Refuses to overwrite symlinks in trust stores (prevents symlink attacks)
- Regular files only: The hook runs as root on the node, outside the container's device cgroup, so it reads image files (trust stores,
os-release,cacerts) only if they are regular files of at most 16 MiB. A FIFO or device node placed by the image is refused rather than read - Fail-open: Hook and wrapper failures never block container startup
- Per-container isolation: Each container gets its own staged CA file; containers can't interfere with each other
- No network calls: The plugin operates entirely on local filesystem; no external communication
The CA bundle is validated at staging time: certs.ValidatePEM() checks that the file contains at least one parseable X.509 certificate in PEM format. Invalid bundles are rejected with a clear error.
When a container has a read-only root filesystem:
- The OS store processor detects the read-only rootfs by probing writability
- Trust store modification is skipped (no error)
- The
FactRootfsReadOnlyfact is set - The
FactIndividualCAPathis set to the dynamic CA file path (host-mounted, always readable) - Language processors use this path for env vars — they work regardless of rootfs writability
- A warning is logged in the hook output
This means: OS-level tools (curl, wget) won't trust the CA on read-only rootfs, but language runtimes (Java, Node.js, Python, Ruby, Go) will, because they use the env var pointing to the dynamic CA file.
The NRI plugin exposes metrics on :9443/metrics:
| Metric | Type | Description |
|---|---|---|
cainjekt_injections_total |
Counter | Total injection attempts |
cainjekt_injections_errors_total |
Counter | Injection errors |
cainjekt_skipped_total |
Counter | Containers without opt-in annotation |
cainjekt_active_containers |
Gauge | Currently tracked injected containers |
cainjekt_cleanups_total |
Counter | Dynamic CA cleanups on removal |
cainjekt_cleanups_errors_total |
Counter | Cleanup errors |
cainjekt_orphans_cleaned_total |
Counter | Orphaned CA dirs cleaned |
cainjekt_processor_detected_total{processor} |
Counter | Per-processor detection count |
cainjekt_processor_applied_total{processor} |
Counter | Per-processor application count |
cainjekt_hook_incomplete_total |
Counter | OCI hooks that did not finish (SIGKILLed on timeout). Alert on sustained increase: rate(cainjekt_hook_incomplete_total[5m]) > 0 — usually means CAINJEKT_HOOK_TIMEOUT_SEC is too low. |
cainjekt_ns_lookup_errors_total |
Counter | Namespace label lookups that failed (expired service account token, API unreachable). Alert on any increase: rate(cainjekt_ns_lookup_errors_total[5m]) > 0 — namespace-label opt-in pods are being silently skipped. |
cainjekt_ns_lookup_skipped_total |
Counter | Containers skipped because the namespace lookup failed, so opt-in could not be determined. Unlike cainjekt_skipped_total these may well have been opted in and got no CA. |
cainjekt_missed_containers |
Gauge | Opted-in containers already running when the plugin connected that carry no hook.done breadcrumb — created during a gap in coverage and never injected. Set by the startup scan and decremented as each one is removed, so it reads as outstanding work, not a historical total. Cannot be fixed in place: the pods need a restart. |
Plus standard Go runtime and process metrics via prometheus/client_golang.
Namespace-label opt-in depends on the service account token. The plugin re-reads the projected token from disk (the kubelet rotates it before expiry), so it survives the short-lived bound tokens that some clusters (e.g. AKS) issue. If
cainjekt_ns_lookup_errors_totalclimbs, the API lookup is failing and namespace-label opt-in stops working until it recovers — pod-annotation and pod-label opt-in are unaffected because they need no API call.
Every injected container has /etc/cainjekt/status.json:
{
"injected": true,
"timestamp": "2026-04-15T08:27:15Z",
"distro": "debian",
"trust_store": "/etc/ssl/certs/ca-certificates.crt",
"ca_file": "/run/cainjekt/containers/.../ca-bundle.pem",
"processors": [
{ "name": "os-debian", "category": "os", "applicable": true },
{ "name": "lang-python", "category": "language", "applicable": true },
...
]
}Structured logging via log/slog. Configurable via CAINJEKT_LOG_LEVEL (debug/info/warn/error).
Key log points:
- Hook start: processor count, rootfs path, CA file
- Hook end: applied/skipped/failed counts, distro, trust store path
- Hook warning: read-only rootfs detected
- Wrapper: each applied processor, exec command
- Plugin: container create/remove events, injection decisions
If the plugin crashes or a container is removed without triggering RemoveContainer, the per-container CA directory at /run/cainjekt/containers/<id>/ becomes orphaned. A background goroutine sweeps every 5 minutes and removes directories that:
- Are not tracked by any currently active container
- Are older than 10 minutes
On SIGTERM/SIGINT:
- Stop accepting new NRI events
- Stop the orphan cleaner
- Gracefully shut down the HTTP server
- Exit cleanly
The CA bundle ConfigMap is read fresh on every CreateContainer call. When the ConfigMap is updated, Kubernetes refreshes the mounted file (via symlink swap). New containers automatically get the updated CA — no DaemonSet restart needed. Existing containers keep their staged CA.