Skip to content

trust-model.md

Latest commit

 

History

History
472 lines (408 loc) · 25.3 KB

trust-model.md

File metadata and controls

472 lines (408 loc) · 25.3 KB
Raw

Trust model and multi-tenant isolation

Audience: operators running crate on a shared host (multiple operators on one machine) and contributors extending the privileged surface.

Applies to: 1.1.33 (rootless model + per-tenant authz series 1.1.12 → 1.1.17 covering every privops verb that carries an operator-controlled ownership signal). For the ≤ 0.9.x setuid model and the migration, see rootless-migration.md.

This document states, explicitly, where cross-tenant isolation is enforced and where it is not — so deployments size their trust boundaries correctly and future work doesn't silently regress them.

TL;DR

Crate has two planes, with two different trust models:

  1. The privileged execution plane — crated's privops surface. Since 1.0.0 crate(1) is no longer setuid (Makefile, -m 0755); it is an unprivileged client. Every root operation (jail create/destroy, ZFS attach, mount, RCTL, interface/firewall config, signal) is performed by crated (which runs as root) when asked over privops — either POST /api/v1/privops/<verb> (admin-only) or the libnv AF_UNIX privops socket (group-gated). This plane does not arbitrate between operators. Anyone who can reach privops has root-equivalent control over every jail on the host. It is a single trust domain — the same property the old setuid crate(1) had, relocated into the daemon.

  2. The pooled observability / control plane. This is where per-tenant isolation is enforced, via pool ACLs + token scope, on the entry points that carry caller identity: dedicated control sockets (getpeereid gid + pool ACL), remote bearer tokens (scope + pool ACL), and the ws-console (bearer + pool ACL). Surface: list / get / stats / logs / start / stop / restart / PATCH resources / interactive console.

The invariant: multi-tenant isolation between mutually-distrusting operators lives only on the pooled plane's identity-carrying entry points (control-socket pools, token pools + scope), enforced daemon-side per request and keyed on the target jail's pool. The privops plane, and local Unix-socket access to the main HTTP API, are single trust domains. A hostile-multi-tenant deployment must therefore hand untrusted operators a pool-scoped control socket or bearer token only — never privops-socket access, never an admin token, and never the main API's Unix socket.

What changed since 0.7.19 — the privileged plane relocated

In 0.7.19 the single-trust-domain privileged plane was the setuid-root crate(1) binary. The rootless track (0.9.0–1.0.0, see rootless-migration.md) moved every privileged operation out of crate(1) and into crated's privops surface; 1.0.0 removed the setuid bit (Makefile, comment at the crate install line: "setuid bit removed. crate(1) runs as the operator and delegates privileged operations to crated(8)").

The single-trust-domain property did not disappear — it relocated. Reasoning about isolation on 1.1.33 means reasoning about who can reach privops, not who can run crate(1).

Plane 1 — the privops privileged execution plane (single trust domain)

crate(1) is installed -m 0755 (Makefile; setuid removed 1.0.0). crated runs as root (rc.d) and is the only privileged binary. It performs root operations only in response to a closed set of privops verbs (create_jail, destroy_jail, attach_zfs, set_rctl, configure_iface, add_pf_rule, signal_jail, apply_devfs_ruleset, … — lib/privops_pure.h). Two transports reach it:

  • HTTP — POST /api/v1/privops/<verb>. Gated admin-only: isAuthorized(req, config, "admin") (handlePrivOp(), daemon/routes.cpp). The handler comment states the design intent explicitly — "Privops touch host-wide state … so per-container scope from the F2 surface doesn't apply" (handlePrivOp(), daemon/routes.cpp). On this path the operator uid stays 0 (cpp-httplib doesn't expose the connection fd for getpeereid), so per-user audit is a no-op (handlePrivOp(), daemon/routes.cpp).
  • libnv AF_UNIX socket (0.9.14). Group-gated: the listener chmods the socket to its mode and chowns it root:<group> (openListener(), daemon/privops_listener.cpp; default mode 0660, Config::privopsSocketMode, daemon/config.h). getpeereid(2) extracts the peer uid (handleConnection(), daemon/privops_listener.cpp) and feeds both the per-user audit / namespacing hook and the authorize-before-dispatch gate below.

Verb dispatch is parse → validate → handle (dispatchPrivOp / dispatchPrivOpFromMap, daemon/privops_handlers.cpp). Authorization differs by transport:

  • HTTP: no per-resource check — admin-only and host-wide by design (handlePrivOp(), daemon/routes.cpp).

  • libnv (real peer uid): as of 1.1.12 an authorize-before-dispatch gate (dispatchPrivOpFromMap → PrivOpsAuthzPure::authorize, lib/privops_authz_pure.cpp) enforces per-user ownership for the verbs that carry a robust ownership signal, keyed on the caller's composeForUid env: attach_zfs/detach_zfs (the dataset must lie within the caller's ZFS prefix <master>/<uid>) and set_loginclass_rctl/clear_loginclass_rctl (the loginclass must be the caller's crate-<uid>). A foreign target is denied 403 before the handler runs (fail closed). 1.1.13 extends the same gate to jid- and name-scoped verbs: set_rctl, clear_rctl, set_jail_cpuset, query_jail_rctl, signal_jail, destroy_jail (1.1.40: exec_in_jail too — and strictly: an unregistered jid is denied as well, since exec-as-root into a jail is exactly what the registry exists to confine; the bootstrap "unknown → allow" concession the other jid verbs keep does not apply). A daemon-owned jid→owner registry (lib/jid_owner_registry.*) records the operator uid at create_jail time; subsequent verbs from a different operator are denied 403 against the same registry. 1.1.14 extends the same registry with a longest-prefix byPath lookup that gates the path-scoped verbs — mount_nullfs / unmount_nullfs (by target) and apply_devfs_ruleset / add_devfs_unhide_rule / apply_devfs_rules / mount_tmpfs (by mount_path). A path that lies inside another operator's registered jail is denied 403 (DenyForeignPath). 1.1.15 closes the last narrow item: create_jail's brand-new path argument is matched against the caller's per-user pathPrefix (composed from path_master_prefix: in crated.conf); a foreign target is denied 403 (DenyForeignCreatePath). 1.1.17 closes the last cross-tenant holes: mount_nullfs's source end (a source inside another tenant's path_master_prefix/<uid> is denied 403 DenyForeignSource; host paths and GUI runtime sockets stay allowed), and the two interface verbs that operate on a specific jail rather than shared host state — configure_iface (it jexecs into the named jid and moves an iface into its vnet → gated by jid) and reclaim_iface_from_vnet (pulls an iface out of a named jail → gated by jail_name).

  • 1.1.36: add_pf_rule / remove_pf_rule / flush_pf_anchor — the anchor is crate's per-jail crate/<kernel jail name>, so it is gated like destroy_jail's name (DenyForeignAnchor for another tenant's jail, for the root crate anchor and for any non-crate anchor); attach_zfs / detach_zfs gate the jid end as well as the dataset. The jid→owner registry is reconciled against the live kernel at startup and on every lookup hit (a jid that is gone or now carries a different name drops its row), and destroy_jail forgets by the jid resolved before removal.

The remaining verbs still pass the gate: genuinely host-global verbs (set_iface_up, add_ipfw_rule, configure_ipfw_nat, create_epair) act on shared host state with no tenant-specific target, so they cannot be pool-scoped and stay host-wide by design. The interface-named verbs (configure_iface's ifname, teardown_iface, bridge_add_member / bridge_del_member's member, the iface of reclaim_iface_from_vnet, set_iface_up, disable_iface_offload, set_iface_inet_addr, set_iface_mac) carry a tenant signal too — an epair half belongs to whoever's create_epair made it — and since 1.1.43 the daemon resolves it through an iface→owner registry (lib/iface_owner_registry.*, persisted at /var/db/crate/iface_owners.tsv): both halves are recorded at create_epair for the peer uid, forgotten at teardown_iface, and reconciled against the host's live interface list. A per-user caller may name only epair halves: another operator's half is 403 (DenyForeignIface), a host interface by name (em0, bridge0, vlan…) is 403 (DenyHostIface) — the "move any host NIC into my vnet" hole is closed — and an unregistered half gets the same bootstrap Allow as an unknown jid. The bridge named in bridge_*_member / configure_iface stays shared host state. Everything else that carries an operator-controlled ownership signal is gated. The per-verb handlers remain uid-blind; the gate runs ahead of them.

Consequence: whoever can reach privops — an admin bearer token, or membership in the privops socket's group — still has host-wide control over the un-gated surface (firewall, host interface plumbing, other host-global verbs that touch shared state). The 1.1.12 → 1.1.43 gates close cross-tenant ZFS-dataset, RCTL-umbrella, jid-, name-, path-scoped, create-jail-path, mount-source, and per-jail interface access on the libnv path — every verb that carries an operator-controlled ownership signal in its request is now gated. The shared host-global verbs remain, by design, a single trust domain — handing an operator privops access is still close to handing them the old setuid crate(1) for those.

Per-user namespacing is convenience, not a boundary

Rootless mode derives per-operator paths, ZFS prefixes, network sub-CIDRs and an RCTL umbrella class from the connecting operator's uid (lib/per_user_*, lib/runtime_paths_pure.*; see rootless-migration.md). This cleanly separates honest operators — alice's and bob's web jails land in different ZFS subtrees and CIDRs.

Sizing note (1.1.18): the network sub-CIDR maps each uid into a slot of subPrefix − masterPrefix bits. The master CIDR / sub-prefix must be sized for the operator uid range — a /16 master with /24 subs holds only uids 0–255, so realistic uids (≥ 1000) need a wider master (e.g. /8 → /24 = 16 slot bits). An out-of-range uid now errors at compose time rather than silently aliasing another operator's sub-CIDR (pre-1.1.18 it wrapped, two honest operators could collide).

It is not an adversarial boundary on the privops plane. The per-uid prefix is computed client-side (in crate(1)) and passed in the request; crated does not re-derive or validate the request's jid / dataset / path against the peer uid. A hostile privops-group member can craft a raw nvlist request naming another operator's prefix and crated will act on it. So "bob can't run a jail in alice's ZFS prefix" (rootless-migration.md) holds for honest clients going through an honest crate(1); it is not a daemon-enforced cross-tenant boundary. Enforced adversarial isolation lives on Plane 2.

Plane 2 — the pooled observability / control plane (isolated)

crated's non-privileged surface. Per-tenant isolation is enforced on the entry points that carry caller identity.

2a. Dedicated control sockets (0.7.10) — isolated

Per-group AF_UNIX sockets under /var/run/crate/control/, with three layers of defense:

  1. Filesystem perms (kernel). Socket chmod'd to the spec mode and chown'd root:<group> so only group members can connect(2) — bindSocketOrThrow(), daemon/control_socket.cpp.
  2. getpeereid(2) gid re-check. Even if the mode is loosened, the peer's gid must equal the socket's expected gid — handleConnection(), daemon/control_socket.cpp feeding ControlSocketPure::authorize (ControlSocketPure::authorize(), daemon/control_socket_pure.cpp, Decision::DenyGidMismatch).
  3. Pool ACL. For per-container actions the container's pool (PoolPure::inferPool, ControlSocketPure::authorize()) must be visible on the socket's pools list — poolVisibleOnSocket (authorize(), defined daemon/control_socket_pure.cpp). Mutating actions (PATCH resources, POST start/stop/restart) additionally require the admin role (authorize() → DenyRoleMismatch, 0.8.13).

Result: alice's socket (pools: ["alice"]) cannot observe, patch, or start/stop a jail in bob's pool. This is the mechanism a multi-tenant deployment relies on. Note these sockets cover the control surface only — they do not expose create_jail / attach_zfs, which are privops (Plane 1). The pool ACL confines which jails a socket may act on, not what a started container may do: start from an admin socket runs that container's recorded .crate as root, with whatever host shares and scripts its spec declares. Hand out admin sockets only to operators you would also trust with the spec.

2b. Remote bearer tokens (0.7.1 scope, 0.7.4 pools) — isolated

HTTP API clients authenticate with a bearer token carrying expiry (AuthToken::expiresAt, daemon/config.h; 0 == never), scope path-globs (AuthToken::scope, daemon/config.h; matched by AuthPure::pathInScope, lib/auth_pure.cpp), a role, and a pool ACL (AuthToken::pools, daemon/config.h). checkBearerAuthFull gates expiry + scope + role (lib/auth_pure.cpp); the per-container pool gate is isAuthorizedForContainer → PoolPure::tokenAllowsContainer (daemon/auth.cpp).

Caveat: an admin bearer token also unlocks the privops HTTP plane (2a above is role-gated, privops is admin-gated). An admin token is therefore host-wide, not pool-confined. Only viewer/pool-scoped tokens are isolation-bearing.

2c. ws-console interactive shell — isolated (bearer + pool)

The websocket console grants an interactive jexec shell inside a jail, so its gate is load-bearing. It requires an admin bearer token (handleClient(), daemon/ws_console.cpp) and that the jail's pool be allowed by the token (PoolPure::inferPool / tokenAllowsContainer, handleClient(), daemon/ws_console.cpp).

2d. Local Unix-socket access to the main HTTP API — NOT isolated

The main crated HTTP API is also reachable over a local Unix socket. On that path cpp-httplib does not expose the peer fd, so getpeereid(2) is not wired in. Unix-socket peers are trusted wholesale:

  • isAuthorized returns true immediately for Unix peers, bypassing bearer auth — isAuthorized(), daemon/auth.cpp.
  • isAuthorizedForContainer likewise bypasses the pool ACL for Unix peers — isAuthorizedForContainer(), daemon/auth.cpp.

How "Unix peer" is decided (1.1.23). crated runs two separate httplib::Server instances — one on TCP, one on the Unix socket (Server::start(), daemon/server.cpp). registerRoutes() is told which one it is wiring and installs a pre-routing handler that, on every request, erases any client-supplied X-Crated-Listener header and stamps the authoritative value (unix / tcp) for the accepting server (registerRoutes(), daemon/routes.cpp). isUnixSocketPeer() reads only that marker. A TCP client therefore cannot impersonate a socket peer, and a missing marker reads as untrusted TCP (fail-closed). Before 1.1.23 locality was inferred from an empty REMOTE_ADDR header — which a TCP client could shadow with its own empty header, a potential token bypass whenever the TCP listener was enabled.

The socket file mode (default 0660 root:wheel, Config::unixSocketMode, daemon/config.h) is therefore the only gate on who may reach the trusted listener. So local access to the main API is, like privops, a single trust domain. getpeereid-based auth here is still future work (roadmap §5.3). Only the dedicated control sockets (2a) carry per-pool identity locally.

The invariant, stated for implementers

Cross-tenant isolation = pool ACLs (control-socket pools, token pools) + token scope, enforced per request, keyed on the target jail's pool. It exists only on entry points that carry caller identity (Plane 2a/2b/2c). The privops plane and the main API's Unix socket are single trust domains.

This is the contract any new privileged surface inherits.

The open gap, and the guardrail for closing it

1.1.12 began closing this gap: the libnv path now authorizes attach_zfs/detach_zfs (by ZFS prefix) and the loginclass-RCTL verbs (by crate-<uid>) before dispatch. The rest of Plane 1 is still a single trust domain — host-wide admin token on HTTP, host-wide group membership on the libnv socket for every un-gated verb. That is fine as long as privops access is treated as equivalent to handing out the old setuid crate(1) — i.e. only ever given to fully-trusted operators.

The unresolved tension: the rootless model requires crate(1) to reach privops to create a jail at all, while the per-user namespacing (above) is marketed as multi-tenant isolation. To make privops itself safe for mutually-distrusting operators, the following MUST hold for every verb — or the per-user split is honest-operator hygiene, not a security boundary:

  1. Authorize before dispatch. getpeereid uid/gid identifies the caller; it does not authorize the operation. Every privileged verb must run an ownership check — the same shape as poolVisibleOnSocket / tokenAllowsContainer — keyed on the target jail/pool/dataset, before the operation runs. Done (1.1.12): dataset and loginclass verbs (lib/privops_authz_pure.cpp). Done (1.1.13): the jid- and name-scoped verbs — set_rctl, clear_rctl, set_jail_cpuset, query_jail_rctl, signal_jail (gated by jid) and destroy_jail (gated by jail name). A daemon-owned jid→owner registry (lib/jid_owner_registry.*, persisted at /var/db/crate/jid_owners.tsv) records the operator uid at create_jail time; subsequent jid/name-scoped verbs from a different operator are denied 403 before the handler runs. Jails that pre-date 1.1.13 are not in the registry — the gate's bootstrap concession allows them through to preserve the upgrade path. Done (1.1.14): the path-scoped verbs — mount_nullfs / unmount_nullfs / mount_tmpfs (gated by target) and apply_devfs_ruleset / apply_devfs_rules / add_devfs_unhide_rule (gated by mount_path). The registry now exposes a longest-prefix byPath lookup; a path inside a tracked jail owned by another uid is denied 403 (DenyForeignPath); paths outside every registered jail fall through under the same bootstrap concession — narrowed in 1.1.35: with path_master_prefix configured, an unregistered target / mount_path is allowed only inside the caller's own <master>/<uid> slice (DenyUnownedPath otherwise), so host directories, the registry file, the host /dev and another tenant's stopped rootfs (forgotten by destroy_jail) are no longer reachable through mount_nullfs / unmount_nullfs / devfs. Without the split the single-trust-domain behaviour is unchanged. Done (1.1.15): the last narrow item — the create_jail path argument. PerUserEnvPure::Config gains pathMasterPrefix (configured via crated.conf path_master_prefix:); composeForUid() derives env.pathPrefix = <master>/<uid>. The gate runs PrivOpsAuthzPure::pathOwned(req.path, env.pathPrefix) — slash-anchored prefix match, identical shape to datasetOwned for ZFS. A foreign target is denied 403 (DenyForeignCreatePath). Empty pathMasterPrefix keeps the legacy shape (Allow), so existing deployments don't need to reconfigure on upgrade. Done (1.1.35): the create_jail parameters field. It used to be checked only for newlines and shell metacharacters and then split straight into jail -c argv, so any privops-group member could pass exec.prestart=… (runs on the host as root), mount.fstab=, allow.mount.*, devfs_ruleset, ip4.addr or a second name= / path= overriding the gated ones. PrivOpsPure::validateJailParameters now whitelists each key=value: the nine allow.* booleans crate itself sets, enforce_statfs (0–2), securelevel (−1–3) and children.max; everything else is 400.

  2. Per-operator namespacing is convenience, not a boundary. Any path / jid / dataset argument crossing the privops socket must be re-derived or validated daemon-side against the caller's uid-prefix — never taken at face value. Done for dataset (1.1.12), for jid / jail name (1.1.13, via the registry), for path-scoped runtime verbs (1.1.14, longest-prefix byPath on the same registry), and for create_jail's brand-new path (1.1.15, slash-anchored prefix against env.pathPrefix). Done (1.1.35): the string checks above are backed by canonicalisation — crated realpath()s every path argument (target, source, mount_path, create_jail path) before the handler acts and refuses one that does not resolve to itself (400 path_not_canonical). A tenant controls its own jail root, so a symlink planted there could otherwise make a string that passed the registry/prefix checks while the kernel mounted or created somewhere else. The client side (lib/run_gui.cpp) likewise lstat()s each nullfs mount target inside the jail and refuses anything but an empty regular file it created itself. Residual: a symlink swapped in between the check and the syscall — closing that needs fd-based mounts (O_RESOLVE_BENEATH), tracked in TODO.

  3. Fail closed on identity loss. On any path that authorizes, a getpeereid failure must deny. Done (1.1.19): the libnv listener rejects a connection with 403 when getpeereid fails and rootless_per_user is on, instead of degrading to peerUid = 0 (which the dispatcher would treat as the admin/host-wide path and skip every per-tenant gate). It degrades to a no-op only when per-user enforcement is off — there peerUid fed only the audit trail, not an access decision.

What still remains host-wide after (1)–(3) is by design, not a pending gate: the genuinely host-global verbs (set_iface_up, add_ipfw_rule, configure_ipfw_nat, create_epair, create_bridge — shared host state with no tenant target; the interface-named verbs are gated since 1.1.43, and the shared firewall state — ipfw slot, forwarding refcount — moved into the daemon as the jid-gated acquire_fw_lease / release_fw_lease pair in 1.1.57, so a tenant can take a slot only for its own jail and never flips a sysctl itself), the admin-only HTTP transport (no peer uid), and the bootstrap concession that lets jids / names absent from the registry through (for paths it is limited to the caller's own prefix since 1.1.35). A multi-tenant deployment that must keep operators off that surface still needs a trusted broker rather than handing them raw privops-socket access.

Deployment guidance (mutually-distrusting tenants)

  • Give untrusted operators: a pool-scoped dedicated control socket (2a) and/or a viewer/pool-scoped bearer token (2b) only.
  • Do not give untrusted operators: privops-socket (group) access, an admin bearer token, the main API's Unix socket, or a crate(1) that must reach privops to create jails.
  • Honest-operator separation (per-user paths/datasets/CIDRs) and enforced adversarial isolation (pool-ACL'd Plane 2) are different guarantees — don't conflate them.

GUI device exposure (gui.mode: compositor)

Running a Wayland compositor inside a jail can expose host device nodes that the default devfs ruleset hides:

  • gui.backend: headless (default) — renders offscreen and is surfaced over VNC (wayvnc). It exposes no input devices; it touches /dev/dri/* only when a render node is present (GPU acceleration). Safe to hand to a semi-trusted workload.
  • gui.backend: drm — the jail drives the physical GPU and input directly. crate unhides /dev/dri/* and /dev/input/* in the jail's devfs view and binds the host seatd socket. This is a real privilege surface: a process in such a jail can read all input events host-wide (every keystroke, every pointer event) and talk to the GPU at the KMS level. Treat a drm-backend jail as part of the host's trusted display domain — do not grant it to a mutually-distrusting tenant. It is opt-in precisely because the default (headless) avoids this exposure.

See also

  • rootless-migration.md — the 0.9.0–1.0.0 setuid → privops migration and the per-user model.
  • security-command-paths.md — absolute command paths / env-sanitization (CWE-426). Now most relevant to crated, which is the process that execs host tools as root.
  • implementation-roadmap.md §5.1 (high-level REST write endpoints), §5.3 (getpeereid auth on the main API).