diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e4e7523..57ee3fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -107,6 +107,8 @@ jobs: "$(go env GOPATH)/bin/gitleaks" detect --source . --no-git --redact --no-banner - name: Helm Insights contracts run: scripts/test-helm-insights.sh + - name: Helm custom image contracts + run: scripts/test-helm-images.sh - name: Render LoadBalancer installation run: helm template devboxes charts/devboxes --namespace devboxes > /tmp/devboxes.yaml - name: Validate Kubernetes resources diff --git a/CHANGELOG.md b/CHANGELOG.md index 3540f74..32dba5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,19 @@ All notable changes to Devboxes are documented here. The project follows [Keep a ## [Unreleased] +### Added + +- Added opt-in custom image profiles across Helm, the API, CLI, dashboard, authenticated documentation, and public documentation. Profiles support isolated pod-local service sidecars and explicitly vetted Devboxes-compatible workspace derivatives. +- Added `devbox image profiles` and `devbox create --image PROFILE_OR_IMAGE`, resolved image allocation reporting, bounded sidecar resources and ports, strict chart/controller validation, and Helm contract tests. + +### Changed + +- Persist the fully resolved custom image profile on each workspace so stop, start, TTL expiry, and Insights template reconciliation cannot silently change an existing image contract after Helm configuration changes. + +### Security + +- Restrict custom image selection to an operator-approved catalog. Sidecars receive no Devboxes Secret, persistent-home mount, Kubernetes API token, public Service, extra capability, command override, or scheduling injection surface. + ## [0.4.0] - 2026-07-20 ### Added diff --git a/Makefile b/Makefile index 36e0ba9..4dc81a0 100644 --- a/Makefile +++ b/Makefile @@ -21,6 +21,7 @@ test: helm: scripts/test-helm-insights.sh scripts/test-helm-gpu.sh + scripts/test-helm-images.sh helm template devboxes charts/devboxes --namespace devboxes --set workspace.sshService.type=NodePort --set workspace.sshService.host=192.0.2.10 >/dev/null images: diff --git a/README.md b/README.md index a8832a1..998578d 100644 --- a/README.md +++ b/README.md @@ -20,10 +20,11 @@ Each workspace includes Rust, Node.js, Python, `uv`, GitHub CLI, Codex CLI, Clau ## What ships -- A Rust `devbox` CLI for create, list, inspect, SSH, start, stop, delete, and opt-in Insights workflows. +- A Rust `devbox` CLI for create, list, inspect, SSH, start, stop, delete, custom-image profiles, and opt-in Insights workflows. - A FastAPI controller with an authenticated API, accessible browser workbench, Insights dashboard, documentation, metrics, health checks, and TTL cleanup. - A versioned Helm chart with values schema validation and namespace-scoped RBAC. - Optional operator-approved GPU profiles for NVIDIA, AMD, Intel, partitioned, or shared accelerators. +- Optional operator-approved custom image profiles for pod-local services or vetted workspace derivatives. - Multi-architecture controller and workspace images for `linux/amd64` and `linux/arm64`. - Persistent SSH host identity, shell state, tool installs, account state, and source under `/home/dev`. - GitHub Releases with macOS and Linux CLI binaries and SHA-256 checksums. @@ -169,6 +170,35 @@ devbox create training --gpu-profile nvidia-l4 --preset large --ssh The dashboard exposes the same profiles in its create form. Devboxes sets the resource in both container requests and limits, preserves the resolved allocation across stop and start, and surfaces scheduler reasons when capacity is unavailable. Read [GPU acceleration](docs/gpu.md) for driver prerequisites, NVIDIA and AMD examples, image contracts, sharing, security, upgrades, and troubleshooting. +### Enable approved custom images + +Devboxes does not accept an unrestricted container image from a client. Instead, an operator publishes a reviewed catalog. Service profiles run an unprivileged image such as NGINX as a credential-free sidecar beside the prepared SSH workspace; workspace profiles are only for compatible Devboxes-derived images. + +```yaml +workspace: + customImages: + enabled: true + profiles: + - name: nginx + displayName: NGINX preview + description: Serve a local static-site preview + image: docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine + mode: sidecar + ports: + - name: http + containerPort: 8080 +``` + +Users discover and select the same catalog in the terminal or dashboard: + +```bash +devbox image profiles +devbox create docs-preview --image nginx --ssh +devbox ssh docs-preview -- -L 8080:127.0.0.1:8080 +``` + +Read [custom image profiles](docs/images.md) for the image contract, resource bounds, security boundary, upgrade behavior, and workspace-mode requirements. + ### Enable Insights Insights is disabled by default. Enable it to collect privacy-bounded local AI metrics and aggregate Git activity into a persistent controller database: @@ -313,6 +343,7 @@ Read [CONTRIBUTING.md](CONTRIBUTING.md) before proposing a change. Security repo - [Golden path](docs/golden-path.md) for a performance-oriented installation and daily workflow. - [CLI reference](docs/cli.md) and [API reference](docs/api.md) for client contracts. - [GPU acceleration](docs/gpu.md) for accelerator profiles, images, scheduling, and operations. +- [Custom image profiles](docs/images.md) for approved sidecars, compatible workspace derivatives, and their security boundary. - [Insights](docs/insights.md) for telemetry semantics, privacy, storage, backup, and purge. - [Configuration](docs/configuration.md) and [credentials](docs/credentials.md) for installation details. - [Operations](docs/operations.md) and [troubleshooting](docs/troubleshooting.md) for production ownership. diff --git a/charts/devboxes/templates/deployment.yaml b/charts/devboxes/templates/deployment.yaml index a0c3988..f6c7383 100644 --- a/charts/devboxes/templates/deployment.yaml +++ b/charts/devboxes/templates/deployment.yaml @@ -24,6 +24,27 @@ {{- if and .Values.gpu.enabled (not .Values.gpu.defaultProfile) }} {{- fail "gpu.defaultProfile is required when gpu.enabled=true" }} {{- end }} +{{- $customImageNames := dict }} +{{- $customImageReferences := dict }} +{{- range .Values.workspace.customImages.profiles }} +{{- if hasKey $customImageNames .name }} +{{- fail (printf "workspace.customImages contains duplicate name %q" .name) }} +{{- end }} +{{- if hasKey $customImageReferences .image }} +{{- fail (printf "workspace.customImages contains duplicate image %q" .image) }} +{{- end }} +{{- if not (trim .displayName) }} +{{- fail (printf "custom image profile %q displayName must not be blank" .name) }} +{{- end }} +{{- if contains "://" .image }} +{{- fail (printf "custom image profile %q image must not contain a URL scheme" .name) }} +{{- end }} +{{- $_ := set $customImageNames .name true }} +{{- $_ := set $customImageReferences .image true }} +{{- end }} +{{- if and .Values.workspace.customImages.enabled (eq (len .Values.workspace.customImages.profiles) 0) }} +{{- fail "workspace.customImages.profiles is required when workspace.customImages.enabled=true" }} +{{- end }} apiVersion: apps/v1 kind: Deployment metadata: @@ -161,6 +182,10 @@ spec: {{- end }} - name: DEVBOXES_WORKSPACE_IMAGE value: {{ include "devboxes.workspaceImage" . | quote }} + - name: DEVBOXES_CUSTOM_IMAGES_ENABLED + value: {{ .Values.workspace.customImages.enabled | quote }} + - name: DEVBOXES_CUSTOM_IMAGES + value: {{ .Values.workspace.customImages.profiles | toJson | quote }} - name: DEVBOXES_WORKSPACE_SECRET_NAME value: {{ .Values.workspace.existingSecret | quote }} - name: DEVBOXES_WORKSPACE_SERVICE_ACCOUNT_NAME diff --git a/charts/devboxes/values.schema.json b/charts/devboxes/values.schema.json index 15ad818..1d90a48 100644 --- a/charts/devboxes/values.schema.json +++ b/charts/devboxes/values.schema.json @@ -42,6 +42,25 @@ "required": ["image", "existingSecret", "serviceAccount", "sshService"], "properties": { "image": {"$ref": "#/definitions/imageWithoutPullPolicy"}, + "customImages": { + "type": "object", + "additionalProperties": false, + "required": ["enabled", "profiles"], + "properties": { + "enabled": {"type": "boolean"}, + "profiles": { + "type": "array", + "maxItems": 32, + "items": {"$ref": "#/definitions/customImageProfile"} + } + }, + "allOf": [ + { + "if": {"properties": {"enabled": {"const": true}}}, + "then": {"properties": {"profiles": {"minItems": 1}}} + } + ] + }, "existingSecret": {"type": "string", "minLength": 1}, "imagePullSecret": {"type": "string"}, "storageClass": {"type": "string"}, @@ -274,6 +293,64 @@ "tag": {"type": "string"} } }, + "customImageProfile": { + "type": "object", + "additionalProperties": false, + "required": ["name", "displayName", "image"], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "maxLength": 40, + "pattern": "^[a-z0-9](?:[a-z0-9-]{0,38}[a-z0-9])?$" + }, + "displayName": {"type": "string", "minLength": 1, "maxLength": 80}, + "description": {"type": "string", "maxLength": 160}, + "image": {"type": "string", "minLength": 1, "maxLength": 512}, + "mode": {"type": "string", "enum": ["sidecar", "workspace"]}, + "pullPolicy": {"type": "string", "enum": ["Always", "IfNotPresent", "Never"]}, + "resources": {"$ref": "#/definitions/customImageResources"}, + "ports": { + "type": "array", + "maxItems": 8, + "items": {"$ref": "#/definitions/customImagePort"} + } + }, + "allOf": [ + { + "if": { + "properties": {"mode": {"const": "workspace"}}, + "required": ["mode"] + }, + "then": {"not": {"required": ["resources"]}} + } + ] + }, + "customImageResources": { + "type": "object", + "additionalProperties": false, + "properties": { + "cpuRequest": {"type": "string", "minLength": 1, "maxLength": 32}, + "memoryRequest": {"type": "string", "minLength": 1, "maxLength": 32}, + "cpuLimit": {"type": "string", "minLength": 1, "maxLength": 32}, + "memoryLimit": {"type": "string", "minLength": 1, "maxLength": 32} + } + }, + "customImagePort": { + "type": "object", + "additionalProperties": false, + "required": ["name", "containerPort"], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "maxLength": 63, + "pattern": "^[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?$" + }, + "containerPort": {"type": "integer", "minimum": 1024, "maximum": 65535}, + "protocol": {"type": "string", "enum": ["TCP", "UDP", "SCTP"]} + } + }, "gpuToleration": { "type": "object", "additionalProperties": false, diff --git a/charts/devboxes/values.yaml b/charts/devboxes/values.yaml index 8011757..34ca620 100644 --- a/charts/devboxes/values.yaml +++ b/charts/devboxes/values.yaml @@ -41,6 +41,28 @@ workspace: image: repository: ghcr.io/vicotrbb/devboxes-workspace tag: "" + # Custom images are opt-in and operator-owned. A sidecar profile runs a + # compatible non-root application image beside the prepared Devboxes workspace. + # A workspace profile replaces it and must preserve the complete SSH contract. + customImages: + enabled: false + # Example profile fields: + # - name: nginx + # displayName: NGINX preview + # description: Serve a local static-site preview over the pod network + # image: docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine + # mode: sidecar + # pullPolicy: IfNotPresent + # resources: + # cpuRequest: 25m + # memoryRequest: 32Mi + # cpuLimit: 500m + # memoryLimit: 512Mi + # ports: + # - name: http + # containerPort: 8080 + # protocol: TCP + profiles: [] existingSecret: devboxes-workspace imagePullSecret: "" storageClass: "" diff --git a/cli/README.md b/cli/README.md index dbcd8c6..0a5f754 100644 --- a/cli/README.md +++ b/cli/README.md @@ -8,6 +8,8 @@ devbox login --url https://devboxes.example.com devbox create atlas --repo owner/project --ssh devbox gpu profiles devbox create inference --gpu --ssh +devbox image profiles +devbox create docs-preview --image nginx --ssh ``` -See the [CLI reference](../docs/cli.md) for every command, option, environment variable, output contract, and SSH workflow. [GPU acceleration](../docs/gpu.md) covers operator-approved accelerator profiles. The [golden path](../docs/golden-path.md) covers the recommended installation and performance setup. +See the [CLI reference](../docs/cli.md) for every command, option, environment variable, output contract, and SSH workflow. [GPU acceleration](../docs/gpu.md) covers operator-approved accelerator profiles, and [custom image profiles](../docs/images.md) covers approved service and workspace images. The [golden path](../docs/golden-path.md) covers the recommended installation and performance setup. diff --git a/cli/src/main.rs b/cli/src/main.rs index 86113b9..d5756a4 100644 --- a/cli/src/main.rs +++ b/cli/src/main.rs @@ -16,8 +16,9 @@ use tokio::time::{Instant, sleep}; use client::ApiClient; use config::StoredConfig; use models::{ - CollectorStatus, CreateDevbox, Devbox, GpuCapabilities, GpuRequest, InsightsActivity, - InsightsActivityData, InsightsEnvelope, InsightsStatusData, InsightsSummary, Preset, + CollectorStatus, CreateDevbox, CustomImageCapabilities, Devbox, GpuCapabilities, GpuRequest, + InsightsActivity, InsightsActivityData, InsightsEnvelope, InsightsStatusData, InsightsSummary, + Preset, }; #[derive(Parser)] @@ -65,6 +66,8 @@ enum Commands { Metrics(MetricsArgs), /// Inspect GPU acceleration profiles configured by the operator. Gpu(GpuArgs), + /// Inspect custom image profiles configured by the operator. + Image(ImageArgs), } #[derive(Args)] @@ -94,6 +97,10 @@ struct CreateArgs { #[arg(long)] repo: Option, + /// Request an operator-approved image profile or exact approved image reference. + #[arg(long, value_name = "PROFILE_OR_IMAGE", value_parser = validate_image_selector)] + image: Option, + /// Request the operator's default GPU profile. #[arg(long)] gpu: bool, @@ -117,12 +124,24 @@ struct GpuArgs { command: Option, } +#[derive(Args)] +struct ImageArgs { + #[command(subcommand)] + command: Option, +} + #[derive(Subcommand)] enum GpuCommand { /// List the GPU profiles available for new devboxes. Profiles, } +#[derive(Subcommand)] +enum ImageCommand { + /// List the custom image profiles available for new devboxes. + Profiles, +} + #[derive(Args)] struct NameArgs { #[arg(value_parser = validate_name)] @@ -272,6 +291,7 @@ async fn main() -> Result<()> { Commands::Delete(args) => delete(&client, args).await, Commands::Metrics(args) => metrics(&client, args, cli.json).await, Commands::Gpu(args) => gpu(&client, args, cli.json).await, + Commands::Image(args) => image(&client, args, cli.json).await, } } @@ -375,6 +395,22 @@ fn validate_name(value: &str) -> std::result::Result { } } +fn validate_image_selector(value: &str) -> std::result::Result { + let value = value.trim(); + if value.is_empty() + || value.len() > 512 + || value.chars().any(char::is_whitespace) + || value.contains("://") + { + Err( + "use an operator-approved image profile or image reference without whitespace or a URL scheme" + .to_owned(), + ) + } else { + Ok(value.to_owned()) + } +} + async fn create( client: &ApiClient, args: CreateArgs, @@ -393,6 +429,7 @@ async fn create( preset: args.preset, ttl_hours: args.ttl, repository: args.repo.as_deref(), + image: args.image.as_deref(), gpu, }; let mut box_info = client.create(&payload).await?; @@ -418,17 +455,18 @@ async fn list(client: &ApiClient, json: bool) -> Result<()> { return Ok(()); } println!( - "{:<22} {:<11} {:<8} {:<16} {:<18} SSH", - "NAME", "STATE", "SIZE", "AUTO-STOP", "ACCELERATOR" + "{:<22} {:<11} {:<8} {:<16} {:<18} {:<18} SSH", + "NAME", "STATE", "SIZE", "AUTO-STOP", "ACCELERATOR", "IMAGE" ); for box_info in boxes { println!( - "{:<22} {:<11} {:<8} {:<16} {:<18} {}", + "{:<22} {:<11} {:<8} {:<16} {:<18} {:<18} {}", box_info.name, box_info.state, box_info.preset, human_expiry(&box_info), gpu_label(&box_info), + image_label(&box_info), box_info.ssh_command.as_deref().unwrap_or("pending"), ); } @@ -569,6 +607,21 @@ async fn gpu(client: &ApiClient, args: GpuArgs, json: bool) -> Result<()> { Ok(()) } +async fn image(client: &ApiClient, args: ImageArgs, json: bool) -> Result<()> { + let ImageArgs { command } = args; + match command { + None | Some(ImageCommand::Profiles) => { + let capabilities = client.capabilities().await?.images; + if json { + println!("{}", serde_json::to_string_pretty(&capabilities)?); + } else { + print_image_profiles(&capabilities); + } + } + } + Ok(()) +} + fn print_gpu_profiles(capabilities: &GpuCapabilities) { if !capabilities.enabled { println!("GPU acceleration is disabled by the operator."); @@ -591,6 +644,33 @@ fn print_gpu_profiles(capabilities: &GpuCapabilities) { } } +fn print_image_profiles(capabilities: &CustomImageCapabilities) { + if !capabilities.enabled { + println!("Custom images are disabled by the operator."); + return; + } + println!( + "{:<18} {:<26} {:<10} PORTS DESCRIPTION", + "PROFILE", "NAME", "MODE" + ); + for profile in &capabilities.profiles { + let ports = profile + .ports + .iter() + .map(|port| format!("{}:{}/{}", port.name, port.container_port, port.protocol)) + .collect::>() + .join(","); + println!( + "{:<18} {:<26} {:<10} {:<16} {}", + profile.name, + profile.display_name, + profile.mode, + ports, + profile.description.as_deref().unwrap_or(""), + ); + } +} + fn metrics_query(filters: &MetricsFilters) -> Vec<(String, String)> { let mut query = vec![("since".to_owned(), filters.since.clone())]; for (key, value) in [ @@ -847,6 +927,18 @@ fn print_box(box_info: &Devbox, json: bool) -> Result<()> { gpu.display_name, gpu.count, gpu.resource_name ); } + if let Some(image) = &box_info.image { + println!(" image: {} ({})", image.display_name, image.mode); + if !image.ports.is_empty() { + let ports = image + .ports + .iter() + .map(|port| format!("{}:{}/{}", port.name, port.container_port, port.protocol)) + .collect::>() + .join(", "); + println!(" image ports: {ports}"); + } + } println!(" auto-stop: {}", box_info.expires_at.to_rfc3339()); if let Some(repository) = &box_info.repository { println!(" repository: {repository}"); @@ -866,6 +958,13 @@ fn gpu_label(box_info: &Devbox) -> &str { .map_or("cpu", |gpu| gpu.profile.as_str()) } +fn image_label(box_info: &Devbox) -> &str { + box_info + .image + .as_ref() + .map_or("prepared", |image| image.profile.as_str()) +} + fn human_expiry(box_info: &Devbox) -> String { if box_info.state == "stopped" { return "stopped".to_owned(); @@ -883,8 +982,8 @@ mod tests { use clap::Parser; use super::{ - Cli, Commands, GpuCommand, MetricsCommand, human_duration, metrics_query, - resolve_login_token, ssh_arguments, validate_name, + Cli, Commands, GpuCommand, ImageCommand, MetricsCommand, human_duration, metrics_query, + resolve_login_token, ssh_arguments, validate_image_selector, validate_name, }; #[test] @@ -915,6 +1014,16 @@ mod tests { assert!(validate_name(&"a".repeat(41)).is_err()); } + #[test] + fn image_selectors_match_the_controller_contract() { + assert_eq!( + validate_image_selector(" docker.io/library/nginx:1.27 ").unwrap(), + "docker.io/library/nginx:1.27" + ); + assert!(validate_image_selector("https://registry.example/image:tag").is_err()); + assert!(validate_image_selector("image with spaces").is_err()); + } + #[test] fn extra_ssh_options_come_before_the_destination() { let extra = vec!["-L".to_owned(), "3000:127.0.0.1:3000".to_owned()]; @@ -1020,6 +1129,38 @@ mod tests { assert!(matches!(profiles.command, Some(GpuCommand::Profiles))); } + #[test] + fn image_profiles_and_create_image_are_discoverable() { + let root = Cli::try_parse_from(["devbox", "image"]).unwrap(); + let Commands::Image(root) = root.command else { + panic!("expected image command"); + }; + assert!(root.command.is_none()); + + let profiles = Cli::try_parse_from(["devbox", "image", "profiles", "--json"]).unwrap(); + assert!(profiles.json); + let Commands::Image(profiles) = profiles.command else { + panic!("expected image command"); + }; + assert!(matches!(profiles.command, Some(ImageCommand::Profiles))); + + let create = Cli::try_parse_from([ + "devbox", + "create", + "nginx", + "--image", + "docker.io/library/nginx:1.27", + ]) + .unwrap(); + let Commands::Create(create) = create.command else { + panic!("expected create command"); + }; + assert_eq!( + create.image.as_deref(), + Some("docker.io/library/nginx:1.27") + ); + } + #[test] fn active_time_format_is_compact_and_bounded() { assert_eq!(human_duration(0.0), "0m"); diff --git a/cli/src/models.rs b/cli/src/models.rs index edeedb3..e392844 100644 --- a/cli/src/models.rs +++ b/cli/src/models.rs @@ -34,6 +34,8 @@ pub struct CreateDevbox<'a> { pub ttl_hours: u16, pub repository: Option<&'a str>, #[serde(skip_serializing_if = "Option::is_none")] + pub image: Option<&'a str>, + #[serde(skip_serializing_if = "Option::is_none")] pub gpu: Option>, } @@ -45,6 +47,22 @@ pub struct GpuAllocation { pub count: u16, } +#[derive(Clone, Debug, Deserialize, Serialize)] +pub struct CustomImagePort { + pub name: String, + pub container_port: u16, + pub protocol: String, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +pub struct CustomImageAllocation { + pub profile: String, + pub display_name: String, + pub mode: String, + #[serde(default)] + pub ports: Vec, +} + #[derive(Clone, Debug, Deserialize, Serialize)] pub struct Devbox { pub name: String, @@ -63,6 +81,8 @@ pub struct Devbox { pub message: Option, #[serde(default)] pub gpu: Option, + #[serde(default)] + pub image: Option, } #[derive(Debug, Deserialize)] @@ -84,6 +104,8 @@ pub struct WhoAmI { #[derive(Debug, Deserialize, Serialize)] pub struct Capabilities { pub gpu: GpuCapabilities, + #[serde(default)] + pub images: CustomImageCapabilities, } #[derive(Debug, Deserialize, Serialize)] @@ -93,6 +115,22 @@ pub struct GpuCapabilities { pub profiles: Vec, } +#[derive(Debug, Default, Deserialize, Serialize)] +pub struct CustomImageCapabilities { + pub enabled: bool, + pub profiles: Vec, +} + +#[derive(Debug, Deserialize, Serialize)] +pub struct CustomImageProfileSummary { + pub name: String, + pub display_name: String, + pub description: Option, + pub mode: String, + #[serde(default)] + pub ports: Vec, +} + #[derive(Debug, Deserialize, Serialize)] pub struct GpuProfileSummary { pub name: String, @@ -265,6 +303,7 @@ mod tests { preset: Preset::Medium, ttl_hours: 24, repository: None, + image: None, gpu: None, }; @@ -286,6 +325,7 @@ mod tests { preset: Preset::Small, ttl_hours: 24, repository: None, + image: None, gpu: Some(GpuRequest { profile: None }), }; let named_gpu = CreateDevbox { @@ -293,6 +333,7 @@ mod tests { preset: Preset::Large, ttl_hours: 72, repository: None, + image: None, gpu: Some(GpuRequest { profile: Some("nvidia-l4"), }), @@ -305,6 +346,23 @@ mod tests { ); } + #[test] + fn create_payload_includes_an_explicit_image_selector() { + let payload = CreateDevbox { + name: "nginx", + preset: Preset::Small, + ttl_hours: 24, + repository: None, + image: Some("docker.io/library/nginx:1.27"), + gpu: None, + }; + + assert_eq!( + serde_json::to_value(payload).unwrap()["image"], + json!("docker.io/library/nginx:1.27") + ); + } + #[test] fn devbox_deserialization_accepts_pre_gpu_controller_responses() { let box_info: Devbox = serde_json::from_value(json!({ diff --git a/controller/README.md b/controller/README.md index 6ebfc2f..76eff13 100644 --- a/controller/README.md +++ b/controller/README.md @@ -13,4 +13,4 @@ uv run pytest For local development, set `DEVBOXES_KUBECONFIG_CONTEXT` to a disposable Kubernetes context and provide a non-production `DEVBOXES_ACCESS_TOKEN`. -See the [API reference](../docs/api.md), [architecture](../docs/architecture.md), [GPU acceleration](../docs/gpu.md), and [operations runbook](../docs/operations.md) for supported behavior and deployment guidance. +See the [API reference](../docs/api.md), [architecture](../docs/architecture.md), [GPU acceleration](../docs/gpu.md), [custom image profiles](../docs/images.md), and [operations runbook](../docs/operations.md) for supported behavior and deployment guidance. diff --git a/controller/src/devboxes_controller/app.py b/controller/src/devboxes_controller/app.py index 16ff769..a4f0f33 100644 --- a/controller/src/devboxes_controller/app.py +++ b/controller/src/devboxes_controller/app.py @@ -53,6 +53,9 @@ CliTokenRequest, CliTokenResponse, CreateDevboxRequest, + CustomImageCapabilities, + CustomImagePortSummary, + CustomImageProfileSummary, DeleteResult, Devbox, DevboxList, @@ -93,7 +96,28 @@ def _capabilities(settings: Settings) -> Capabilities: for profile in settings.gpu_profiles if settings.gpu_enabled ], - ) + ), + images=CustomImageCapabilities( + enabled=settings.custom_images_enabled, + profiles=[ + CustomImageProfileSummary( + name=profile.name, + display_name=profile.display_name, + description=profile.description, + mode=profile.mode, + ports=[ + CustomImagePortSummary( + name=port.name, + container_port=port.container_port, + protocol=port.protocol, + ) + for port in profile.ports + ], + ) + for profile in settings.custom_images + if settings.custom_images_enabled + ], + ), ) @@ -399,6 +423,7 @@ async def dashboard(request: Request) -> Response: "storage_class": settings.storage_class or "cluster default", "workspace_service_type": settings.workspace_service_type, "gpu": capabilities.gpu, + "images": capabilities.images, "version": __version__, }, ) @@ -444,6 +469,7 @@ async def documentation(request: Request) -> Response: ], "insights_enabled": insights.enabled, "gpu": capabilities.gpu, + "images": capabilities.images, "version": __version__, }, ) diff --git a/controller/src/devboxes_controller/config.py b/controller/src/devboxes_controller/config.py index 6bf6602..98d19a4 100644 --- a/controller/src/devboxes_controller/config.py +++ b/controller/src/devboxes_controller/config.py @@ -5,6 +5,7 @@ from typing import Annotated, Literal, Self from urllib.parse import urlsplit +from kubernetes.utils.quantity import parse_quantity from pydantic import BaseModel, ConfigDict, Field, SecretStr, field_validator, model_validator from pydantic_settings import BaseSettings, SettingsConfigDict @@ -15,6 +16,14 @@ SupplementalGroup = Annotated[int, Field(strict=True, ge=1, le=2_147_483_647)] +def _container_image_reference(value: str) -> str: + """Normalize a container image reference without accepting URL syntax.""" + value = value.strip() + if not value or any(character.isspace() for character in value) or "://" in value: + raise ValueError("must be a whitespace-free container image reference without a URL scheme") + return value + + def _valid_dns_subdomain(value: str) -> bool: """Return whether a string follows Kubernetes DNS subdomain syntax.""" return 1 <= len(value) <= 253 and all( @@ -145,13 +154,7 @@ def resource_name_is_extended(cls, value: str) -> str: @classmethod def workspace_image_is_valid(cls, value: str | None) -> str | None: """Reject image values Kubernetes cannot interpret as references.""" - if value is not None and ( - any(character.isspace() for character in value) or "://" in value - ): - raise ValueError( - "must be a whitespace-free container image reference without a URL scheme" - ) - return value + return _container_image_reference(value) if value is not None else None @field_validator("runtime_class_name") @classmethod @@ -186,6 +189,138 @@ def node_selector_is_valid(cls, value: dict[str, str]) -> dict[str, str]: return value +class CustomImagePort(BaseModel): + """Describe one container port exposed only inside a devbox pod.""" + + model_config = ConfigDict(extra="forbid", populate_by_name=True, frozen=True) + + name: str = Field(min_length=1, max_length=63) + container_port: int = Field(alias="containerPort", ge=1024, le=65_535) + protocol: Literal["TCP", "UDP", "SCTP"] = "TCP" + + @field_validator("name") + @classmethod + def name_is_dns_safe(cls, value: str) -> str: + """Keep port names valid when Kubernetes renders the sidecar.""" + value = value.strip().lower() + if not DNS_LABEL_RE.fullmatch(value): + raise ValueError("must be a valid lowercase Kubernetes DNS label") + return value + + +class CustomImageResources(BaseModel): + """Define the bounded compute envelope for an approved service image.""" + + model_config = ConfigDict(extra="forbid", populate_by_name=True, frozen=True) + + cpu_request: str = Field(default="25m", alias="cpuRequest", min_length=1, max_length=32) + memory_request: str = Field(default="32Mi", alias="memoryRequest", min_length=1, max_length=32) + cpu_limit: str = Field(default="500m", alias="cpuLimit", min_length=1, max_length=32) + memory_limit: str = Field(default="512Mi", alias="memoryLimit", min_length=1, max_length=32) + + @field_validator("cpu_request", "memory_request", "cpu_limit", "memory_limit") + @classmethod + def quantity_is_positive(cls, value: str) -> str: + """Reject malformed or non-positive Kubernetes resource quantities.""" + value = value.strip() + try: + if parse_quantity(value) <= 0: + raise ValueError + except ValueError as error: + raise ValueError("must be a positive Kubernetes resource quantity") from error + return value + + @model_validator(mode="after") + def limits_cover_requests(self) -> Self: + """Avoid a catalog entry Kubernetes would reject at scheduling time.""" + if parse_quantity(self.cpu_request) > parse_quantity(self.cpu_limit): + raise ValueError("cpuLimit must be greater than or equal to cpuRequest") + if parse_quantity(self.memory_request) > parse_quantity(self.memory_limit): + raise ValueError("memoryLimit must be greater than or equal to memoryRequest") + return self + + +class CustomImageProfile(BaseModel): + """Define one operator-approved service or complete workspace image.""" + + model_config = ConfigDict(extra="forbid", populate_by_name=True, frozen=True) + + name: str = Field(min_length=1, max_length=40) + display_name: str = Field(alias="displayName", min_length=1, max_length=80) + description: str | None = Field(default=None, max_length=160) + image: str = Field(min_length=1, max_length=512) + mode: Literal["sidecar", "workspace"] = "sidecar" + pull_policy: Literal["Always", "IfNotPresent", "Never"] = Field( + default="IfNotPresent", + alias="pullPolicy", + ) + resources: CustomImageResources | None = None + ports: list[CustomImagePort] = Field(default_factory=list, max_length=8) + + @model_validator(mode="before") + @classmethod + def populate_sidecar_resource_defaults(cls, value: object) -> object: + """Persist sidecar defaults so a resolved snapshot cannot drift later.""" + if not isinstance(value, dict): + return value + if value.get("mode", "sidecar") == "sidecar" and "resources" not in value: + return {**value, "resources": {}} + return value + + @field_validator("name") + @classmethod + def name_is_safe(cls, value: str) -> str: + """Require a compact profile identifier safe for every client surface.""" + value = value.strip().lower() + if not GPU_PROFILE_NAME_RE.fullmatch(value): + raise ValueError( + "use 1-40 lowercase letters, digits, or hyphens; start and end alphanumeric" + ) + return value + + @field_validator("display_name") + @classmethod + def display_name_is_not_blank(cls, value: str) -> str: + """Normalize the user-visible image label.""" + value = value.strip() + if not value: + raise ValueError("must not be blank") + return value + + @field_validator("description", mode="before") + @classmethod + def optional_text_is_normalized(cls, value: object) -> object: + """Trim optional descriptive text and treat blanks as absent.""" + if isinstance(value, str): + return value.strip() or None + return value + + @field_validator("image") + @classmethod + def image_is_valid(cls, value: str) -> str: + """Reject image values Kubernetes cannot interpret as references.""" + return _container_image_reference(value) + + @field_validator("ports") + @classmethod + def ports_are_unique(cls, value: list[CustomImagePort]) -> list[CustomImagePort]: + """Keep container-port declarations unambiguous for status and manifests.""" + names = [port.name for port in value] + bindings = [(port.container_port, port.protocol) for port in value] + if len(names) != len(set(names)): + raise ValueError("must not contain duplicate port names") + if len(bindings) != len(set(bindings)): + raise ValueError("must not contain duplicate port and protocol bindings") + return value + + @model_validator(mode="after") + def mode_has_only_applicable_settings(self) -> Self: + """Reject sidecar-only scheduling knobs on an interactive workspace image.""" + if self.mode == "workspace" and self.resources is not None: + raise ValueError("workspace profiles cannot define sidecar resources") + return self + + class Settings(BaseSettings): """Define validated runtime settings for one Devboxes installation.""" @@ -196,6 +331,8 @@ class Settings(BaseSettings): display_name: str = "operator" cluster_name: str = "Kubernetes" workspace_image: str = "ghcr.io/vicotrbb/devboxes-workspace:latest" + custom_images_enabled: bool = False + custom_images: list[CustomImageProfile] = Field(default_factory=list, max_length=32) workspace_secret_name: str = "devboxes-workspace" # noqa: S105 - Kubernetes Secret name workspace_service_account_name: str = "devboxes-workspace" workspace_priority_class: str | None = None @@ -365,6 +502,14 @@ def settings_are_consistent(self) -> Self: raise ValueError("gpu_enabled requires at least one GPU profile") if self.gpu_default_profile is None: raise ValueError("gpu_enabled requires gpu_default_profile") + custom_image_names = [profile.name for profile in self.custom_images] + if len(custom_image_names) != len(set(custom_image_names)): + raise ValueError("custom image profile names must be unique") + custom_image_references = [profile.image for profile in self.custom_images] + if len(custom_image_references) != len(set(custom_image_references)): + raise ValueError("custom image profile references must be unique") + if self.custom_images_enabled and not custom_image_names: + raise ValueError("custom_images_enabled requires at least one custom image profile") return self def resolve_gpu_profile(self, requested_name: str | None) -> GpuProfile: @@ -378,6 +523,16 @@ def resolve_gpu_profile(self, requested_name: str | None) -> GpuProfile: available = ", ".join(profile.name for profile in self.gpu_profiles) raise ValueError(f"unknown GPU profile {profile_name!r}; available profiles: {available}") + def resolve_custom_image(self, selector: str) -> CustomImageProfile: + """Resolve a profile name or exact approved image reference before pod creation.""" + if not self.custom_images_enabled: + raise ValueError("custom images are disabled by the operator") + for profile in self.custom_images: + if selector in {profile.name, profile.image}: + return profile + available = ", ".join(profile.name for profile in self.custom_images) + raise ValueError(f"unknown custom image {selector!r}; available profiles: {available}") + @lru_cache def get_settings() -> Settings: diff --git a/controller/src/devboxes_controller/insights_store.py b/controller/src/devboxes_controller/insights_store.py index d77f06f..f83a86a 100644 --- a/controller/src/devboxes_controller/insights_store.py +++ b/controller/src/devboxes_controller/insights_store.py @@ -1048,15 +1048,15 @@ def _backup_sync(self, destination: Path) -> None: source.close() def _database_size_sync(self) -> int: - return sum( - candidate.stat().st_size - for candidate in ( - self.path, - Path(f"{self.path}-wal"), - Path(f"{self.path}-shm"), - ) - if candidate.exists() - ) + total = 0 + for candidate in (self.path, Path(f"{self.path}-wal"), Path(f"{self.path}-shm")): + try: + total += candidate.stat().st_size + except FileNotFoundError: + # SQLite may remove or recreate WAL auxiliary files while a + # concurrent read asks for the aggregate database footprint. + continue + return total def _summarize_ai(rows: Iterable[sqlite3.Row]) -> dict[str, Any]: diff --git a/controller/src/devboxes_controller/manager.py b/controller/src/devboxes_controller/manager.py index 184534c..7ca56fd 100644 --- a/controller/src/devboxes_controller/manager.py +++ b/controller/src/devboxes_controller/manager.py @@ -6,7 +6,9 @@ import hmac import logging import uuid -from collections.abc import Iterable +from collections.abc import AsyncIterator, Iterable +from contextlib import asynccontextmanager +from dataclasses import dataclass from datetime import UTC, datetime, timedelta from typing import Any @@ -15,9 +17,11 @@ from kubernetes.utils.quantity import parse_quantity from .auth import Authenticator -from .config import GpuProfile, Settings +from .config import CustomImageProfile, GpuProfile, Settings from .models import ( CreateDevboxRequest, + CustomImageAllocation, + CustomImagePortSummary, DeleteResult, Devbox, DevboxState, @@ -28,6 +32,8 @@ from .resources import ( ANNOTATION_AUTO_STOPPED_AT, ANNOTATION_CREATED_AT, + ANNOTATION_CUSTOM_IMAGE_CONFIG, + ANNOTATION_CUSTOM_IMAGE_PROFILE, ANNOTATION_EXPIRES_AT, ANNOTATION_GPU_CONFIG, ANNOTATION_GPU_COUNT, @@ -61,6 +67,14 @@ class DevboxConflictError(Exception): """Signal that a requested devbox name is already active.""" +@dataclass +class _LifecycleLock: + """Retain one keyed lifecycle lock while callers are using or awaiting it.""" + + lock: asyncio.Lock + references: int = 0 + + class DevboxManager: """Translate lifecycle requests into namespaced Kubernetes resources.""" @@ -76,6 +90,33 @@ def __init__( self.apps = apps_api or client.AppsV1Api() self.core = core_api or client.CoreV1Api() self.authenticator = Authenticator(settings) + self._lifecycle_locks: dict[str, _LifecycleLock] = {} + self._lifecycle_locks_guard = asyncio.Lock() + + @asynccontextmanager + async def _lock_lifecycle(self, resource: str) -> AsyncIterator[None]: + """Serialize conflicting lifecycle and Insights work for one workspace. + + The reference count keeps a lock alive while another task is waiting for it, + without retaining locks forever for deleted ephemeral workspaces. + """ + async with self._lifecycle_locks_guard: + lifecycle_lock = self._lifecycle_locks.setdefault( + resource, + _LifecycleLock(lock=asyncio.Lock()), + ) + lifecycle_lock.references += 1 + try: + async with lifecycle_lock.lock: + yield + finally: + async with self._lifecycle_locks_guard: + lifecycle_lock.references -= 1 + if ( + lifecycle_lock.references == 0 + and self._lifecycle_locks.get(resource) is lifecycle_lock + ): + del self._lifecycle_locks[resource] @staticmethod def _load_config(settings: Settings) -> None: @@ -107,8 +148,32 @@ async def create(self, request: CreateDevboxRequest) -> Devbox: if request.gpu is not None else None ) + custom_image = ( + self.settings.resolve_custom_image(request.image) if request.image is not None else None + ) + if ( + custom_image is not None + and custom_image.mode == "workspace" + and gpu_profile is not None + and gpu_profile.workspace_image is not None + ): + raise ValueError( + "a custom workspace image cannot be combined with a GPU profile " + "that selects a workspace image" + ) name = resource_name(request.name) + async with self._lock_lifecycle(name): + return await self._create_locked(request, gpu_profile, custom_image, name) + + async def _create_locked( + self, + request: CreateDevboxRequest, + gpu_profile: GpuProfile | None, + custom_image: CustomImageProfile | None, + name: str, + ) -> Devbox: + """Create a devbox while holding its lifecycle lock.""" if await self._deployment_exists(name): raise DevboxConflictError(f"devbox {request.name!r} already exists") @@ -168,6 +233,7 @@ async def create(self, request: CreateDevboxRequest) -> Devbox: self.settings.workspace_priority_class, self.settings.image_pull_secret, gpu_profile=gpu_profile, + custom_image=custom_image, instance_id=instance_id, insights_enabled=self.settings.insights_enabled, insights_endpoint=self.settings.insights_controller_url, @@ -275,6 +341,11 @@ async def get(self, name: str) -> Devbox: async def scale(self, name: str, replicas: int) -> Devbox: """Start or stop a devbox while preserving its home volume.""" resource = resource_name(name) + async with self._lock_lifecycle(resource): + return await self._scale_locked(name, resource, replicas) + + async def _scale_locked(self, name: str, resource: str, replicas: int) -> Devbox: + """Scale a devbox while holding its lifecycle lock.""" try: if replicas == 1: deployment = await asyncio.to_thread( @@ -329,59 +400,76 @@ async def reconcile_insights(self) -> builtins.list[str]: label_selector=selector, ) changed: builtins.list[str] = [] - for deployment in deployments.items: - name = deployment.metadata.labels.get(LABEL_NAME) + for listed_deployment in deployments.items: + name = listed_deployment.metadata.labels.get(LABEL_NAME) if not name: continue - if (deployment.spec.replicas or 0) == 0: - prepared = await self._prepare_insights_template(deployment) - if prepared is not deployment: - changed.append(name) - else: - annotations = deployment.metadata.annotations or {} - desired, instance_id = await self._desired_insights_deployment(deployment) - if not _insights_template_matches(deployment, desired) and ( - annotations.get(ANNOTATION_INSIGHTS_STATE) - != InsightsState.RESTART_REQUIRED.value - or annotations.get(ANNOTATION_INSTANCE_ID) != instance_id - ): - await asyncio.to_thread( - self.apps.patch_namespaced_deployment, - deployment.metadata.name, + resource = resource_name(name) + async with self._lock_lifecycle(resource): + try: + deployment = await asyncio.to_thread( + self.apps.read_namespaced_deployment, + resource, self.settings.namespace, - { - "metadata": { - "annotations": { - ANNOTATION_INSIGHTS_STATE: InsightsState.RESTART_REQUIRED.value, - ANNOTATION_INSTANCE_ID: instance_id, - } - } - }, ) - changed.append(name) + except ApiException as error: + if error.status == 404: + continue + raise + if _deletion_in_progress(deployment): + continue + if (deployment.spec.replicas or 0) == 0: + prepared = await self._prepare_insights_template(deployment) + if prepared is not deployment: + changed.append(name) + else: + annotations = deployment.metadata.annotations or {} + desired, instance_id = await self._desired_insights_deployment(deployment) + if not _insights_template_matches(deployment, desired) and ( + annotations.get(ANNOTATION_INSIGHTS_STATE) + != InsightsState.RESTART_REQUIRED.value + or annotations.get(ANNOTATION_INSTANCE_ID) != instance_id + ): + await asyncio.to_thread( + self.apps.patch_namespaced_deployment, + deployment.metadata.name, + self.settings.namespace, + { + "metadata": { + "annotations": { + ANNOTATION_INSIGHTS_STATE: ( + InsightsState.RESTART_REQUIRED.value + ), + ANNOTATION_INSTANCE_ID: instance_id, + } + } + }, + ) + changed.append(name) return changed async def delete(self, name: str, purge: bool) -> DeleteResult: """Delete compute and SSH resources, optionally deleting storage.""" resource = resource_name(name) - if not await self._deployment_exists(resource): - raise DevboxNotFoundError(name) - await asyncio.gather( - self._delete_deployment(resource), - self._delete_service(f"{resource}-ssh"), - self._delete_secret(f"{resource}-insights"), - ) - if purge: - await self._delete_pvc(f"{resource}-home") - return DeleteResult( - name=name, - purged=purge, - message=( - "Devbox and home volume deleted" - if purge - else "Devbox deleted; home volume retained for reuse" - ), - ) + async with self._lock_lifecycle(resource): + if not await self._deployment_exists(resource): + raise DevboxNotFoundError(name) + await asyncio.gather( + self._delete_deployment(resource), + self._delete_service(f"{resource}-ssh"), + self._delete_secret(f"{resource}-insights"), + ) + if purge: + await self._delete_pvc(f"{resource}-home") + return DeleteResult( + name=name, + purged=purge, + message=( + "Devbox and home volume deleted" + if purge + else "Devbox deleted; home volume retained for reuse" + ), + ) async def stop_expired(self) -> builtins.list[str]: """Stop every active devbox whose TTL has expired.""" @@ -565,8 +653,10 @@ async def _desired_insights_deployment(self, deployment: Any) -> tuple[dict[str, self.settings.max_ttl_hours, ), repository=annotations.get(ANNOTATION_REPOSITORY), + image=annotations.get(ANNOTATION_CUSTOM_IMAGE_PROFILE), ) gpu_profile = _resolved_gpu_profile(annotations, self.settings) + custom_image = _resolved_custom_image(annotations, self.settings) credential = self.authenticator.issue_insights_token(instance_id, name) secret_name = await self._ensure_insights_secret(name, instance_id, credential) desired = build_deployment( @@ -578,6 +668,7 @@ async def _desired_insights_deployment(self, deployment: Any) -> tuple[dict[str, self.settings.workspace_priority_class, self.settings.image_pull_secret, gpu_profile=gpu_profile, + custom_image=custom_image, instance_id=instance_id, insights_enabled=True, insights_endpoint=self.settings.insights_controller_url, @@ -659,6 +750,7 @@ def _to_model(self, deployment: Any, service: Any | None, pod: Any | None) -> De storage_size=annotations.get(ANNOTATION_STORAGE, "20Gi"), message=message, gpu=_gpu_allocation(annotations), + image=_custom_image_allocation(annotations), instance_id=annotations.get(ANNOTATION_INSTANCE_ID), insights_state=_insights_state(self.settings.insights_enabled, deployment, desired), ) @@ -671,6 +763,12 @@ def _parse_datetime(value: str | None, fallback: datetime) -> datetime: return fallback if fallback.tzinfo else fallback.replace(tzinfo=UTC) +def _deletion_in_progress(deployment: Any) -> bool: + """Return whether Kubernetes has begun deleting a Deployment.""" + metadata = getattr(deployment, "metadata", None) + return getattr(metadata, "deletion_timestamp", None) is not None + + def _ttl_hours(value: str | None, default: int, maximum: int) -> int: try: parsed = int(value) if value is not None else default @@ -794,6 +892,29 @@ def _resolved_gpu_profile( return None +def _resolved_custom_image( + annotations: dict[str, str], + settings: Settings, +) -> CustomImageProfile | None: + """Recover the pinned custom image profile used by an existing workspace.""" + raw_profile = annotations.get(ANNOTATION_CUSTOM_IMAGE_CONFIG) + if raw_profile: + try: + return CustomImageProfile.model_validate_json(raw_profile) + except ValueError as error: + raise ValueError("stored custom image configuration is invalid") from error + profile_name = annotations.get(ANNOTATION_CUSTOM_IMAGE_PROFILE) + if profile_name: + for profile in settings.custom_images: + if profile.name == profile_name: + return profile + raise ValueError( + f"stored custom image profile {profile_name!r} is unavailable " + "and has no resolved snapshot" + ) + return None + + def _gpu_allocation(annotations: dict[str, str]) -> GpuAllocation | None: """Build a stable user-facing GPU allocation from Deployment annotations.""" profile_name = annotations.get(ANNOTATION_GPU_PROFILE) @@ -824,6 +945,40 @@ def _gpu_allocation(annotations: dict[str, str]) -> GpuAllocation | None: ) +def _custom_image_allocation( + annotations: dict[str, str], +) -> CustomImageAllocation | None: + """Build a stable user-facing custom image allocation from Deployment annotations.""" + raw_profile = annotations.get(ANNOTATION_CUSTOM_IMAGE_CONFIG) + if raw_profile: + try: + profile = CustomImageProfile.model_validate_json(raw_profile) + except ValueError: + profile = None + if profile is not None: + return CustomImageAllocation( + profile=profile.name, + display_name=profile.display_name, + mode=profile.mode, + ports=[ + CustomImagePortSummary( + name=port.name, + container_port=port.container_port, + protocol=port.protocol, + ) + for port in profile.ports + ], + ) + profile_name = annotations.get(ANNOTATION_CUSTOM_IMAGE_PROFILE) + if profile_name: + return CustomImageAllocation( + profile=profile_name, + display_name=profile_name, + mode="unknown", + ) + return None + + def _has_insights_sidecar(deployment: Any) -> bool: containers = getattr( getattr(getattr(deployment, "spec", None), "template", None), diff --git a/controller/src/devboxes_controller/models.py b/controller/src/devboxes_controller/models.py index d8175c5..1d78f12 100644 --- a/controller/src/devboxes_controller/models.py +++ b/controller/src/devboxes_controller/models.py @@ -68,6 +68,7 @@ class CreateDevboxRequest(BaseModel): ttl_hours: int = Field(default=24, ge=1, le=168) repository: str | None = Field(default=None, max_length=240) gpu: GpuRequest | None = None + image: str | None = Field(default=None, max_length=512) @field_validator("name") @classmethod @@ -91,6 +92,20 @@ def valid_repository(cls, value: str | None) -> str | None: raise ValueError("use owner/repository or an https://github.com/owner/repository URL") return value + @field_validator("image") + @classmethod + def valid_image_selector(cls, value: str | None) -> str | None: + """Accept only a compact operator-approved profile or image selector.""" + if value is None or not value.strip(): + return None + value = value.strip() + if any(character.isspace() for character in value) or "://" in value: + raise ValueError( + "use an operator-approved image profile or image reference without " + "whitespace or a URL scheme" + ) + return value + class GpuAllocation(BaseModel): """Describe the pinned GPU allocation attached to a devbox.""" @@ -101,6 +116,23 @@ class GpuAllocation(BaseModel): count: int +class CustomImagePortSummary(BaseModel): + """Describe an application port available only within one devbox pod.""" + + name: str + container_port: int + protocol: str + + +class CustomImageAllocation(BaseModel): + """Describe the resolved image profile pinned to a devbox.""" + + profile: str + display_name: str + mode: str + ports: list[CustomImagePortSummary] = Field(default_factory=list) + + class Devbox(BaseModel): """Represent the observable state of one managed devbox.""" @@ -119,6 +151,7 @@ class Devbox(BaseModel): storage_size: str message: str | None = None gpu: GpuAllocation | None = None + image: CustomImageAllocation | None = None instance_id: str | None = None insights_state: InsightsState = InsightsState.DISABLED @@ -155,10 +188,28 @@ class GpuCapabilities(BaseModel): profiles: list[GpuProfileSummary] +class CustomImageProfileSummary(BaseModel): + """Expose safe user-facing metadata for one custom image profile.""" + + name: str + display_name: str + description: str | None = None + mode: str + ports: list[CustomImagePortSummary] = Field(default_factory=list) + + +class CustomImageCapabilities(BaseModel): + """Describe custom image profiles clients may request for new devboxes.""" + + enabled: bool + profiles: list[CustomImageProfileSummary] + + class Capabilities(BaseModel): """Describe installation capabilities that shape user workflows.""" gpu: GpuCapabilities + images: CustomImageCapabilities class CliTokenRequest(BaseModel): diff --git a/controller/src/devboxes_controller/resources.py b/controller/src/devboxes_controller/resources.py index b9b98a2..1462af0 100644 --- a/controller/src/devboxes_controller/resources.py +++ b/controller/src/devboxes_controller/resources.py @@ -5,7 +5,7 @@ from datetime import UTC, datetime, timedelta from typing import Any -from .config import GpuProfile +from .config import CustomImageProfile, GpuProfile from .models import CreateDevboxRequest, Preset MANAGED_BY = "devboxes-controller" @@ -25,6 +25,8 @@ ANNOTATION_GPU_RESOURCE = "gpu.devboxes.bonalab.org/resource" ANNOTATION_GPU_COUNT = "gpu.devboxes.bonalab.org/count" ANNOTATION_GPU_CONFIG = "gpu.devboxes.bonalab.org/resolved-config" +ANNOTATION_CUSTOM_IMAGE_PROFILE = "image.devboxes.bonalab.org/profile" +ANNOTATION_CUSTOM_IMAGE_CONFIG = "image.devboxes.bonalab.org/resolved-config" PRESETS: dict[Preset, dict[str, str]] = { @@ -70,6 +72,7 @@ def annotations( now: datetime | None = None, instance_id: str | None = None, gpu_profile: GpuProfile | None = None, + custom_image: CustomImageProfile | None = None, ) -> dict[str, str]: """Return lifecycle and user-input annotations for a new devbox.""" now = now or datetime.now(UTC) @@ -97,6 +100,17 @@ def annotations( ), } ) + if custom_image is not None: + result.update( + { + ANNOTATION_CUSTOM_IMAGE_PROFILE: custom_image.name, + ANNOTATION_CUSTOM_IMAGE_CONFIG: json.dumps( + custom_image.model_dump(mode="json", by_alias=True, exclude_none=True), + sort_keys=True, + separators=(",", ":"), + ), + } + ) return result @@ -137,6 +151,7 @@ def build_deployment( workspace_priority_class: str | None = None, image_pull_secret: str | None = None, gpu_profile: GpuProfile | None = None, + custom_image: CustomImageProfile | None = None, now: datetime | None = None, instance_id: str | None = None, insights_enabled: bool = False, @@ -151,11 +166,13 @@ def build_deployment( """Build the disposable workspace Deployment for a devbox.""" name = resource_name(request.name) box_labels = labels(request.name) - effective_workspace_image = ( - gpu_profile.workspace_image - if gpu_profile is not None and gpu_profile.workspace_image is not None - else workspace_image - ) + effective_workspace_image = workspace_image + effective_workspace_pull_policy = "IfNotPresent" + if gpu_profile is not None and gpu_profile.workspace_image is not None: + effective_workspace_image = gpu_profile.workspace_image + if custom_image is not None and custom_image.mode == "workspace": + effective_workspace_image = custom_image.image + effective_workspace_pull_policy = custom_image.pull_policy env = [ {"name": "DEVBOX_NAME", "value": request.name}, {"name": "DEVBOX_PRESET", "value": request.preset.value}, @@ -221,7 +238,7 @@ def build_deployment( { "name": "devbox", "image": effective_workspace_image, - "imagePullPolicy": "IfNotPresent", + "imagePullPolicy": effective_workspace_pull_policy, "ports": [{"name": "ssh", "containerPort": 2222, "protocol": "TCP"}], "env": env, "resources": { @@ -313,6 +330,36 @@ def build_deployment( for toleration in gpu_profile.tolerations ] + if custom_image is not None and custom_image.mode == "sidecar": + if custom_image.resources is None: + raise ValueError("sidecar custom image profile is missing its resource envelope") + sidecar_resources = custom_image.resources + sidecar: dict[str, Any] = { + "name": "custom-image", + "image": custom_image.image, + "imagePullPolicy": custom_image.pull_policy, + "resources": { + "requests": { + "cpu": sidecar_resources.cpu_request, + "memory": sidecar_resources.memory_request, + }, + "limits": { + "cpu": sidecar_resources.cpu_limit, + "memory": sidecar_resources.memory_limit, + }, + }, + "securityContext": { + "runAsNonRoot": True, + "allowPrivilegeEscalation": False, + "capabilities": {"drop": ["ALL"]}, + }, + } + if custom_image.ports: + sidecar["ports"] = [ + port.model_dump(mode="json", by_alias=True) for port in custom_image.ports + ] + pod_spec["containers"].append(sidecar) + if insights_enabled: if not all((instance_id, insights_endpoint, insights_credential)): raise ValueError( @@ -374,7 +421,13 @@ def build_deployment( } ) - deployment_annotations = annotations(request, now, instance_id, gpu_profile) + deployment_annotations = annotations( + request, + now, + instance_id, + gpu_profile, + custom_image, + ) deployment_annotations[ANNOTATION_INSIGHTS_STATE] = ( "collecting" if insights_enabled else "disabled" ) @@ -387,6 +440,8 @@ def build_deployment( ANNOTATION_GPU_COUNT: str(gpu_profile.count), } ) + if custom_image is not None: + template_annotations[ANNOTATION_CUSTOM_IMAGE_PROFILE] = custom_image.name manifest: dict[str, Any] = { "apiVersion": "apps/v1", "kind": "Deployment", diff --git a/controller/src/devboxes_controller/static/app.js b/controller/src/devboxes_controller/static/app.js index 07cacc3..753addc 100644 --- a/controller/src/devboxes_controller/static/app.js +++ b/controller/src/devboxes_controller/static/app.js @@ -21,6 +21,8 @@ const elements = { purgeVolume: document.querySelector("#purge-volume"), confirmDelete: document.querySelector("#confirm-delete"), toastRegion: document.querySelector("#toast-region"), + image: document.querySelector("#image"), + imageHelp: document.querySelector("#image-help"), }; function cookie(name) { @@ -123,6 +125,18 @@ function renderBox(box) { const unit = box.gpu.count === 1 ? "unit" : "units"; details.push(`${box.gpu.display_name} · ${box.gpu.count} ${unit}`); } + if (box.image) { + const mode = + box.image.mode === "sidecar" ? "service image" : "workspace image"; + details.push(`${box.image.display_name} · ${mode}`); + if (box.image.ports.length) { + details.push( + box.image.ports + .map((port) => `${port.name}:${port.container_port}/${port.protocol}`) + .join(", "), + ); + } + } if (box.repository) { details.push(box.repository); } @@ -270,6 +284,7 @@ elements.createForm.addEventListener("submit", async (event) => { ttl_hours: Number(form.get("ttl_hours")), repository: form.get("repository") || null, gpu: form.get("gpu_profile") ? { profile: form.get("gpu_profile") } : null, + image: form.get("image") || null, }; submit.disabled = true; submit.setAttribute("aria-busy", "true"); @@ -291,6 +306,35 @@ elements.createForm.addEventListener("submit", async (event) => { } }); +function updateImageHelp() { + if (!elements.image || !elements.imageHelp) { + return; + } + const selected = elements.image.selectedOptions[0]; + if (!selected?.value) { + elements.imageHelp.textContent = + "Select an operator-approved image profile. Service images run as non-root sidecars on high ports without Secret mounts or Kubernetes credentials."; + return; + } + const mode = selected.dataset.mode; + const details = [selected.dataset.description].filter(Boolean); + if (mode === "workspace") { + details.push( + "This approved image replaces the prepared workspace and preserves its SSH lifecycle contract.", + ); + } else { + details.push( + "This service image runs as a non-root sidecar on high ports without Secret mounts or Kubernetes credentials.", + ); + } + if (selected.dataset.ports) { + details.push(`Pod-local ports: ${selected.dataset.ports}.`); + } + elements.imageHelp.textContent = details.join(" "); +} + +elements.image?.addEventListener("change", updateImageHelp); + elements.rows.addEventListener("click", (event) => { const button = event.target.closest("button[data-action]"); if (!button) { @@ -366,5 +410,6 @@ document.addEventListener("click", (event) => { } }); +updateImageHelp(); loadBoxes(); window.setInterval(() => loadBoxes({ quiet: true }), 8_000); diff --git a/controller/src/devboxes_controller/static/styles.css b/controller/src/devboxes_controller/static/styles.css index f91a37e..e220bcf 100644 --- a/controller/src/devboxes_controller/static/styles.css +++ b/controller/src/devboxes_controller/static/styles.css @@ -362,6 +362,14 @@ h3 { gap: var(--space-3); } +.field-image { + grid-column: span 2; +} + +.field-repository { + grid-column: span 4; +} + .field { display: flex; min-width: 0; @@ -476,8 +484,10 @@ select:focus { } .create-button { + width: 100%; min-width: 8.5rem; min-height: 2.7rem; + grid-column: span 2; } .create-button[aria-busy="true"] span::after { @@ -1424,6 +1434,11 @@ kbd { font-size: 0.78rem; } +.image-profile-help { + grid-column: 1 / -1; + margin: 0; +} + .security-note { margin: var(--space-6) 0 0; color: var(--muted); @@ -1539,6 +1554,14 @@ kbd { grid-column: span 3; } + .create-button { + grid-column: span 1; + } + + .field-image { + grid-column: span 2; + } + .docs-layout { grid-template-columns: 13rem minmax(0, 1fr); gap: var(--space-7); @@ -1615,6 +1638,10 @@ kbd { grid-column: span 2; } + .field-image { + grid-column: span 2; + } + .create-button { width: 100%; grid-column: span 2; @@ -1688,6 +1715,14 @@ kbd { grid-column: auto; } + .field-image { + grid-column: auto; + } + + .image-profile-help { + grid-column: auto; + } + .create-button { width: 100%; grid-column: auto; diff --git a/controller/src/devboxes_controller/templates/docs.html b/controller/src/devboxes_controller/templates/docs.html index 1deef90..b8edaa0 100644 --- a/controller/src/devboxes_controller/templates/docs.html +++ b/controller/src/devboxes_controller/templates/docs.html @@ -38,6 +38,7 @@ What persists Create a box GPU acceleration + Approved images SSH and tmux Accounts and tools Daily workflow @@ -280,6 +281,65 @@

Use an operator-approved GPU

{% endif %} +
+

Use an operator-approved image

+ {% if images.enabled %} +

+ This installation exposes {{ images.profiles | length }} approved image profile{% if images.profiles | length != 1 %}s{% endif %}. + Select a profile with --image or in the workbench. A service profile runs beside + the prepared SSH workspace without its Secret mounts or Kubernetes credentials. A workspace + profile is a vetted Devboxes-derived image that replaces the interactive container. +

+ +
+ + + + + + + + + + + {% for profile in images.profiles %} + + + + + + + {% endfor %} + +
ProfileModePod-local portsPurpose
{{ profile.name }}{{ "Service sidecar" if profile.mode == "sidecar" else "Workspace" }}{% if profile.ports %}{% for port in profile.ports %}{{ port.name }}:{{ port.container_port }}{% if not loop.last %}, {% endif %}{% endfor %}{% else %}None declared{% endif %}{{ profile.description or profile.display_name }}
+
+ +
+
+ Discover and request an approved image + +
+ 
devbox image profiles
+{% if images.profiles %}devbox create preview --image {{ images.profiles[0].name }} --ssh{% else %}devbox create preview --image PROFILE --ssh{% endif %}
+
+ +

+ The selected profile is resolved before any Kubernetes resource is created and is pinned to + the devbox Deployment. Service sidecars must run non-root and use a declared port from 1024 + through 65535; their generated containers receive no Devboxes credentials or volume mounts. + Stop, start, TTL expiry, and Insights reconciliation retain the same contract. For a declared + service port, tunnel through SSH, for example + devbox ssh preview -- -L 8080:127.0.0.1:8080. Do not use a generic application + image as a workspace profile unless it preserves the complete Devboxes SSH and lifecycle contract. +

+ {% else %} +

+ Approved custom images are disabled for this installation. An operator can configure a + reviewed image catalog through Helm. The API does not accept unrestricted image references. +

+ {% endif %} +
+

Work through SSH and tmux

@@ -444,6 +504,7 @@

Use the browser workbench

  • The fleet refreshes automatically every eight seconds; manual refresh is also available.
    • +
  • When enabled, the create form presents reviewed image profiles rather than accepting a raw container reference.
  • State is written as text, ready, starting, stopped, or degraded, rather than conveyed by color alone.
  • Deleting opens a confirmation that distinguishes retained storage from permanent purge.
  • The same access token creates a signed browser session; the raw token is not kept in the session cookie.
    • @@ -464,7 +525,8 @@

    Command reference

    devbox loginVerifies and saves API access.--url, --token - devbox create NAMECreates a prepared box and waits for readiness.--preset, --ttl, --repo, --ssh, --no-wait + devbox create NAMECreates a prepared box and waits for readiness.--preset, --ttl, --repo, --image, --ssh, --no-wait + devbox image profilesLists the operator-approved image catalog.--json devbox listLists every active or stopped box.--json devbox status NAMEShows one box, its expiry, storage, and SSH state.--json devbox ssh NAMEConnects and attaches to tmux session main.-- <ssh options> @@ -492,6 +554,10 @@

    Troubleshooting

    A box stays in “starting”

    Allow the first image pull and workspace initialization a few minutes. If it remains pending, inspect the dashboard message, SSH service address availability, and pod events.

    +
    + An approved image will not start +

    Check the selected profile in devbox image profiles, then inspect the pod events and the custom-image container logs. Confirm the operator-approved image can pull on the selected node and that its declared application port matches the tunnel target.

    +
    The repository did not clone

    Connect to the empty box, run gh auth status, then retry with gh repo clone OWNER/REPOSITORY ~/workspace/project. Confirm the shared GitHub token can read the repository.

    diff --git a/controller/src/devboxes_controller/templates/index.html b/controller/src/devboxes_controller/templates/index.html index 0f352d6..48fa1e8 100644 --- a/controller/src/devboxes_controller/templates/index.html +++ b/controller/src/devboxes_controller/templates/index.html @@ -55,7 +55,7 @@

    Create a devbox

    Compute expires into a safe stopped state. Your volume is never deleted automatically.

    - {{ storage_class }} storage · {{ workspace_service_type }} SSH{% if gpu.enabled %} · GPU profiles{% endif %} + {{ storage_class }} storage · {{ workspace_service_type }} SSH{% if gpu.enabled %} · GPU profiles{% endif %}{% if images.enabled %} · approved images{% endif %} @@ -102,6 +102,22 @@

    Create a devbox

    {% endfor %} + {% if images.enabled %} +
    + + +
    + {% endif %}
    @@ -109,6 +125,11 @@

    Create a devbox

    + {% if images.enabled %} +

    + Select an operator-approved image profile. Service images run as non-root sidecars on high ports without Secret mounts or Kubernetes credentials. +

    + {% endif %}
diff --git a/controller/tests/fakes.py b/controller/tests/fakes.py index dcea097..220b305 100644 --- a/controller/tests/fakes.py +++ b/controller/tests/fakes.py @@ -5,6 +5,8 @@ from devboxes_controller.manager import DevboxConflictError, DevboxNotFoundError from devboxes_controller.models import ( CreateDevboxRequest, + CustomImageAllocation, + CustomImagePortSummary, DeleteResult, Devbox, DevboxState, @@ -91,6 +93,22 @@ async def create(self, request: CreateDevboxRequest) -> Devbox: resource_name=resource_name, count=1, ) + if request.image is not None: + profile = request.image + display_name, mode = { + "nginx": ("NGINX preview", "sidecar"), + "rust-nightly": ("Rust nightly", "workspace"), + }.get(profile, (profile, "sidecar")) + box.image = CustomImageAllocation( + profile=profile, + display_name=display_name, + mode=mode, + ports=( + [CustomImagePortSummary(name="http", container_port=8080, protocol="TCP")] + if profile == "nginx" + else [] + ), + ) box.ssh_host = None box.ssh_command = None box.pod_ready = False diff --git a/controller/tests/preview_app.py b/controller/tests/preview_app.py index 82a76c7..199a726 100644 --- a/controller/tests/preview_app.py +++ b/controller/tests/preview_app.py @@ -1,5 +1,5 @@ from devboxes_controller.app import create_app -from devboxes_controller.config import GpuProfile, Settings +from devboxes_controller.config import CustomImagePort, CustomImageProfile, GpuProfile, Settings from .fakes import FakeManager @@ -25,5 +25,16 @@ count=1, ), ], + custom_images_enabled=True, + custom_images=[ + CustomImageProfile( + name="nginx", + displayName="NGINX preview", + description="Serve a local static-site preview over the pod network", + image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + mode="sidecar", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + ], ) app = create_app(settings, FakeManager()) # type: ignore[arg-type] diff --git a/controller/tests/test_app.py b/controller/tests/test_app.py index 08f8787..cb024a8 100644 --- a/controller/tests/test_app.py +++ b/controller/tests/test_app.py @@ -4,7 +4,7 @@ from devboxes_controller.app import create_app from devboxes_controller.auth import pkce_s256 -from devboxes_controller.config import GpuProfile, Settings +from devboxes_controller.config import CustomImagePort, CustomImageProfile, GpuProfile, Settings from .fakes import FakeManager @@ -243,7 +243,8 @@ def test_gpu_capabilities_expose_only_safe_profile_metadata() -> None: "default": True, } ], - } + }, + "images": {"enabled": False, "profiles": []}, } assert "workspaceImage" not in response.text assert "runtimeClassName" not in response.text @@ -262,6 +263,63 @@ def test_gpu_capabilities_expose_only_safe_profile_metadata() -> None: assert "devbox create inference --gpu --ssh" in documentation.text +def test_custom_image_capabilities_expose_only_safe_profile_metadata() -> None: + settings = Settings( + access_token="test-access-token-at-least-32-characters", + cookie_secure=False, + cleanup_interval_seconds=3600, + custom_images_enabled=True, + custom_images=[ + CustomImageProfile( + name="nginx", + displayName="NGINX preview", + description="Serve a local static-site preview", + image="private.example/nginx:1.27.5", + pullPolicy="Always", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + ], + ) + headers = {"Authorization": "Bearer test-access-token-at-least-32-characters"} + + with app_client(settings) as client: + response = client.get("/api/v1/capabilities", headers=headers) + created = client.post( + "/api/v1/devboxes", + headers=headers, + json={"name": "preview", "image": "nginx"}, + ) + browser_login(client) + dashboard = client.get("/") + documentation = client.get("/docs") + + assert response.status_code == 200 + assert response.json()["images"] == { + "enabled": True, + "profiles": [ + { + "name": "nginx", + "display_name": "NGINX preview", + "description": "Serve a local static-site preview", + "mode": "sidecar", + "ports": [{"name": "http", "container_port": 8080, "protocol": "TCP"}], + } + ], + } + assert "private.example/nginx:1.27.5" not in response.text + assert "pullPolicy" not in response.text + assert created.status_code == 201 + assert created.json()["image"] == { + "profile": "nginx", + "display_name": "NGINX preview", + "mode": "sidecar", + "ports": [{"name": "http", "container_port": 8080, "protocol": "TCP"}], + } + assert 'value="nginx"' in dashboard.text + assert "approved images" in dashboard.text + assert "Use an operator-approved image" in documentation.text + + def test_api_rejects_invalid_path_names_before_kubernetes() -> None: with app_client() as client: response = client.get( diff --git a/controller/tests/test_config.py b/controller/tests/test_config.py index f4de196..99b6de6 100644 --- a/controller/tests/test_config.py +++ b/controller/tests/test_config.py @@ -3,7 +3,14 @@ import pytest from pydantic import ValidationError -from devboxes_controller.config import GpuProfile, GpuToleration, Settings +from devboxes_controller.config import ( + CustomImagePort, + CustomImageProfile, + CustomImageResources, + GpuProfile, + GpuToleration, + Settings, +) def test_access_token_is_required() -> None: @@ -204,3 +211,106 @@ def test_gpu_profiles_load_from_the_helm_json_environment(monkeypatch: pytest.Mo settings = Settings(_env_file=None) assert settings.resolve_gpu_profile(None).description == "Dedicated inference" + + +def test_custom_image_profiles_are_disabled_by_default_and_resolve_exact_images() -> None: + profile = CustomImageProfile( + name="nginx", + displayName="NGINX preview", + image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + settings = Settings( + access_token="test-access-token-at-least-32-characters", + custom_images_enabled=True, + custom_images=[profile], + _env_file=None, + ) + + assert settings.resolve_custom_image("nginx") is profile + assert settings.resolve_custom_image(profile.image) is profile + + disabled = Settings( + access_token="test-access-token-at-least-32-characters", + custom_images=[profile], + _env_file=None, + ) + with pytest.raises(ValueError, match="disabled by the operator"): + disabled.resolve_custom_image("nginx") + + +def test_custom_image_profiles_reject_unsafe_or_ambiguous_contracts() -> None: + with pytest.raises(ValidationError, match="container image reference"): + CustomImageProfile( + name="bad-image", + displayName="Bad image", + image="https://registry.example/nginx:latest", + ) + + with pytest.raises(ValidationError, match="duplicate port"): + CustomImageProfile( + name="duplicate-port", + displayName="Duplicate port", + image="registry.example/duplicate:1", + ports=[ + CustomImagePort(name="http", containerPort=8080), + CustomImagePort(name="https", containerPort=8080), + ], + ) + + with pytest.raises(ValidationError, match="cpuLimit"): + CustomImageResources(cpuRequest="1", cpuLimit="500m") + + with pytest.raises(ValidationError, match="greater than or equal to 1024"): + CustomImagePort(name="http", containerPort=80) + + with pytest.raises(ValidationError, match="cannot define sidecar resources"): + CustomImageProfile( + name="workspace-with-resources", + displayName="Workspace with resources", + image="registry.example/devboxes-workspace:1", + mode="workspace", + resources=CustomImageResources(), + ) + + profile = CustomImageProfile( + name="nginx", + displayName="NGINX", + image="registry.example/nginx:1", + ) + with pytest.raises(ValidationError, match="references must be unique"): + Settings( + access_token="test-access-token-at-least-32-characters", + custom_images=[profile, profile.model_copy(update={"name": "nginx-copy"})], + _env_file=None, + ) + + +def test_custom_image_profiles_load_from_the_helm_json_environment( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("DEVBOXES_ACCESS_TOKEN", "test-access-token-at-least-32-characters") + monkeypatch.setenv("DEVBOXES_CUSTOM_IMAGES_ENABLED", "true") + monkeypatch.setenv( + "DEVBOXES_CUSTOM_IMAGES", + json.dumps( + [ + { + "name": "nginx", + "displayName": " NGINX preview ", + "description": " Serve static content ", + "image": "docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + "mode": "sidecar", + "ports": [{"name": "http", "containerPort": 8080}], + } + ] + ), + ) + + settings = Settings(_env_file=None) + profile = settings.resolve_custom_image("nginx") + + assert profile.display_name == "NGINX preview" + assert profile.description == "Serve static content" + assert profile.resources is not None + assert profile.ports[0].container_port == 8080 diff --git a/controller/tests/test_insights_service.py b/controller/tests/test_insights_service.py index 0d825c4..9de812b 100644 --- a/controller/tests/test_insights_service.py +++ b/controller/tests/test_insights_service.py @@ -106,8 +106,10 @@ def test_service_ingests_gzip_and_builds_all_query_envelopes(tmp_path: Path) -> assert asyncio.run(service.ready()) is True filters = service.filters( - since="7d", - until=None, + # Keep this envelope test tied to the fixed fixture timestamps instead + # of letting a rolling relative range silently age the points out. + since="2026-07-14T19:00:00Z", + until="2026-07-14T19:20:00Z", box=None, provider=None, model=None, diff --git a/controller/tests/test_insights_store.py b/controller/tests/test_insights_store.py index 1b94d3a..b40cea9 100644 --- a/controller/tests/test_insights_store.py +++ b/controller/tests/test_insights_store.py @@ -4,6 +4,7 @@ from contextlib import closing from datetime import UTC, datetime, timedelta from pathlib import Path +from types import SimpleNamespace import pytest @@ -85,6 +86,30 @@ def test_store_migrates_readies_and_creates_an_online_backup(tmp_path: Path) -> assert copy.execute("SELECT MAX(version) FROM schema_migrations").fetchone()[0] == 1 +def test_database_size_tolerates_a_disappearing_sqlite_auxiliary_file( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = InsightsStore(tmp_path / "insights.db") + wal_path = Path(f"{store.path}-wal") + shm_path = Path(f"{store.path}-shm") + sizes: dict[Path, int | FileNotFoundError] = { + store.path: 100, + wal_path: FileNotFoundError(), + shm_path: 20, + } + + def stat(path: Path, *args: object, **kwargs: object) -> SimpleNamespace: + result = sizes[path] + if isinstance(result, FileNotFoundError): + raise result + return SimpleNamespace(st_size=result) + + monkeypatch.setattr(Path, "stat", stat) + + assert store._database_size_sync() == 120 + + def test_store_migrates_a_previous_empty_schema_transactionally(tmp_path: Path) -> None: database = tmp_path / "insights.db" with closing(sqlite3.connect(database)) as connection: diff --git a/controller/tests/test_manager.py b/controller/tests/test_manager.py index 3a6ef04..1c585d3 100644 --- a/controller/tests/test_manager.py +++ b/controller/tests/test_manager.py @@ -6,12 +6,14 @@ import pytest from kubernetes.client.exceptions import ApiException -from devboxes_controller.config import GpuProfile, Settings +from devboxes_controller.config import CustomImagePort, CustomImageProfile, GpuProfile, Settings from devboxes_controller.manager import ( DevboxConflictError, DevboxManager, DevboxNotFoundError, + _custom_image_allocation, _gpu_allocation, + _resolved_custom_image, _resolved_gpu_profile, _service_endpoint, _state, @@ -20,6 +22,8 @@ from devboxes_controller.models import CreateDevboxRequest, DevboxState, GpuRequest, Preset from devboxes_controller.resources import ( ANNOTATION_CREATED_AT, + ANNOTATION_CUSTOM_IMAGE_CONFIG, + ANNOTATION_CUSTOM_IMAGE_PROFILE, ANNOTATION_EXPIRES_AT, ANNOTATION_GPU_CONFIG, ANNOTATION_GPU_PROFILE, @@ -375,6 +379,93 @@ def test_create_rejects_gpu_requests_when_the_feature_is_disabled() -> None: core.create_namespaced_persistent_volume_claim.assert_not_called() +def test_create_resolves_an_approved_custom_image_before_kubernetes_writes() -> None: + apps = Mock() + apps.read_namespaced_deployment.side_effect = ApiException(status=404) + core = Mock() + core.read_namespaced_persistent_volume_claim.side_effect = ApiException(status=404) + profile = CustomImageProfile( + name="nginx", + displayName="NGINX preview", + image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + manager = DevboxManager( + Settings( + access_token="test-access-token-at-least-32-characters", + custom_images_enabled=True, + custom_images=[profile], + ), + apps_api=apps, + core_api=core, + ) + manager.get = AsyncMock(return_value=Mock()) # type: ignore[method-assign] + + asyncio.run( + manager.create( + CreateDevboxRequest( + name="nginx", image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine" + ) + ) + ) + + deployment = apps.create_namespaced_deployment.call_args.args[1] + assert deployment["metadata"]["annotations"][ANNOTATION_CUSTOM_IMAGE_PROFILE] == "nginx" + assert [ + container["name"] for container in deployment["spec"]["template"]["spec"]["containers"] + ] == [ + "devbox", + "custom-image", + ] + + +def test_create_rejects_disabled_or_conflicting_custom_workspace_images() -> None: + apps = Mock() + core = Mock() + disabled = DevboxManager( + Settings(access_token="test-access-token-at-least-32-characters"), + apps_api=apps, + core_api=core, + ) + with pytest.raises(ValueError, match="disabled by the operator"): + asyncio.run(disabled.create(CreateDevboxRequest(name="nginx", image="nginx"))) + + gpu = GpuProfile( + name="nvidia-l4", + displayName="NVIDIA L4", + resourceName="nvidia.com/gpu", + count=1, + workspaceImage="registry.example/devboxes-cuda:12.8", + ) + workspace = CustomImageProfile( + name="rust-nightly", + displayName="Rust nightly", + image="registry.example/devboxes-rust:nightly", + mode="workspace", + ) + conflicting = DevboxManager( + Settings( + access_token="test-access-token-at-least-32-characters", + gpu_enabled=True, + gpu_default_profile="nvidia-l4", + gpu_profiles=[gpu], + custom_images_enabled=True, + custom_images=[workspace], + ), + apps_api=apps, + core_api=core, + ) + with pytest.raises(ValueError, match="cannot be combined"): + asyncio.run( + conflicting.create( + CreateDevboxRequest(name="rust", gpu=GpuRequest(), image="rust-nightly") + ) + ) + + apps.read_namespaced_deployment.assert_not_called() + core.create_namespaced_persistent_volume_claim.assert_not_called() + + def _legacy_deployment(replicas: int) -> SimpleNamespace: now = datetime.now(UTC) return SimpleNamespace( @@ -422,6 +513,31 @@ def test_resolved_gpu_profile_snapshot_survives_later_operator_changes() -> None } +def test_resolved_custom_image_snapshot_survives_later_operator_changes() -> None: + profile = CustomImageProfile( + name="nginx", + displayName="NGINX preview", + image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + annotations = { + ANNOTATION_CUSTOM_IMAGE_PROFILE: profile.name, + ANNOTATION_CUSTOM_IMAGE_CONFIG: profile.model_dump_json( + by_alias=True, + exclude_none=True, + ), + } + settings = Settings(access_token="test-access-token-at-least-32-characters") + + assert _resolved_custom_image(annotations, settings) == profile + assert _custom_image_allocation(annotations).model_dump() == { + "profile": "nginx", + "display_name": "NGINX preview", + "mode": "sidecar", + "ports": [{"name": "http", "container_port": 8080, "protocol": "TCP"}], + } + + def test_insights_create_scopes_identity_to_the_retained_home_volume() -> None: apps = Mock() apps.read_namespaced_deployment.side_effect = ApiException(status=404) @@ -462,6 +578,7 @@ def test_active_legacy_workspace_is_marked_restart_required_without_template_pat deployment = _legacy_deployment(1) apps = Mock() apps.list_namespaced_deployment.return_value = SimpleNamespace(items=[deployment]) + apps.read_namespaced_deployment.return_value = deployment core = Mock() core.read_namespaced_persistent_volume_claim.return_value = SimpleNamespace( metadata=SimpleNamespace(annotations={}) @@ -484,11 +601,67 @@ def test_active_legacy_workspace_is_marked_restart_required_without_template_pat core.patch_namespaced_persistent_volume_claim.assert_called_once() +def test_insights_reconciliation_skips_a_stale_deployment_after_delete() -> None: + deployment = _legacy_deployment(1) + apps = Mock() + apps.list_namespaced_deployment.return_value = SimpleNamespace(items=[deployment]) + apps.read_namespaced_deployment.side_effect = ApiException(status=404) + core = Mock() + manager = DevboxManager( + Settings( + access_token="test-access-token-at-least-32-characters", + insights_enabled=True, + ), + apps_api=apps, + core_api=core, + ) + + assert asyncio.run(manager.reconcile_insights()) == [] + + core.create_namespaced_secret.assert_not_called() + apps.patch_namespaced_deployment.assert_not_called() + + +def test_lifecycle_lock_serializes_waiters_and_releases_deleted_workspace_entries() -> None: + manager = DevboxManager( + Settings(access_token="test-access-token-at-least-32-characters"), + apps_api=Mock(), + core_api=Mock(), + ) + + async def exercise_lock() -> None: + entered = asyncio.Event() + release = asyncio.Event() + second_acquired = False + + async def first() -> None: + async with manager._lock_lifecycle("devbox-atlas"): + entered.set() + await release.wait() + + async def second() -> None: + nonlocal second_acquired + async with manager._lock_lifecycle("devbox-atlas"): + second_acquired = True + + first_task = asyncio.create_task(first()) + await entered.wait() + second_task = asyncio.create_task(second()) + await asyncio.sleep(0) + assert second_acquired is False + release.set() + await asyncio.gather(first_task, second_task) + + asyncio.run(exercise_lock()) + + assert manager._lifecycle_locks == {} + + def test_stopped_legacy_workspace_template_is_reconciled_without_starting_it() -> None: deployment = _legacy_deployment(0) apps = Mock() apps.list_namespaced_deployment.return_value = SimpleNamespace(items=[deployment]) - apps.read_namespaced_deployment.return_value = _legacy_deployment(0) + apps.read_namespaced_deployment.side_effect = [deployment, _legacy_deployment(0)] core = Mock() core.read_namespaced_persistent_volume_claim.return_value = SimpleNamespace( metadata=SimpleNamespace( @@ -535,7 +708,7 @@ def test_insights_reconciliation_uses_the_pinned_gpu_snapshot() -> None: ) apps = Mock() apps.list_namespaced_deployment.return_value = SimpleNamespace(items=[deployment]) - apps.read_namespaced_deployment.return_value = _legacy_deployment(0) + apps.read_namespaced_deployment.side_effect = [deployment, _legacy_deployment(0)] core = Mock() core.read_namespaced_persistent_volume_claim.return_value = SimpleNamespace( metadata=SimpleNamespace( diff --git a/controller/tests/test_models.py b/controller/tests/test_models.py index 4ef8f3f..4c8d4a4 100644 --- a/controller/tests/test_models.py +++ b/controller/tests/test_models.py @@ -39,3 +39,14 @@ def test_gpu_request_rejects_raw_kubernetes_configuration() -> None: name="atlas", gpu={"profile": "nvidia", "resource_name": "nvidia.com/gpu"}, ) + + +def test_custom_image_selector_is_compact_and_does_not_accept_url_syntax() -> None: + request = CreateDevboxRequest( + name="nginx", + image=" docker.io/library/nginx:1.27.5-alpine ", + ) + + assert request.image == "docker.io/library/nginx:1.27.5-alpine" + with pytest.raises(ValidationError, match="image profile"): + CreateDevboxRequest(name="nginx", image="https://registry.example/nginx:latest") diff --git a/controller/tests/test_resources.py b/controller/tests/test_resources.py index a35e3c5..136333d 100644 --- a/controller/tests/test_resources.py +++ b/controller/tests/test_resources.py @@ -1,10 +1,18 @@ +import json from datetime import UTC, datetime import pytest -from devboxes_controller.config import GpuProfile, GpuToleration +from devboxes_controller.config import ( + CustomImagePort, + CustomImageProfile, + GpuProfile, + GpuToleration, +) from devboxes_controller.models import CreateDevboxRequest, Preset from devboxes_controller.resources import ( + ANNOTATION_CUSTOM_IMAGE_CONFIG, + ANNOTATION_CUSTOM_IMAGE_PROFILE, ANNOTATION_GPU_CONFIG, ANNOTATION_GPU_COUNT, ANNOTATION_GPU_PROFILE, @@ -198,3 +206,84 @@ def test_gpu_profile_is_applied_only_to_the_workspace_container() -> None: assert "nvidia.com/gpu" not in pod["containers"][1]["resources"]["requests"] assert "nvidia.com/gpu" not in pod["containers"][1]["resources"]["limits"] assert pod["containers"][1]["image"] == "ghcr.io/vicotrbb/devboxes-workspace:test" + + +def test_custom_service_image_is_an_isolated_sidecar_with_pinned_contract() -> None: + profile = CustomImageProfile( + name="nginx", + displayName="NGINX preview", + image="docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + pullPolicy="Always", + ports=[CustomImagePort(name="http", containerPort=8080)], + ) + deployment = build_deployment( + request(), + "devboxes", + "ghcr.io/vicotrbb/devboxes-workspace:test", + "devboxes-workspace", + "devboxes-workspace", + custom_image=profile, + ) + annotations = deployment["metadata"]["annotations"] + pod = deployment["spec"]["template"]["spec"] + main, sidecar = pod["containers"] + + assert annotations[ANNOTATION_CUSTOM_IMAGE_PROFILE] == "nginx" + assert '"mode":"sidecar"' in annotations[ANNOTATION_CUSTOM_IMAGE_CONFIG] + assert json.loads(annotations[ANNOTATION_CUSTOM_IMAGE_CONFIG])["resources"] == { + "cpuRequest": "25m", + "memoryRequest": "32Mi", + "cpuLimit": "500m", + "memoryLimit": "512Mi", + } + assert ( + deployment["spec"]["template"]["metadata"]["annotations"][ANNOTATION_CUSTOM_IMAGE_PROFILE] + == "nginx" + ) + assert main["image"] == "ghcr.io/vicotrbb/devboxes-workspace:test" + assert sidecar == { + "name": "custom-image", + "image": "docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine", + "imagePullPolicy": "Always", + "resources": { + "requests": {"cpu": "25m", "memory": "32Mi"}, + "limits": {"cpu": "500m", "memory": "512Mi"}, + }, + "securityContext": { + "runAsNonRoot": True, + "allowPrivilegeEscalation": False, + "capabilities": {"drop": ["ALL"]}, + }, + "ports": [{"name": "http", "containerPort": 8080, "protocol": "TCP"}], + } + assert all( + "workspace-secrets" not in mount["name"] for mount in sidecar.get("volumeMounts", []) + ) + + +def test_custom_workspace_image_replaces_only_the_interactive_container() -> None: + profile = CustomImageProfile( + name="rust-nightly", + displayName="Rust nightly", + image="registry.example/devboxes-workspace-rust:nightly", + mode="workspace", + pullPolicy="Always", + ) + deployment = build_deployment( + request(), + "devboxes", + "ghcr.io/vicotrbb/devboxes-workspace:test", + "devboxes-workspace", + "devboxes-workspace", + custom_image=profile, + insights_enabled=True, + instance_id="99999999-9999-4999-8999-999999999999", + insights_endpoint="http://devboxes:8000", + insights_credential="v1.99999999-9999-4999-8999-999999999999.atlas." + "a" * 43, + ) + main, insights = deployment["spec"]["template"]["spec"]["containers"] + + assert main["image"] == "registry.example/devboxes-workspace-rust:nightly" + assert main["imagePullPolicy"] == "Always" + assert insights["name"] == "insights-agent" + assert insights["image"] == "ghcr.io/vicotrbb/devboxes-workspace:test" diff --git a/docs/api.md b/docs/api.md index 2b96549..4ef6cc8 100644 --- a/docs/api.md +++ b/docs/api.md @@ -29,7 +29,7 @@ The shared token controls every devbox in the installation, including permanent | `POST` | `/auth/cli/authorize` | 303 | Approve or deny a CSRF-protected CLI request | | `POST` | `/api/v1/auth/cli/token` | 200 | Exchange a one-time code and PKCE verifier | | `GET` | `/api/v1/whoami` | 200 | Verify authentication and identity | -| `GET` | `/api/v1/capabilities` | 200 | Discover installation GPU profiles | +| `GET` | `/api/v1/capabilities` | 200 | Discover installation GPU and custom image profiles | | `GET` | `/api/v1/devboxes` | 200 | List managed devboxes | | `POST` | `/api/v1/devboxes` | 201 | Create a devbox | | `GET` | `/api/v1/devboxes/{name}` | 200 | Read one devbox | @@ -54,7 +54,7 @@ responses are `no-store`, codes are single-use, and no refresh token is returned ## Installation capabilities -Authenticated clients use `GET /api/v1/capabilities` to discover optional installation features. GPU capability discovery returns only the safe user contract, not profile images, RuntimeClasses, supplemental groups, selectors, or tolerations: +Authenticated clients use `GET /api/v1/capabilities` to discover optional installation features. Capability discovery returns only the safe user contract, not profile images, pull policies, resource limits, RuntimeClasses, supplemental groups, selectors, or tolerations: ```json { @@ -71,11 +71,25 @@ Authenticated clients use `GET /api/v1/capabilities` to discover optional instal "default": true } ] + }, + "images": { + "enabled": true, + "profiles": [ + { + "name": "nginx", + "display_name": "NGINX preview", + "description": "Serve a local static-site preview", + "mode": "sidecar", + "ports": [ + {"name": "http", "container_port": 8080, "protocol": "TCP"} + ] + } + ] } } ``` -When GPU support is disabled, `enabled` is false, `default_profile` is null, and `profiles` is empty. Clients should treat new top-level capabilities as additive. +When GPU support is disabled, `gpu.enabled` is false, `default_profile` is null, and its `profiles` is empty. When custom images are disabled, `images.enabled` is false and its `profiles` is empty. Clients should treat new top-level capabilities as additive. ## Create a devbox @@ -103,6 +117,7 @@ Request fields: | `repository` | string or null | `owner/repository` or an HTTPS GitHub repository URL | | `gpu` | object or null | Optional GPU request; omit or use null for CPU-only | | `gpu.profile` | string or null | Configured profile name; null selects the operator default | +| `image` | string or null | Operator-approved image profile name or its exact configured image reference | Unknown fields are rejected. Creating an existing name returns `409 Conflict`. Recreating a deleted name reuses its retained PVC, and expands it when the new preset requests more storage. @@ -119,6 +134,8 @@ Request the operator's default GPU profile with an empty nested object: Select an exact profile with `"gpu": {"profile": "nvidia-l4"}`. The controller rejects GPU requests while the feature is disabled and rejects unknown profile names before creating any Kubernetes resource. Clients cannot send resource names, counts, images, RuntimeClasses, supplemental groups, selectors, or tolerations. Read [GPU acceleration](gpu.md) for the operator contract. +Select a custom image with `"image": "nginx"`. The controller resolves the selector against the enabled catalog before it creates the PVC, Deployment, or SSH Service. A selector may be the stable profile name or an exact configured image reference; a raw unapproved reference is rejected. Clients cannot send a command, volume, Service, resource envelope, port mapping, capability, or scheduling policy. Read [custom image profiles](images.md) for sidecar and workspace requirements. + ## Devbox response ```json @@ -137,7 +154,8 @@ Select an exact profile with `"gpu": {"profile": "nvidia-l4"}`. The controller r "restarts": 0, "storage_size": "30Gi", "message": null, - "gpu": null + "gpu": null, + "image": null } ``` @@ -154,6 +172,21 @@ GPU boxes return their resolved allocation: } ``` +Custom image boxes return their resolved user-facing allocation: + +```json +{ + "image": { + "profile": "nginx", + "display_name": "NGINX preview", + "mode": "sidecar", + "ports": [ + {"name": "http", "container_port": 8080, "protocol": "TCP"} + ] + } +} +``` + States are: - `starting`, the pod or SSH address is not ready. @@ -161,7 +194,7 @@ States are: - `stopped`, the Deployment has zero replicas and the home volume remains. - `degraded`, the pod failed or a known image or restart failure is visible. -Timestamps are RFC 3339 values. `ssh_host`, `ssh_command`, `pod_name`, and `message` can be null while resources converge. `gpu` is null for CPU-only boxes and remains stable across stop and start. +Timestamps are RFC 3339 values. `ssh_host`, `ssh_command`, `pod_name`, and `message` can be null while resources converge. `gpu` is null for CPU-only boxes and `image` is null when no custom profile was selected. Both resolved allocations remain stable across stop and start. ## Lifecycle semantics @@ -215,7 +248,7 @@ Validation failures contain a list of structured errors. Common status codes are | 403 | Missing or invalid browser CSRF token | | 404 | Devbox does not exist | | 409 | Devbox name already exists | -| 422 | Invalid path, request field, repository, preset, TTL, disabled GPU feature, or unknown GPU profile | +| 422 | Invalid path, request field, repository, preset, TTL, disabled feature, or unknown GPU or image profile | | 503 | Controller cannot reach the Kubernetes API through `/ready` | Clients should preserve the status code, treat error payload text as diagnostic rather than stable machine data, and retry only transient transport or readiness failures. diff --git a/docs/architecture.md b/docs/architecture.md index cc4267a..6b68bfd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -29,7 +29,7 @@ The controller translates lifecycle operations into native Kubernetes resources - One `PersistentVolumeClaim` per devbox, mounted at `/home/dev`. - When Insights is enabled, one scoped ingest `Secret` per devbox and one central Insights `PersistentVolumeClaim` for the controller. -Resource names are deterministic (`devbox-NAME`), and labels plus annotations carry controller ownership, creation time, expiry, preset, repository, retained storage size, and any resolved GPU allocation. +Resource names are deterministic (`devbox-NAME`), and labels plus annotations carry controller ownership, creation time, expiry, preset, repository, retained storage size, and any resolved GPU or custom image allocation. ## GPU resolution path @@ -56,6 +56,32 @@ The create API accepts only an optional profile name. The controller resolves th The resolved profile is stored as a bounded Deployment annotation. Existing boxes therefore retain their allocation across stop, start, TTL expiry, and template reconciliation even if the Helm catalog later changes. Capability discovery publishes only profile names, labels, descriptions, resources, and counts. Scheduling details and images remain operator policy. +## Custom image resolution path + +Custom images use the same operator-owned catalog pattern while preserving the Devboxes workspace contract: + +```text +Helm custom image profiles + | + v +validated controller settings + | + +---- authenticated capability catalog ----> CLI and dashboard + | +user selects profile name or exact approved reference + | + v +resolved pinned snapshot + | + +---- sidecar profile ----> credential-free custom-image container + | + +---- workspace profile --> verified Devboxes-compatible main container +``` + +The controller resolves a selector before it creates a PVC, Deployment, or SSH Service. A default `sidecar` profile runs a compatible non-root service image in the same pod network namespace as the prepared workspace. It has bounded configured resources and optional high pod-local ports, but no Devboxes Secret mount, home PVC mount, Kubernetes API token, public Service, extra capability, command override, or scheduling policy. `runAsNonRoot` is enforced, privilege escalation is disabled, and all capabilities are dropped. The SSH workspace remains the only interactive and credential-bearing container. + +A `workspace` profile deliberately replaces the main interactive image and is restricted to a compatible Devboxes-derived image. The controller prevents it from competing with a GPU profile that independently selects `workspaceImage`. The complete resolved profile is stored in a Deployment annotation, so stop, start, TTL expiry, and Insights template reconciliation retain the original image policy even if Helm values later change. Capability discovery exposes only profile labels, modes, descriptions, and declared ports. + ## Persistence model Disconnecting SSH leaves the pod and tmux session running. Stopping scales the Deployment to zero, which ends processes but leaves the PVC. Deleting removes the Deployment and Service while retaining the PVC by default. Purging explicitly deletes the PVC. @@ -79,6 +105,7 @@ The workspace entrypoint refuses to start without `SSH_AUTHORIZED_KEYS`. It prep - Controller RBAC is a Role scoped to the release namespace; it cannot manage cluster-wide resources. - Workspace service accounts have no RBAC binding and do not mount Kubernetes API tokens. - GPU clients can choose only a configured profile name. They cannot inject images, device resources, RuntimeClasses, supplemental groups, selectors, tolerations, privileged mode, host paths, or device paths. +- Custom-image clients can choose only a configured profile name or exact configured image reference. They cannot inject a registry, command, Service, volume, port mapping, resource request, capability, host path, ServiceAccount, or scheduling field. Sidecar containers drop all Linux capabilities and cannot mount Devboxes credentials or persistent storage. - Workspace Secrets are mounted read-only with mode `0440`, scoped to the workspace group, and are not embedded in either image. - Each Insights ingest credential is HMAC-signed, write-only, scoped to one box and UUID instance, and stored in a dedicated namespaced Secret. It is never a controller, browser, or CLI credential. - The controller runs as a non-root user with a read-only root filesystem and all Linux capabilities dropped. diff --git a/docs/cli.md b/docs/cli.md index f136e02..da45f30 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -105,6 +105,7 @@ devbox create atlas --preset medium --ttl 24 --repo owner/project --ssh | `--repo OWNER/REPOSITORY` | Clone a GitHub repository on first boot | | `--gpu` | Request the operator's default GPU profile | | `--gpu-profile PROFILE` | Request an exact operator-approved profile; implies `--gpu` | +| `--image PROFILE_OR_IMAGE` | Select an operator-approved custom image profile or its exact configured image reference | | `--no-wait` | Return after the API accepts the request | | `--ssh` | Wait for readiness, then connect | @@ -119,6 +120,17 @@ devbox create training --gpu-profile nvidia-l4 --preset large --ssh The controller rejects a GPU request when the feature is disabled or the profile is unknown. If Kubernetes cannot schedule the requested resource before the wait timeout, the error includes the latest scheduler reason. Use `--no-wait` for intentionally queued work and inspect it with `devbox status`. +Custom image selection is also opt-in. Discover profiles before creating a box, then use the stable profile name. The exact configured image reference is accepted for parity with existing container-oriented workflows, but arbitrary references are rejected before Kubernetes resources are created: + +```bash +devbox image profiles +devbox create docs-preview --image nginx --ssh +devbox create docs-preview --image docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine --ssh +devbox ssh docs-preview -- -L 8080:127.0.0.1:8080 +``` + +Service profiles run as isolated, non-root, pod-local sidecars on high ports. Workspace profiles are only for compatible Devboxes workspace derivatives. See [custom image profiles](images.md) for the full contract and security boundary. + ### `devbox gpu profiles` Discover the profiles available for new boxes: @@ -130,9 +142,20 @@ devbox gpu profiles --json `devbox gpu` is a shorthand for the same catalog. Human output shows the profile identifier, display name, resource count, Kubernetes resource name, description, and default marker. JSON returns `enabled`, `default_profile`, and `profiles`. When the operator disables GPU support, the command reports that state instead of presenting stale profiles. +### `devbox image profiles` + +Discover custom image profiles available for new boxes: + +```bash +devbox image profiles +devbox image profiles --json +``` + +Human output shows the profile identifier, display name, mode, declared pod-local ports, and description. It intentionally omits the underlying image reference, pull policy, resource envelope, and other operator policy. JSON returns `enabled` and `profiles`. When the operator disables custom images, the command reports that state rather than accepting stale profile names. + ### `devbox list` -List boxes sorted by creation time, newest first. Human output includes an `ACCELERATOR` column containing `cpu` or the resolved GPU profile. +List boxes sorted by creation time, newest first. Human output includes an `ACCELERATOR` column containing `cpu` or the resolved GPU profile and an `IMAGE` column containing the resolved custom profile when present. ```bash devbox list @@ -141,7 +164,7 @@ devbox list --json ### `devbox status NAME` -Show state, preset, storage, accelerator allocation, expiry, repository, SSH address, and any readiness or scheduling message. +Show state, preset, storage, accelerator and custom image allocations, expiry, repository, SSH address, and any readiness or scheduling message. ```bash devbox status atlas @@ -236,7 +259,7 @@ JSON preserves nullable measurements and response metadata. CSV prefixes spreads ## Output and scripting -Human-readable results go to stdout. Progress and connection-wait messages go to stderr. Failures return a nonzero exit status. `--json` emits formatted JSON for list, status, create, start, stop, GPU capability, and metrics workflows, which can be consumed with `jq`: +Human-readable results go to stdout. Progress and connection-wait messages go to stderr. Failures return a nonzero exit status. `--json` emits formatted JSON for list, status, create, start, stop, GPU and custom-image capability, and metrics workflows, which can be consumed with `jq`: ```bash devbox list --json | jq -r '.[] | select(.state == "ready") | .name' diff --git a/docs/configuration.md b/docs/configuration.md index 71354f6..ef7af67 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -56,6 +56,54 @@ master token in derived mode, or rotating the dedicated key, revokes all issued The namespace must permit the workspace pod's documented `sudo` capability set. Kubernetes Pod Security `baseline` is compatible; `restricted` is not. +## Custom image profiles + +Custom images are disabled by default. Enable a small operator-owned catalog when users need a prebuilt service or a tested Devboxes workspace derivative. The controller accepts only a configured profile name or its exact configured image reference, resolves it before creating Kubernetes resources, and pins the complete profile on the resulting Deployment. + +| Value | Default | Meaning | +| --- | --- | --- | +| `workspace.customImages.enabled` | `false` | Publish configured profiles and accept image requests | +| `workspace.customImages.profiles[].name` | none | Stable lowercase identifier exposed to clients | +| `workspace.customImages.profiles[].displayName` | none | Human-readable catalog label | +| `workspace.customImages.profiles[].description` | empty | Short service or workspace purpose | +| `workspace.customImages.profiles[].image` | none | Reviewed container reference, preferably digest-pinned | +| `workspace.customImages.profiles[].mode` | `sidecar` | `sidecar` for application images or `workspace` for a compatible Devboxes derivative | +| `workspace.customImages.profiles[].pullPolicy` | `IfNotPresent` | `Always`, `IfNotPresent`, or `Never` | +| `workspace.customImages.profiles[].resources.cpuRequest` | `25m` | Sidecar CPU request; rejected for a workspace profile | +| `workspace.customImages.profiles[].resources.memoryRequest` | `32Mi` | Sidecar memory request; rejected for a workspace profile | +| `workspace.customImages.profiles[].resources.cpuLimit` | `500m` | Sidecar CPU limit; rejected for a workspace profile | +| `workspace.customImages.profiles[].resources.memoryLimit` | `512Mi` | Sidecar memory limit; rejected for a workspace profile | +| `workspace.customImages.profiles[].ports` | `[]` | Optional named pod-local ports from 1024 through 65535 for discovery and SSH tunnels | + +Example service sidecar profile: + +```yaml +workspace: + customImages: + enabled: true + profiles: + - name: nginx + displayName: NGINX preview + description: Serve a local static-site preview + image: docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine + mode: sidecar + pullPolicy: IfNotPresent + resources: + cpuRequest: 25m + memoryRequest: 32Mi + cpuLimit: 500m + memoryLimit: 512Mi + ports: + - name: http + containerPort: 8080 +``` + +A sidecar receives no Devboxes Secret, persistent-volume mount, Kubernetes service-account token, extra capability, or public Kubernetes Service. It must run as a non-root user on an unprivileged port from 1024 through 65535. It shares only the pod network namespace, so a user can reach a declared port through `devbox ssh NAME -- -L LOCAL:127.0.0.1:PORT`. + +`mode: workspace` replaces the interactive container. Use it only for a tested derivative of the matching Devboxes workspace image that preserves the entrypoint, `dev` user, SSH service on port `2222`, persistent-home setup, mounted Secret handling, and readiness behavior. Workspace profiles use the selected Devboxes preset for compute and cannot declare the sidecar-only `resources` envelope. A workspace image cannot combine with a GPU profile that defines `workspaceImage`; a sidecar profile can combine with GPU because it does not alter the primary workspace container. + +The chart rejects unknown fields, duplicate profile names or image references, unsafe image strings, privileged ports, more than 32 profiles, and enabled configurations without profiles. The controller validates resource quantities and limits before serving requests. Existing devboxes retain their pinned profiles if the catalog later changes or is disabled. See [custom image profiles](images.md) for the full security and lifecycle model. + ## GPU acceleration GPU acceleration is disabled by default. Operators configure one or more named profiles and choose a default profile for `devbox create --gpu`. CPU-only creation remains available regardless of whether GPU support is enabled. diff --git a/docs/development.md b/docs/development.md index 59f9f52..3356be9 100644 --- a/docs/development.md +++ b/docs/development.md @@ -49,7 +49,7 @@ Set `DEVBOX_CONFIG` to a temporary path during development to avoid changing you The server-rendered UI lives under `controller/src/devboxes_controller/templates`, with plain CSS and JavaScript under `static`. Preserve WCAG 2.2 AA contrast, complete keyboard operation, visible focus, textual status, responsive layouts, and reduced-motion support. Keep inline scripts and handlers out of templates so the Content Security Policy remains strict. -The test fake can preview all lifecycle states and multiple GPU profiles without a Kubernetes cluster: +The test fake can preview all lifecycle states, multiple GPU profiles, and an approved non-root, high-port custom image profile without a Kubernetes cluster: ```bash cd controller @@ -91,6 +91,8 @@ The CI Kind job builds both images and the CLI, loads the images into a clean cl GPU coverage is intentionally layered because ordinary Kind workers do not expose production accelerator hardware. Controller tests prove disabled and named-profile API behavior, exact pod resources and scheduling fields, scheduler diagnostics, allocation reporting, and pinned profile reconciliation. CLI tests prove command parsing and request shape. `scripts/test-helm-gpu.sh` proves disabled, enabled, and invalid chart contracts. The clean-cluster test enables an intentionally unschedulable extended resource and proves capability discovery, CLI and API selection, the generated pod contract, Pending diagnostics, and complete cleanup. A real GPU cluster remains the acceptance environment for vendor driver, runtime, image, and workload compatibility. +Custom image coverage is layered too. Controller tests prove catalog parsing, disabled and unknown-selector rejection before Kubernetes writes, exact sidecar and workspace manifests, secret and volume isolation, response reporting, pinned-profile reconciliation, and GPU workspace-image conflict rejection. CLI tests prove profile discovery and `--image` request shape. `scripts/test-helm-images.sh` proves disabled, enabled, and invalid chart contracts. The clean-cluster test builds a small service fixture, loads it into Kind, verifies the sidecar has no workspace mounts or credentials, reaches it over pod loopback, exercises SSH tunneling, and verifies cleanup. A deployment with real private registries or workspace-mode derivatives also requires the operator's production image, registry, architecture, and SSH lifecycle acceptance checks. + Privacy tests use fixtures derived from exact Codex and Claude Code clients pinned in the workspace image. Fixtures must use synthetic values. Never commit a real prompt, response, command, path, repository name, email address, provider credential, account identifier, or session identifier. The agent and controller sanitizers are separate trust boundaries and both require regression coverage. ## Release contract diff --git a/docs/golden-path.md b/docs/golden-path.md index b962cf8..29dd139 100644 --- a/docs/golden-path.md +++ b/docs/golden-path.md @@ -62,6 +62,8 @@ Do not depend on a cache existing on only one node. Kubernetes may schedule the GPU profiles may select a larger derived workspace image. Pre-pull each configured GPU image only on nodes eligible for that profile, and validate host driver compatibility before making the profile the default. +When using custom image profiles, pre-pull each reviewed sidecar image on every node that can host a regular workspace, or use a nearby registry mirror. Keep profile resource envelopes conservative and verify that an application port is reachable over the pod loopback interface. For a workspace-mode profile, apply the same SSH, retained-home, and restart checks as the release workspace image before publishing it. + ## 4. Install and verify the CLI Use the checksummed release installer, authenticate over HTTPS, and verify the identity returned by the controller. @@ -102,6 +104,16 @@ devbox create inference --gpu --preset medium --ssh CPU and memory presets remain independent from accelerator selection. Start with the smallest preset that satisfies host-side preprocessing and compilation, then measure before increasing it. +When approved custom images are enabled, use a profile rather than a raw registry reference. A service profile is useful for a local application dependency or preview server while the Devboxes workspace remains your SSH entry point: + +```bash +devbox image profiles +devbox create docs-preview --preset small --image nginx --ssh +devbox ssh docs-preview -- -L 8080:127.0.0.1:8080 +``` + +Inspect the profile mode before creation. A sidecar profile can combine with a GPU profile; a workspace profile cannot be paired with a GPU profile that sets its own workspace image. The profile is pinned on the created box, so delete and recreate an intentional test box when you want to verify a catalog revision. + ## 6. Tune in the right order Measure before changing the cluster. Startup and interactive performance usually improve in this order: diff --git a/docs/images.md b/docs/images.md new file mode 100644 index 0000000..d525ebc --- /dev/null +++ b/docs/images.md @@ -0,0 +1,121 @@ +# Custom image profiles + +Devboxes can run an operator-approved container image with a new devbox. This makes it practical to test a prebuilt open-source service without Docker-in-Docker, while keeping the prepared SSH workspace, persistent home volume, and Kubernetes policy intact. + +The feature is disabled by default. An operator defines a small catalog through Helm, and users select a profile through the CLI or browser workbench. The controller never accepts an arbitrary image reference that is not already in that catalog. + +## Why profiles are the API + +An ordinary application image is not a complete Devboxes workspace. For example, an NGINX image does not contain the Devboxes entrypoint, the `dev` SSH user, persistent-home initialization, or the required secret bootstrap. Replacing the interactive container with it would make SSH readiness fail. + +Image profiles make this distinction explicit: + +- `sidecar`, the default, runs an application image that is compatible with the non-root sidecar contract beside the prepared Devboxes workspace. This is the right mode for NGINX, databases, emulators, documentation servers, and similar prebuilt services. +- `workspace` replaces the interactive Devboxes image. Use it only for a tested derivative of the matching Devboxes workspace image that preserves the complete SSH and lifecycle contract. + +The catalog is returned by the authenticated API, printed by `devbox image profiles`, and rendered in the dashboard. It publishes safe user-facing names, descriptions, modes, and pod-local ports. It does not publish the underlying image reference, pull policy, or resource limits. + +## Configure approved images + +Set `workspace.customImages.enabled=true` and define one or more profiles in the Helm values file: + +```yaml +workspace: + customImages: + enabled: true + profiles: + - name: nginx + displayName: NGINX preview + description: Serve a local static-site preview over the pod network + image: docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine + mode: sidecar + pullPolicy: IfNotPresent + resources: + cpuRequest: 25m + memoryRequest: 32Mi + cpuLimit: 500m + memoryLimit: 512Mi + ports: + - name: http + containerPort: 8080 + protocol: TCP +``` + +Apply the values with the normal Helm upgrade: + +```bash +helm upgrade --install devboxes oci://ghcr.io/vicotrbb/charts/devboxes \ + --version VERSION \ + --namespace devboxes \ + --values values.yaml +``` + +| Value | Meaning | +| --- | --- | +| `workspace.customImages.enabled` | Accept custom image requests and publish the catalog. Defaults to `false`. | +| `workspace.customImages.profiles[].name` | Stable lowercase profile identifier used by clients. | +| `workspace.customImages.profiles[].displayName` | Human-readable label in CLI and dashboard. | +| `workspace.customImages.profiles[].description` | Short task-oriented description. | +| `workspace.customImages.profiles[].image` | Approved container image reference. Pin a tag or digest in durable values. | +| `workspace.customImages.profiles[].mode` | `sidecar` or `workspace`. Omit for the safer `sidecar` default. | +| `workspace.customImages.profiles[].pullPolicy` | `Always`, `IfNotPresent`, or `Never`. Omit for `IfNotPresent`. | +| `workspace.customImages.profiles[].resources` | CPU and memory requests and limits for a sidecar profile. Workspace profiles reject this sidecar-only field. | +| `workspace.customImages.profiles[].ports` | Optional named, pod-local application ports from 1024 through 65535 for status and documentation. | + +The chart rejects unknown fields, duplicate profile names, duplicate image references, blank labels, URL-scheme image strings, privileged ports, and enabled configurations without profiles. The controller also validates resource quantities and limits when it starts. + +## Use a service image + +Discover the catalog first. The profile name is the recommended stable selector: + +```bash +devbox image profiles +devbox create docs-preview --image nginx --ssh +``` + +For direct parity with an existing container reference, the CLI also accepts an exact image reference that matches a configured profile: + +```bash +devbox create docs-preview --image docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine --ssh +``` + +The controller resolves either form to the same profile before it creates a PVC, Deployment, or SSH Service. An unapproved reference fails with `422 Unprocessable Content` and lists only the available profile names. + +The image runs as a `custom-image` sidecar in the same pod network namespace as the prepared workspace. It must declare a non-root image user and listen on a port from 1024 through 65535; Kubernetes sets `runAsNonRoot: true`, disables privilege escalation, and drops every Linux capability. It does not receive the Devboxes Secret, a persistent-volume mount, a Kubernetes API token, or a public Kubernetes Service. Reach a declared port from the workspace or tunnel it through SSH: + +```bash +devbox ssh docs-preview -- -L 8080:127.0.0.1:8080 +# Open http://127.0.0.1:8080 in the local browser. +``` + +The dashboard exposes the same approved profiles in the create form. Choosing a profile explains whether it is a service sidecar or replacement workspace, and shows any declared pod-local ports. Existing devbox rows and `devbox status` report the resolved profile and ports. + +## Use a complete workspace image + +Use `mode: workspace` only after proving that the image is a compatible derivative. It replaces the primary interactive container, so it must retain all of these properties: + +1. The Devboxes entrypoint and SSH daemon listening on port `2222`. +2. The `dev` user, persistent `/home/dev` layout, and tmux startup flow. +3. Runtime handling for the mounted workspace Secret and optional repository clone. +4. Compatibility with the selected Kubernetes security context and readiness probe. + +Start from the matching released Devboxes workspace image, add only the required tooling, then test SSH readiness, retained-home reuse, stop and start, and Insights reconciliation before publishing the profile. Workspace profiles keep the selected Devboxes preset for compute and must not set the sidecar-only `resources` field. Do not declare a generic application image such as NGINX as a `workspace` profile. + +A workspace profile can combine with a GPU profile only when that GPU profile does not already select its own `workspaceImage`. This prevents two independent policies from silently competing for the interactive container image. Sidecar profiles can combine with GPU profiles because they do not change the main workspace image or receive GPU resources. + +## Security and operations + +Custom image profiles are appropriate only for Devboxes' existing trusted single-operator model. The operator chooses and reviews every image, resource envelope, pull policy, and port declaration. Users can select a profile but cannot inject a registry, command, Service, volume, host path, capability, resource request, or Kubernetes scheduling field. + +Treat every catalog image as part of the trusted software supply chain: + +1. Prefer a digest or a pinned, regularly reviewed tag over `latest`. +2. Scan and provenance-verify the image according to the registry and organization policy. +3. Test image pulls from every eligible node and keep required pull credentials valid. +4. Use bounded CPU and memory values for sidecars, then inspect scheduler and runtime behavior under the intended preset. +5. Keep application ports pod-local unless a separately reviewed exposure path is required. +6. Choose images that already run unprivileged and use high ports. The tested NGINX example is `nginxinc/nginx-unprivileged`; the standard root-oriented NGINX image is intentionally incompatible with this policy. + +The fully resolved profile is stored as a Deployment annotation when the devbox is created. Stop, start, TTL expiry, and Insights template reconciliation retain that pinned contract even if Helm values later change. Disabling the feature rejects new requests and hides the catalog, but existing boxes keep their stored sidecar or workspace selection. Delete and recreate a devbox to choose a newer profile while retaining its home volume. + +If an approved image cannot pull or its sidecar crashes, Devboxes reports the Kubernetes image or restart failure through `devbox status` and the dashboard. Check the image reference, pull Secret, node network path, image architecture, and application logs before changing the catalog. diff --git a/docs/index.md b/docs/index.md index f494b6f..848fea4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -14,6 +14,7 @@ Devboxes turns Kubernetes capacity into persistent, SSH-accessible development e - [CLI reference](cli.md), commands, global flags, environment variables, output, and SSH forwarding. - [API reference](api.md), authentication, endpoints, request and response contracts, and errors. - [GPU acceleration](gpu.md), profile design, cluster prerequisites, images, scheduling, security, and diagnosis. +- [Custom image profiles](images.md), approved service sidecars, compatible workspace derivatives, security boundaries, and lifecycle behavior. - [Insights](insights.md), opt-in AI telemetry, aggregate Git activity, privacy boundaries, retention, backup, and purge. - The authenticated `/docs` page in a running controller, an operator-focused guide rendered with installation-specific values. @@ -33,4 +34,4 @@ Devboxes turns Kubernetes capacity into persistent, SSH-accessible development e ## Supported scope -Devboxes supports one trusted operator or trusted operator group per installation. The controller is namespaced, each workspace has persistent home storage, each SSH Service uses either `LoadBalancer` or `NodePort`, and GPU devices are optional operator-owned profiles. Multi-tenant authorization, untrusted workload isolation, browser terminals, cluster-wide GPU driver installation, and automatic PVC deletion are outside the current scope. +Devboxes supports one trusted operator or trusted operator group per installation. The controller is namespaced, each workspace has persistent home storage, each SSH Service uses either `LoadBalancer` or `NodePort`, and GPU devices and custom images are optional operator-owned profiles. Multi-tenant authorization, arbitrary untrusted container execution, browser terminals, cluster-wide GPU driver installation, and automatic PVC deletion are outside the current scope. diff --git a/docs/operations.md b/docs/operations.md index b653367..484578e 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -40,6 +40,8 @@ Use node selectors, affinity, tolerations, and an existing PriorityClass only wh For GPU capacity, count the extended-resource units requested by active profile allocations and interpret them according to the vendor plugin's dedicated, partitioned, or sharing mode. `count: 1` means one advertised unit, which is not always one physical board. Monitor allocatable resources and Pending events on every GPU pool. Keep CPU-only headroom because GPU workspaces still request the selected CPU and memory preset. +For custom image profiles, add each sidecar's configured CPU and memory request to the selected workspace preset when forecasting node pressure. A catalog profile is trusted supply-chain policy, not an ad hoc developer image setting. Pin or regularly review image references, validate every supported architecture, verify registry credentials from each eligible node, and record the intended pod-local port and resource envelope. + Profile selectors and tolerations are policy, not capacity detection. Before publishing a profile, prove that at least one node matches all selectors, tolerates the intended taints, advertises the exact resource, and can pull the profile image. Document whether each profile is dedicated or shared. ## Backups and restore @@ -88,6 +90,8 @@ Then run `scripts/verify-install.sh`, confirm `/ready`, list existing boxes, cre When GPU configuration or images change, render the profile JSON before applying, run `devbox gpu profiles` after rollout, and create a disposable box for every changed profile. Verify the vendor diagnostic inside the box, stop and start it, and confirm the same profile contract remains. Stopping releases live device capacity, so Kubernetes may select a different physical device on start. Existing GPU Deployments retain their resolved snapshots; a Helm profile edit affects only later creations. Plan migrations as explicit delete and recreate operations, with a separate PVC retention decision. +When custom image profiles change, render the catalog before applying, run `devbox image profiles`, and create a disposable test box for every changed profile. For a sidecar, verify a declared non-root image user, a port from 1024 through 65535, the container list, no unexpected Secret or PVC mount, application health over pod loopback, SSH tunneling, stop and start, and cleanup. For a workspace-mode profile, also verify SSH readiness, persistent-home reuse, repository bootstrap, and Insights behavior if enabled. Existing Deployments retain their creation-time resolved snapshot; changing or disabling a Helm profile rejects new requests but does not rewrite existing boxes. Retire a profile only after inventorying `devbox list --json` and making an explicit delete or migration plan. + Enabling Insights does not force-restart an active legacy workspace. `devbox metrics status` reports `restart_required` until the box goes through a normal stop and start. Stopped workspaces are updated without starting compute. Confirm the expected state before and after the rollout. Prefer a reviewed values file over `--reuse-values`. It makes removed defaults and configuration drift visible. @@ -105,6 +109,8 @@ A Helm rollback changes controller resources, not the contents of workspace PVCs Disabling GPU support or rolling back the catalog rejects new GPU requests but does not rewrite existing GPU Deployments. This is intentional. Inventory allocations with `devbox list --json` before retiring drivers, runtimes, images, or node pools that those boxes still require. +Disabling custom image support or removing a profile similarly rejects new requests but does not remove a running sidecar or change an existing workspace image. Inventory image allocations before retiring a registry image or pull credential, then delete or migrate each affected box deliberately. + ## Token rotation Replace the configured controller Secret value, then restart the controller: diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 24cefde..60cc18c 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -115,6 +115,31 @@ Replace `nvidia.com/gpu` with the profile's resource. `Insufficient RESOURCE` me Do not patch the generated Deployment or add privileged mode and host device mounts. The controller owns that resource, and manual changes make the box non-reproducible. Correct the driver, capacity, or Helm profile, then delete and recreate a disposable test box. Existing boxes intentionally retain their creation-time profile snapshot. +## A custom image profile fails + +First confirm the selected profile and the generated container contract: + +```bash +devbox status atlas +devbox image profiles +kubectl get deployment devbox-atlas -n devboxes \ + -o jsonpath='{.metadata.annotations.image\.devboxes\.bonalab\.org/profile}{"\n"}{.spec.template.spec.containers[*].name}{"\n"}' +kubectl describe pod -n devboxes \ + -l devboxes.bonalab.org/name=atlas +``` + +For a sidecar profile, inspect its own logs and verify the declared pod-local port from the workspace container: + +```bash +kubectl logs deployment/devbox-atlas -n devboxes -c custom-image --tail=100 +kubectl exec deployment/devbox-atlas -n devboxes -c devbox -- \ + curl --fail http://127.0.0.1:8080/ +``` + +`ErrImagePull` and `ImagePullBackOff` indicate a registry, architecture, reference, or pull-credential problem. Validate the approved image on an eligible node before changing the profile. A crash loop after a successful pull is application behavior; check the image's expected environment and command, remembering that Devboxes intentionally does not inject credentials, persistent storage, or a public Service into a sidecar. Sidecars must already run as a non-root user and use a port from 1024 through 65535. For NGINX, use the unprivileged image variant rather than weakening the generated security context. + +For a workspace-mode profile, test the complete Devboxes contract instead of treating it like a generic container. The primary container must initialize the persistent home and start SSH on port `2222`. Revert the profile to a known compatible Devboxes-derived image if SSH readiness fails. Do not work around this by patching a generated Deployment, adding privilege, or mounting host paths. + ## SSH address remains pending For `LoadBalancer`, inspect `.status.loadBalancer.ingress` and the load-balancer controller events: diff --git a/scripts/fixtures/custom-image/Dockerfile b/scripts/fixtures/custom-image/Dockerfile new file mode 100644 index 0000000..e6395d3 --- /dev/null +++ b/scripts/fixtures/custom-image/Dockerfile @@ -0,0 +1,5 @@ +FROM nginxinc/nginx-unprivileged:1.27.5-alpine + +USER root +RUN printf '%s\n' 'devboxes custom image e2e' > /usr/share/nginx/html/index.html +USER 101 diff --git a/scripts/kind-e2e.sh b/scripts/kind-e2e.sh index 574c4cd..cf1a410 100755 --- a/scripts/kind-e2e.sh +++ b/scripts/kind-e2e.sh @@ -5,6 +5,8 @@ cluster="${DEVBOXES_E2E_CLUSTER:-devboxes-e2e}" namespace="${DEVBOXES_NAMESPACE:-devboxes}" controller_port="${DEVBOXES_E2E_CONTROLLER_PORT:-18000}" ssh_port="${DEVBOXES_E2E_SSH_PORT:-12222}" +custom_image_ssh_port="${DEVBOXES_E2E_CUSTOM_IMAGE_SSH_PORT:-12223}" +custom_image_tunnel_port="${DEVBOXES_E2E_CUSTOM_IMAGE_TUNNEL_PORT:-18080}" node_image="${DEVBOXES_E2E_NODE_IMAGE:-kindest/node:v1.35.0@sha256:452d707d4862f52530247495d180205e029056831160e22870e37e3f6c1ac31f}" published_version="${DEVBOXES_E2E_PUBLISHED_VERSION:-}" released_cli="${DEVBOXES_E2E_CLI:-}" @@ -14,6 +16,8 @@ token="e2e-access-token-at-least-32-characters" temporary_directory="$(mktemp -d)" controller_port_forward="" ssh_port_forward="" +custom_image_ssh_port_forward="" +custom_image_tunnel="" previous_context="" for command in kind kubectl helm docker curl jq ssh ssh-keygen nc python3; do @@ -49,6 +53,8 @@ cleanup() { kubectl --context "kind-$cluster" logs -n "$namespace" deployment/devboxes --tail=200 >&2 || true kubectl --context "kind-$cluster" logs -n "$namespace" deployment/devbox-smoke \ -c insights-agent --tail=200 >&2 || true + kubectl --context "kind-$cluster" logs -n "$namespace" deployment/devbox-image-smoke \ + -c custom-image --tail=200 >&2 || true if [[ -f "$temporary_directory/controller-port-forward.log" ]]; then cat "$temporary_directory/controller-port-forward.log" >&2 fi @@ -68,6 +74,12 @@ cleanup() { if [[ -n "$ssh_port_forward" ]]; then kill "$ssh_port_forward" >/dev/null 2>&1 || true fi + if [[ -n "$custom_image_tunnel" ]]; then + kill "$custom_image_tunnel" >/dev/null 2>&1 || true + fi + if [[ -n "$custom_image_ssh_port_forward" ]]; then + kill "$custom_image_ssh_port_forward" >/dev/null 2>&1 || true + fi if [[ -n "$controller_port_forward" ]]; then kill "$controller_port_forward" >/dev/null 2>&1 || true fi @@ -314,7 +326,11 @@ if [[ -n "$published_version" ]]; then else docker build --tag devboxes-controller:e2e controller docker build --tag devboxes-workspace:e2e workspace - kind load docker-image --name "$cluster" devboxes-controller:e2e devboxes-workspace:e2e + docker build --tag devboxes-test-nginx:e2e scripts/fixtures/custom-image + kind load docker-image --name "$cluster" \ + devboxes-controller:e2e \ + devboxes-workspace:e2e \ + devboxes-test-nginx:e2e DEVBOXES_ACCESS_TOKEN="$token" \ DEVBOXES_SSH_PUBLIC_KEY="$temporary_directory/id_ed25519.pub" \ @@ -335,6 +351,19 @@ else --set-string 'gpu.profiles[0].resourceName=example.com/gpu' \ --set 'gpu.profiles[0].count=1' \ --set 'gpu.profiles[0].supplementalGroups[0]=44' \ + --set workspace.customImages.enabled=true \ + --set 'workspace.customImages.profiles[0].name=test-nginx' \ + --set-string 'workspace.customImages.profiles[0].displayName=Test NGINX' \ + --set-string 'workspace.customImages.profiles[0].description=Kind service sidecar fixture' \ + --set-string 'workspace.customImages.profiles[0].image=devboxes-test-nginx:e2e' \ + --set 'workspace.customImages.profiles[0].mode=sidecar' \ + --set 'workspace.customImages.profiles[0].pullPolicy=Never' \ + --set-string 'workspace.customImages.profiles[0].resources.cpuRequest=25m' \ + --set-string 'workspace.customImages.profiles[0].resources.memoryRequest=32Mi' \ + --set-string 'workspace.customImages.profiles[0].resources.cpuLimit=100m' \ + --set-string 'workspace.customImages.profiles[0].resources.memoryLimit=128Mi' \ + --set 'workspace.customImages.profiles[0].ports[0].name=http' \ + --set 'workspace.customImages.profiles[0].ports[0].containerPort=8080' \ --set workspace.sshService.type=NodePort \ --set workspace.sshService.host=dev-node.example.test fi @@ -412,6 +441,125 @@ if [[ -z "$published_version" ]]; then kubectl -n "$namespace" wait --for=delete deployment/devbox-gpu-smoke --timeout=2m kubectl -n "$namespace" wait --for=delete pvc/devbox-gpu-smoke-home --timeout=2m fi + +if [[ -z "$published_version" ]]; then + image_capabilities="$(api "http://127.0.0.1:$controller_port/api/v1/capabilities")" + jq -e ' + .images == { + "enabled": true, + "profiles": [{ + "name": "test-nginx", + "display_name": "Test NGINX", + "description": "Kind service sidecar fixture", + "mode": "sidecar", + "ports": [{"name": "http", "container_port": 8080, "protocol": "TCP"}] + }] + } + ' <<<"$image_capabilities" >/dev/null + if [[ -n "$released_cli" ]]; then + cli --json image profiles \ + | jq -e '.enabled == true and .profiles[0].name == "test-nginx"' >/dev/null + cli --json create image-smoke --image test-nginx --no-wait \ + | jq -e ' + .image.profile == "test-nginx" + and .image.mode == "sidecar" + and .image.ports == [{"name": "http", "container_port": 8080, "protocol": "TCP"}] + ' >/dev/null + else + api \ + -H 'Content-Type: application/json' \ + -d '{"name":"image-smoke","image":"test-nginx"}' \ + "http://127.0.0.1:$controller_port/api/v1/devboxes" \ + | jq -e ' + .image.profile == "test-nginx" + and .image.mode == "sidecar" + ' >/dev/null + fi + kubectl -n "$namespace" rollout status deployment/devbox-image-smoke --timeout="$workspace_timeout" + image_deployment="$(kubectl -n "$namespace" get deployment devbox-image-smoke -o json)" + jq -e ' + (.spec.template.spec.containers[] | select(.name == "custom-image")) as $sidecar + | .metadata.annotations["image.devboxes.bonalab.org/profile"] == "test-nginx" + and (.metadata.annotations["image.devboxes.bonalab.org/resolved-config"] | fromjson | .name == "test-nginx" and .mode == "sidecar") + and ([.spec.template.spec.containers[].name] == ["devbox", "custom-image", "insights-agent"]) + and .spec.template.spec.automountServiceAccountToken == false + and $sidecar.image == "devboxes-test-nginx:e2e" + and $sidecar.imagePullPolicy == "Never" + and $sidecar.env == null + and $sidecar.volumeMounts == null + and $sidecar.securityContext.runAsNonRoot == true + and $sidecar.securityContext.allowPrivilegeEscalation == false + and $sidecar.securityContext.capabilities.drop == ["ALL"] + and $sidecar.resources.requests == {"cpu": "25m", "memory": "32Mi"} + and $sidecar.resources.limits == {"cpu": "100m", "memory": "128Mi"} + and $sidecar.ports == [{"name": "http", "containerPort": 8080, "protocol": "TCP"}] + ' <<<"$image_deployment" >/dev/null + kubectl -n "$namespace" exec deployment/devbox-image-smoke -c devbox -- \ + curl -fsS http://127.0.0.1:8080/ \ + | grep -Fx 'devboxes custom image e2e' >/dev/null + + kubectl -n "$namespace" port-forward service/devbox-image-smoke-ssh \ + "$custom_image_ssh_port:22" >"$temporary_directory/custom-image-ssh-port-forward.log" 2>&1 & + custom_image_ssh_port_forward=$! + for _ in {1..30}; do + nc -z 127.0.0.1 "$custom_image_ssh_port" >/dev/null 2>&1 && break + sleep 1 + done + nc -z 127.0.0.1 "$custom_image_ssh_port" >/dev/null 2>&1 + ssh \ + -i "$temporary_directory/id_ed25519" \ + -p "$custom_image_ssh_port" \ + -N \ + -L "$custom_image_tunnel_port:127.0.0.1:8080" \ + -o ExitOnForwardFailure=yes \ + -o StrictHostKeyChecking=no \ + -o UserKnownHostsFile=/dev/null \ + dev@127.0.0.1 >"$temporary_directory/custom-image-tunnel.log" 2>&1 & + custom_image_tunnel=$! + for _ in {1..30}; do + if curl -fsS "http://127.0.0.1:$custom_image_tunnel_port/" \ + >"$temporary_directory/custom-image-body.txt"; then + break + fi + kill -0 "$custom_image_tunnel" >/dev/null 2>&1 || break + sleep 1 + done + grep -Fx 'devboxes custom image e2e' "$temporary_directory/custom-image-body.txt" >/dev/null + kill "$custom_image_tunnel" >/dev/null 2>&1 || true + wait "$custom_image_tunnel" 2>/dev/null || true + custom_image_tunnel="" + kill "$custom_image_ssh_port_forward" >/dev/null 2>&1 || true + wait "$custom_image_ssh_port_forward" 2>/dev/null || true + custom_image_ssh_port_forward="" + + if [[ -n "$released_cli" ]]; then + cli --json stop image-smoke >/dev/null + cli --json start image-smoke >/dev/null + else + api -X POST "http://127.0.0.1:$controller_port/api/v1/devboxes/image-smoke/stop" >/dev/null + api -X POST "http://127.0.0.1:$controller_port/api/v1/devboxes/image-smoke/start" >/dev/null + fi + kubectl -n "$namespace" rollout status deployment/devbox-image-smoke --timeout="$workspace_timeout" + image_after_restart="$(kubectl -n "$namespace" get deployment devbox-image-smoke -o json)" + jq -e ' + .metadata.annotations["image.devboxes.bonalab.org/profile"] == "test-nginx" + and ([.spec.template.spec.containers[].name] == ["devbox", "custom-image", "insights-agent"]) + ' <<<"$image_after_restart" >/dev/null + kubectl -n "$namespace" exec deployment/devbox-image-smoke -c devbox -- \ + curl -fsS http://127.0.0.1:8080/ \ + | grep -Fx 'devboxes custom image e2e' >/dev/null + if [[ -n "$released_cli" ]]; then + cli delete image-smoke --purge --yes >/dev/null + else + api -X DELETE \ + "http://127.0.0.1:$controller_port/api/v1/devboxes/image-smoke?purge=true" >/dev/null + fi + kubectl -n "$namespace" wait --for=delete deployment/devbox-image-smoke --timeout=2m + kubectl -n "$namespace" wait --for=delete service/devbox-image-smoke-ssh --timeout=2m + kubectl -n "$namespace" wait --for=delete pvc/devbox-image-smoke-home --timeout=2m + kubectl -n "$namespace" wait --for=delete secret/devbox-image-smoke-insights --timeout=2m +fi + if [[ -n "$released_cli" ]]; then cli --json list | jq -e 'type == "array"' >/dev/null diff --git a/scripts/test-helm-images.sh b/scripts/test-helm-images.sh new file mode 100755 index 0000000..bc31f5d --- /dev/null +++ b/scripts/test-helm-images.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +project_directory="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +chart="$project_directory/charts/devboxes" +temporary_directory="$(mktemp -d)" +trap 'rm -rf "$temporary_directory"' EXIT + +helm lint "$chart" --strict + +helm template devboxes "$chart" --namespace devboxes \ + >"$temporary_directory/disabled.yaml" +grep -Fq 'name: DEVBOXES_CUSTOM_IMAGES_ENABLED' "$temporary_directory/disabled.yaml" +grep -Fq 'name: DEVBOXES_CUSTOM_IMAGES' "$temporary_directory/disabled.yaml" +grep -Fq 'value: "false"' "$temporary_directory/disabled.yaml" + +image_profile=( + --set workspace.customImages.enabled=true + --set 'workspace.customImages.profiles[0].name=nginx' + --set-string 'workspace.customImages.profiles[0].displayName=NGINX preview' + --set-string 'workspace.customImages.profiles[0].description=Serve a local preview' + --set-string 'workspace.customImages.profiles[0].image=docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine' + --set 'workspace.customImages.profiles[0].mode=sidecar' + --set 'workspace.customImages.profiles[0].pullPolicy=Always' + --set-string 'workspace.customImages.profiles[0].resources.cpuRequest=25m' + --set-string 'workspace.customImages.profiles[0].resources.memoryRequest=32Mi' + --set-string 'workspace.customImages.profiles[0].resources.cpuLimit=500m' + --set-string 'workspace.customImages.profiles[0].resources.memoryLimit=512Mi' + --set 'workspace.customImages.profiles[0].ports[0].name=http' + --set 'workspace.customImages.profiles[0].ports[0].containerPort=8080' + --set 'workspace.customImages.profiles[0].ports[0].protocol=TCP' +) + +helm template devboxes "$chart" --namespace devboxes "${image_profile[@]}" \ + >"$temporary_directory/enabled.yaml" +grep -Fq 'name: DEVBOXES_CUSTOM_IMAGES_ENABLED' "$temporary_directory/enabled.yaml" +grep -Fq 'value: "true"' "$temporary_directory/enabled.yaml" +grep -Fq 'docker.io/nginxinc/nginx-unprivileged:1.27.5-alpine' "$temporary_directory/enabled.yaml" +grep -Fq 'containerPort' "$temporary_directory/enabled.yaml" + +if helm template devboxes "$chart" --namespace devboxes \ + --set workspace.customImages.enabled=true \ + >"$temporary_directory/missing.yaml" \ + 2>"$temporary_directory/missing.err"; then + printf 'error: custom image render accepted an enabled feature without profiles\n' >&2 + exit 1 +fi +grep -Fq 'profiles' "$temporary_directory/missing.err" + +if helm template devboxes "$chart" --namespace devboxes \ + "${image_profile[@]}" \ + --set 'workspace.customImages.profiles[1].name=nginx' \ + --set-string 'workspace.customImages.profiles[1].displayName=Duplicate NGINX' \ + --set-string 'workspace.customImages.profiles[1].image=registry.example/nginx:2' \ + >"$temporary_directory/duplicate.yaml" \ + 2>"$temporary_directory/duplicate.err"; then + printf 'error: custom image render accepted duplicate profile names\n' >&2 + exit 1 +fi +grep -Fq 'duplicate name' "$temporary_directory/duplicate.err" + +if helm template devboxes "$chart" --namespace devboxes \ + "${image_profile[@]}" \ + --set-string 'workspace.customImages.profiles[0].image=https://registry.example/nginx:1' \ + >"$temporary_directory/invalid-image.yaml" \ + 2>"$temporary_directory/invalid-image.err"; then + printf 'error: custom image render accepted a URL scheme\n' >&2 + exit 1 +fi +grep -Fq 'image must not contain a URL scheme' "$temporary_directory/invalid-image.err" + +if helm template devboxes "$chart" --namespace devboxes \ + "${image_profile[@]}" \ + --set 'workspace.customImages.profiles[0].ports[0].containerPort=80' \ + >"$temporary_directory/low-port.yaml" \ + 2>"$temporary_directory/low-port.err"; then + printf 'error: custom image render accepted a privileged sidecar port\n' >&2 + exit 1 +fi +grep -Fq 'minimum:' "$temporary_directory/low-port.err" + +if helm template devboxes "$chart" --namespace devboxes \ + "${image_profile[@]}" \ + --set 'workspace.customImages.profiles[0].mode=workspace' \ + >"$temporary_directory/workspace-resources.yaml" \ + 2>"$temporary_directory/workspace-resources.err"; then + printf 'error: custom image render accepted sidecar resources for a workspace profile\n' >&2 + exit 1 +fi + +printf 'Verified disabled, enabled, and invalid custom image Helm contracts.\n'