diff --git a/docs/labels-as-tags.md b/docs/labels-as-tags.md new file mode 100644 index 00000000..0fa204d4 --- /dev/null +++ b/docs/labels-as-tags.md @@ -0,0 +1,245 @@ +# Compose labels → cloud tags/labels + +This document describes how Docker Compose `labels` are propagated to the +cloud resources each service/network produces, and the design decisions and +gotchas behind the implementation. It is a contributor/design reference, not +user-facing provider documentation. + +## Goal + +A Compose project can attach `labels` to a service or a network: + +```yaml +services: + web: + image: nginx + labels: + com.acme.team: core + com.acme.env: prod +networks: + default: + labels: + com.acme.cost-center: "1234" +``` + +These labels must flow **transitively** onto every cloud resource that the +service (or the default network's shared infrastructure) creates: + +- **AWS** → resource `Tags` (a `map[string]string`) +- **Azure** → resource `Tags` +- **GCP** → resource `Labels` (with sanitization — see below) + +`ServiceConfig.Labels` and `NetworkConfig.Labels` are both +`compose.MapOrList[string]` (compose-go normalizes the list form +`["k=v"]` to a map). See `provider/compose/types.go`. + +## Core mechanism: resource transformations, not providers + +We use a Pulumi `ResourceTransformation` threaded through resource `opts`, **not** +a per-service provider with `defaultTags`. + +- **Why not per-service providers.** A resource's provider is part of its + identity. Giving each service its own provider (to leverage the AWS + provider's `defaultTags`) risks **replacing every existing resource** on the + next deploy of every existing stack. Transformations only modify *inputs* at + registration time and never force replacement. +- **Transformations propagate to children.** The codebase threads `opts` into + every resource-creation call (`pulumi.Parent(...)`, `childOpts...`, + `pulumi.Composite`). Attaching the transformation once on the per-service or + per-network `opts` reaches all downstream resources — *including* resources + created under sub-providers (e.g. a cross-account Route 53 provider for DNS), + which a provider-based approach would miss. This is decisive. + +This mirrors the pattern already used by the Azure provider's +`DefaultTagsTransformation` (`provider/defangazure/azure/azure.go`), which +cascades the project-wide `defang-*` tags to every azure-native resource. + +The TypeScript source of truth (`defang-mvp/pulumi/shared/aws/tags.ts`) +implements the same idea for AWS via `labelTagsTransformation`. + +## Taggability: the sharp edge + +You cannot blanket-add tags/labels to *every* resource — sending a `tags` +input to a type that has no such input is a **hard deploy failure**. The +transformation must therefore only touch resources that accept the field. + +### Go advantage: reflection over an allowlist + +The TS implementation maintains a hand-curated **allowlist** of taggable type +tokens, because in JS the resource `props` is an untyped object and there is no +way to ask "does this type accept tags?". + +In Go we have typed `Args` structs, so the transformation inspects the props by +reflection and **only sets the field when it actually exists** with the right +type: + +```go +f := v.FieldByName("Tags") // or "Labels" for GCP +if !f.IsValid() || !f.CanSet() { return nil } +if !f.Type().Implements(stringMapInputType) && f.Type() != stringMapInputType { + return nil +} +``` + +This is strictly more robust than an allowlist: + +- An unrecognized / non-taggable type is silently skipped (graceful + degradation), exactly like an allowlist — but with **no list to maintain**. +- It correctly excludes `aws:autoscaling/group:Group`, whose `Tags` field is a + **list** of `{key,value,propagateAtLaunch}`, not a `StringMap` — the + `StringMapInput` type check rejects it automatically. (The TS allowlist had + to special-case this by hand.) +- Resources whose `Args` struct simply has no `Tags`/`Labels` field + (`certificateValidation`, `rolePolicyAttachment`, `route53/record`, + `lb/targetGroupAttachment`, `s3` sub-resources, etc.) are skipped because + `FieldByName` returns an invalid value. + +We additionally gate on the **type-token prefix** (`aws:`, `gcp:`, +`azure-native:`) so a provider's transformation never touches another +provider's resources. + +### Robust tag/label merge + +`props.Tags` may be a plain `pulumi.StringMap` or a `pulumi.StringMapOutput`. +Do not range over it directly. Convert both the incoming labels and the +existing value to `StringMapOutput` and merge inside `ApplyT`: + +```go +merged := pulumi.All(labelsOut, existingOut).ApplyT(func(parts []any) map[string]string { + out := map[string]string{} + for k, v := range parts[0].(map[string]string) { out[k] = v } // labels first + for k, v := range parts[1].(map[string]string) { out[k] = v } // existing wins + return out +}).(pulumi.StringMapOutput) +``` + +### Precedence + +**Functional/explicit tags win over user labels.** Spread labels *first*, then +the existing tags on top. This ensures `defang:service`, `defang:scope`, and +the project-wide `defang-*` tags can never be clobbered by a user label, which +would break tag-based selectors. Provider/project default tags merge in for +keys the user didn't set. + +## No silent failures (AWS) vs. sanitization (GCP) + +- **AWS / Azure** tags accept a wide character set (dots, mixed case, etc.). + Labels are passed through **verbatim**. If a user supplies a reserved key + (e.g. the `aws:` prefix), let the deploy **fail loudly** rather than silently + dropping it. The label→tags normalizer only collapses empty → `nil`. + +- **GCP** labels are validated strictly by the API: keys and values must match + `[a-z]([a-z0-9_-]){0,62}` (lowercase, start with a letter, ≤63 chars), and + dots are not allowed. Compose labels routinely use reverse-DNS keys with dots + and mixed-case values (`com.acme.team=Core`). Passing those through verbatim + would fail every GCP deploy that uses conventional labels. We therefore + **sanitize** GCP label keys/values: lowercase, replace invalid characters + (including `.`) with `_`, ensure a leading lowercase letter, and truncate to + 63 chars. (This is a deliberate divergence from the AWS verbatim rule, + forced by GCP's validation.) + +## Wiring per provider + +The transformation is built from the relevant labels and merged into `opts` +just before the component is registered, using +`pulumi.Composite(opts, pulumi.Transformations([]pulumi.ResourceTransformation{t}))`. +A `nil` transformation (no labels) means we leave `opts` untouched. + +In this provider both transformations are attached at the **component** level: +the per-service transform on each service component (`newService` / +`buildService` / `createServiceResources`), and the per-network (default-only) +transform on the **Project component** in `Construct`. Attaching the network +transform at the Project component is the simplest mechanism that reaches the +whole shared-infra subtree — including a multi-language component like the awsx +VPC, whose `Tags` it sets and which awsx then propagates to the subnets/NAT it +creates internally (an in-process transform can't reach those directly). The +trade-off: it also cascades onto service resources. That is acceptable — the +default network spans every service — and per-service labels and functional +tags win on key collision, so nothing is clobbered. + +### AWS (`provider/defangaws`, prefix `"aws"` — matches `aws:` and `awsx:`) + +- **Per-service.** From `svc.Labels` on the service component in `newService` + (covers all branches: ECS service, RDS Postgres, ElastiCache Redis). +- **Per-network (default).** From `inputs.Networks[compose.DefaultNetwork].Labels` + on the Project component in `Construct`. +- No existing transformations in the AWS provider — this is the first. + +### GCP (`provider/defanggcp`, SDK: `pulumi-gcp`, prefix `"gcp:"`, sanitized) + +- **Per-service.** Sanitized from `svc.Labels` on the service component in + `buildService`. Resources that carry GCP `Labels`: + `gcp:cloudrunv2/service:Service` and + `gcp:compute/instanceTemplate:InstanceTemplate`. The instance group manager + (`regionInstanceGroupManager`) has **no** `Labels` field — reflection skips it. +- **Per-network (default).** Sanitized, on the Project component in `Construct`. + GCP `Network`/`Subnetwork`/`Firewall` have **no `Labels` field**, so the core + network resources are skipped; the transform still lands on shared resources + that do have `Labels` (e.g. the reserved public IP, DNS zone, Artifact + Registry). + +### Azure (`provider/defangazure`, SDK: `azure-native`, prefix `"azure-native:"`) + +- A project-wide `DefaultTagsTransformation(BaseTags)` already cascades + `defang-org/project/stack/etag` to every azure-native resource. The + per-network transform is composed alongside it on the Project component. +- **Per-service.** From `svc.Labels` on the service component in + `createServiceResources`. Existing `ServiceTags(serviceName)` and base tags + win on collision. +- **Per-network (default).** On the Project component in `Construct`, composed + with the base-tags transform. Lands on VNet, subnets, DNS zones, managed + environment. + +### Standalone Service path (all providers) + +Per-service labels are wired only on the **project** dispatch path, where +`compose.ServiceConfig.Labels` is populated from the parsed compose file. The +standalone `ServiceInputs` structs do not carry a `Labels` field, so a directly +instantiated `Service` is not label-tagged. Adding standalone support would mean +adding `Labels` to each provider's `ServiceInputs` (a schema change) — left as a +follow-up. + +## Scope boundaries + +- **"Network label" = the default network's labels, applied project-wide.** + Because the transform is attached at the Project component, default-network + labels reach all shared infrastructure (VPC/subnets/NAT, ALB, listeners, ECS + cluster, log groups, security groups, DNS zones, etc.) **and** service + resources. Per-service labels and functional/`defang:*` tags win on collision. +- Resources created inside a multi-language component (e.g. awsx VPC subnets) + are tagged via that component's own tag propagation, not the in-process + transform, since transforms don't cross the MLC boundary. + +## Known pre-existing gap (do not rely on it) + +`defang:service` is **not** applied uniformly to every service-owned AWS +resource today (missing on the app-image CodeBuild project, ACME cert IAM +policies, ACME/app listener rules, and ACME target groups). So `defang:service` +is **not** a reliable "which service owns this" oracle for verification. The +label transformation does **not** depend on it — it works purely via `opts` +propagation. (Opportunity: the same per-service transformation could inject +`defang:service` uniformly to close the gap, at the cost of golden-file churn.) + +## Verification + +- Assert the invariant **per resource, scoped to the provider's resource tokens + only**: a resource carries a label tag/label **iff** its owning + service/network is labeled in the compose file. +- When matching ownership by resource name, account for **name sanitization** + (e.g. `example.com` → `examplecom`, dots stripped) or you'll get + false-positive "leaks." +- Unit-test the transformation directly (verbatim AWS/Azure merge, GCP + sanitization, existing-tag precedence, empty → no-op) and end-to-end by + registering a component with the transformation in `opts` and inspecting the + resulting resource inputs via the mock Pulumi server in `tests/testutil`. + +## Open TODOs + +- `DeployConfig.Labels` and `BuildConfig.Labels` are intentionally left out for + now (commented in `types.go`). If added, decide how they merge with the + service-level labels (build labels → build resources only; deploy labels → + runtime resources). +- GCP per-network labels remain unsupported by the SDK; revisit if pulumi-gcp + adds `Labels` to network types. + + diff --git a/provider/cmd/pulumi-resource-defang-aws/schema.json b/provider/cmd/pulumi-resource-defang-aws/schema.json index f1fbffda..5b59dd45 100644 --- a/provider/cmd/pulumi-resource-defang-aws/schema.json +++ b/provider/cmd/pulumi-resource-defang-aws/schema.json @@ -128,6 +128,12 @@ "properties": { "internal": { "type": "boolean" + }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } } }, "type": "object" @@ -214,6 +220,12 @@ "image": { "type": "string" }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "llm": { "$ref": "#/types/defang-aws:compose:LlmConfig" }, diff --git a/provider/cmd/pulumi-resource-defang-azure/schema.json b/provider/cmd/pulumi-resource-defang-azure/schema.json index 61bd87e1..073c166a 100644 --- a/provider/cmd/pulumi-resource-defang-azure/schema.json +++ b/provider/cmd/pulumi-resource-defang-azure/schema.json @@ -98,6 +98,12 @@ "properties": { "internal": { "type": "boolean" + }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } } }, "type": "object" @@ -184,6 +190,12 @@ "image": { "type": "string" }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "llm": { "$ref": "#/types/defang-azure:compose:LlmConfig" }, diff --git a/provider/cmd/pulumi-resource-defang-gcp/schema.json b/provider/cmd/pulumi-resource-defang-gcp/schema.json index 3a6d4162..b9051d1b 100644 --- a/provider/cmd/pulumi-resource-defang-gcp/schema.json +++ b/provider/cmd/pulumi-resource-defang-gcp/schema.json @@ -98,6 +98,12 @@ "properties": { "internal": { "type": "boolean" + }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } } }, "type": "object" @@ -184,6 +190,12 @@ "image": { "type": "string" }, + "labels": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "llm": { "$ref": "#/types/defang-gcp:compose:LlmConfig" }, diff --git a/provider/compose/labels.go b/provider/compose/labels.go new file mode 100644 index 00000000..ac1a5dda --- /dev/null +++ b/provider/compose/labels.go @@ -0,0 +1,109 @@ +package compose + +import ( + "reflect" + "strings" + + "github.com/pulumi/pulumi/sdk/v3/go/pulumi" +) + +// stringMapInputType is the reflect.Type of pulumi.StringMapInput, used to +// detect taggable Args structs by reflection. +var stringMapInputType = reflect.TypeOf((*pulumi.StringMapInput)(nil)).Elem() + +// labelsToStringMap converts Compose labels to a pulumi.StringMap, applying the +// optional normalize func to each key/value. Returns nil when there are no +// labels so callers can skip wiring up a transformation entirely. +func labelsToStringMap(labels MapOrList[string], normalize func(k, v string) (string, string)) pulumi.StringMap { + if len(labels) == 0 { + return nil + } + out := make(pulumi.StringMap, len(labels)) + for k, v := range labels { + if normalize != nil { + k, v = normalize(k, v) + } + out[k] = pulumi.String(v) + } + return out +} + +// LabelTagsTransformation returns a Pulumi resource transformation that merges +// the given Compose labels into the named tag/label field of every resource +// whose type token starts with typePrefix and whose Args struct exposes that +// field as a pulumi.StringMapInput. fieldName is "Tags" for AWS/Azure and +// "Labels" for GCP. +// +// Unlike the TS implementation (which gates on a hand-maintained allowlist of +// taggable type tokens), this relies on Go's typed Args structs: it only sets +// the field when reflection finds it with the right type, so non-taggable +// resources — and AWS's autoscaling Group, whose Tags field is a list rather +// than a StringMap — are skipped automatically. +// +// Existing (functional/explicit) tag values win over labels on key collision, +// so defang-managed tags such as "defang:service" can never be clobbered by a +// user label. normalize optionally rewrites keys/values (e.g. GCP label +// sanitization); pass nil to apply labels verbatim. Returns nil when there are +// no labels, so callers can skip attaching the transformation. +// +// Attach the result via pulumi.Transformations(...) on a component's opts; +// Pulumi cascades component-level transformations to all children (including +// resources created under sub-providers), which is why this reaches every +// downstream resource without threading a provider through each call. +func LabelTagsTransformation( + labels MapOrList[string], + typePrefix string, + fieldName string, + normalize func(k, v string) (string, string), +) pulumi.ResourceTransformation { + sm := labelsToStringMap(labels, normalize) + if len(sm) == 0 { + return nil + } + labelsOut := sm.ToStringMapOutput() + return func(args *pulumi.ResourceTransformationArgs) *pulumi.ResourceTransformationResult { + if !strings.HasPrefix(args.Type, typePrefix) { + return nil + } + v := reflect.ValueOf(args.Props) + if v.Kind() != reflect.Ptr || v.IsNil() { + return nil + } + v = v.Elem() + if v.Kind() != reflect.Struct { + return nil + } + f := v.FieldByName(fieldName) + if !f.IsValid() || !f.CanSet() { + return nil + } + if !f.Type().Implements(stringMapInputType) && f.Type() != stringMapInputType { + return nil + } + + // props.Tags may be a plain StringMap or a StringMapOutput; normalize + // both to a StringMapOutput before merging. + existingOut := pulumi.StringMap{}.ToStringMapOutput() + if existing, ok := f.Interface().(pulumi.StringMapInput); ok && existing != nil { + existingOut = existing.ToStringMapOutput() + } + + merged := pulumi.All(labelsOut, existingOut).ApplyT(func(parts []interface{}) map[string]string { + out := map[string]string{} + if lbls, ok := parts[0].(map[string]string); ok { + for k, v := range lbls { + out[k] = v // labels first + } + } + if existing, ok := parts[1].(map[string]string); ok { + for k, v := range existing { + out[k] = v // existing/functional tags win on collision + } + } + return out + }).(pulumi.StringMapOutput) + + f.Set(reflect.ValueOf(merged)) + return &pulumi.ResourceTransformationResult{Props: args.Props, Opts: args.Opts} + } +} diff --git a/provider/compose/labels_test.go b/provider/compose/labels_test.go new file mode 100644 index 00000000..e8df173c --- /dev/null +++ b/provider/compose/labels_test.go @@ -0,0 +1,109 @@ +package compose + +import ( + "reflect" + "strings" + "testing" + + "github.com/pulumi/pulumi/sdk/v3/go/pulumi" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// taggableArgs mimics an AWS/Azure resource Args struct exposing Tags. +// ElementType makes it satisfy pulumi.Input, the type of ResourceTransformationArgs.Props. +type taggableArgs struct { + Tags pulumi.StringMapInput +} + +func (taggableArgs) ElementType() reflect.Type { return reflect.TypeOf(taggableArgs{}) } + +// labelledArgs mimics a GCP resource Args struct exposing Labels. +type labelledArgs struct { + Labels pulumi.StringMapInput +} + +func (labelledArgs) ElementType() reflect.Type { return reflect.TypeOf(labelledArgs{}) } + +// untaggableArgs has no Tags/Labels field (e.g. an attachment resource). +type untaggableArgs struct { + Name pulumi.StringInput +} + +func (untaggableArgs) ElementType() reflect.Type { return reflect.TypeOf(untaggableArgs{}) } + +func TestLabelTagsTransformation_NilWhenNoLabels(t *testing.T) { + assert.Nil(t, LabelTagsTransformation(nil, "aws", "Tags", nil)) + assert.Nil(t, LabelTagsTransformation(MapOrList[string]{}, "aws", "Tags", nil)) +} + +func TestLabelTagsTransformation_MergesExistingWins(t *testing.T) { + err := pulumi.RunErr(func(ctx *pulumi.Context) error { + tr := LabelTagsTransformation( + MapOrList[string]{"com.acme.team": "core", "keep": "label"}, + "aws", "Tags", nil, + ) + require.NotNil(t, tr) + + props := &taggableArgs{Tags: pulumi.StringMap{ + "defang:service": pulumi.String("web"), + "keep": pulumi.String("existing"), + }} + res := tr(&pulumi.ResourceTransformationArgs{ + Type: "aws:ecs/service:Service", Name: "web", Props: props, + }) + require.NotNil(t, res) + + props.Tags.ToStringMapOutput().ApplyT(func(m map[string]string) string { + assert.Equal(t, "core", m["com.acme.team"], "label applied verbatim (dots kept)") + assert.Equal(t, "web", m["defang:service"], "functional tag preserved") + assert.Equal(t, "existing", m["keep"], "existing tag wins over label on collision") + return "" + }) + return nil + }, pulumi.WithMocks("proj", "stack", testMocks{})) + require.NoError(t, err) +} + +func TestLabelTagsTransformation_SkipsWrongPrefix(t *testing.T) { + tr := LabelTagsTransformation(MapOrList[string]{"a": "b"}, "aws", "Tags", nil) + require.NotNil(t, tr) + props := &taggableArgs{Tags: pulumi.StringMap{"x": pulumi.String("y")}} + res := tr(&pulumi.ResourceTransformationArgs{ + Type: "gcp:cloudrunv2/service:Service", Name: "web", Props: props, + }) + assert.Nil(t, res, "type token not matching prefix is left untouched") +} + +func TestLabelTagsTransformation_SkipsUntaggable(t *testing.T) { + tr := LabelTagsTransformation(MapOrList[string]{"a": "b"}, "aws", "Tags", nil) + require.NotNil(t, tr) + props := &untaggableArgs{Name: pulumi.String("x")} + res := tr(&pulumi.ResourceTransformationArgs{ + Type: "aws:lb/targetGroupAttachment:TargetGroupAttachment", Name: "x", Props: props, + }) + assert.Nil(t, res, "struct without a Tags field is skipped (no hard failure)") +} + +func TestLabelTagsTransformation_Normalize(t *testing.T) { + // Mimics GCP sanitization: dots → underscores, lowercased values. + normalize := func(k, v string) (string, string) { + return strings.ReplaceAll(k, ".", "_"), strings.ToLower(v) + } + err := pulumi.RunErr(func(ctx *pulumi.Context) error { + tr := LabelTagsTransformation(MapOrList[string]{"com.acme.team": "Core"}, "gcp:", "Labels", normalize) + require.NotNil(t, tr) + props := &labelledArgs{} + res := tr(&pulumi.ResourceTransformationArgs{ + Type: "gcp:cloudrunv2/service:Service", Name: "web", Props: props, + }) + require.NotNil(t, res) + require.NotNil(t, props.Labels) + props.Labels.ToStringMapOutput().ApplyT(func(m map[string]string) string { + assert.Equal(t, "core", m["com_acme_team"], "key sanitized, value lowercased") + return "" + }) + return nil + }, pulumi.WithMocks("proj", "stack", testMocks{})) + require.NoError(t, err) +} diff --git a/provider/compose/types.go b/provider/compose/types.go index 38f7a974..1c751eed 100644 --- a/provider/compose/types.go +++ b/provider/compose/types.go @@ -19,7 +19,9 @@ type NetworkID string type Services = map[string]ServiceConfig -type Networks map[NetworkID]NetworkConfig +type Networks = map[NetworkID]NetworkConfig + +type MapOrList[T any] map[string]T // TODO: these can be string[] but normalized to map by compose-go const DefaultNetwork NetworkID = "default" @@ -55,7 +57,7 @@ type ServiceConfig struct { Deploy *DeployConfig `pulumi:"deploy,optional" yaml:"deploy,omitempty"` // Environment variables - Environment map[string]*string `pulumi:"environment,optional" yaml:"environment,omitempty"` + Environment MapOrList[*string] `pulumi:"environment,optional" yaml:"environment,omitempty"` // Command to run Command []string `pulumi:"command,optional" yaml:"command,omitempty"` @@ -84,6 +86,8 @@ type ServiceConfig struct { LLM *LlmConfig `pulumi:"llm,optional" yaml:"x-defang-llm,omitempty"` + Labels MapOrList[string] `pulumi:"labels,optional" yaml:"labels,omitempty"` + // Models map[string]*ServiceModelConfig `pulumi:"models,optional" yaml:"models,omitempty"` } @@ -100,11 +104,12 @@ func (c *LlmConfig) UnmarshalYAML(value *yaml.Node) error { // type ServiceModelConfig struct {} -type DependsOnConfig map[string]ServiceDependency +type DependsOnConfig MapOrList[ServiceDependency] type NetworkConfig struct { Internal bool `pulumi:"internal,optional" yaml:"internal,omitempty"` - // IPAM *IPAMConfigInput `pulumi:"ipam,optional" yaml:"ipam,omitempty"` + // IPAM *IPAMConfigInput `pulumi:"ipam,optional" yaml:"ipam,omitempty"` + Labels MapOrList[string] `pulumi:"labels,optional" yaml:"labels,omitempty"` } type ServiceDependency struct { @@ -165,6 +170,8 @@ type DeployConfig struct { // Resource reservations and limits Resources *Resources `pulumi:"resources,optional" yaml:"resources,omitempty"` + + // Labels MapOrList[string] `pulumi:"labels,optional" yaml:"labels,omitempty"` } // Resources defines resource reservations and limits. @@ -197,7 +204,9 @@ type BuildConfig struct { Dockerfile *string `pulumi:"dockerfile,optional" yaml:"dockerfile,omitempty"` // Build arguments - Args map[string]string `pulumi:"args,optional" yaml:"args,omitempty"` + Args MapOrList[string] `pulumi:"args,optional" yaml:"args,omitempty"` + + // Labels MapOrList[string] `pulumi:"labels,optional" yaml:"labels,omitempty"` // Shared memory size (used for build task memory sizing) ShmSize *string `pulumi:"shmSize,optional" yaml:"shm_size,omitempty"` @@ -213,9 +222,10 @@ func (b *BuildConfig) UnmarshalYAML(value *yaml.Node) error { var raw struct { Context string `yaml:"context"` Dockerfile *string `yaml:"dockerfile,omitempty"` - Args map[string]string `yaml:"args,omitempty"` + Args MapOrList[string] `yaml:"args,omitempty"` ShmSize *string `yaml:"shm_size,omitempty"` Target *string `yaml:"target,omitempty"` + // Labels MapOrList[string] `yaml:"labels,omitempty"` } if err := value.Decode(&raw); err != nil { return err @@ -223,6 +233,7 @@ func (b *BuildConfig) UnmarshalYAML(value *yaml.Node) error { b.Context = pulumi.String(raw.Context) b.Dockerfile = raw.Dockerfile b.Args = raw.Args + // b.Labels = raw.Labels b.ShmSize = raw.ShmSize b.Target = raw.Target return nil diff --git a/provider/compose/yaml_test.go b/provider/compose/yaml_test.go index efb9271f..c1954518 100644 --- a/provider/compose/yaml_test.go +++ b/provider/compose/yaml_test.go @@ -24,7 +24,7 @@ target: builder // Context should be a pulumi.String, not a raw Go string assert.Implements(t, (*pulumi.StringInput)(nil), bc.Context) assert.Equal(t, "Dockerfile.prod", *bc.Dockerfile) - assert.Equal(t, map[string]string{"GO_VERSION": "1.22"}, bc.Args) + assert.Equal(t, MapOrList[string]{"GO_VERSION": "1.22"}, bc.Args) assert.Equal(t, "256m", *bc.ShmSize) assert.Equal(t, "builder", *bc.Target) } @@ -87,7 +87,7 @@ networks: assert.Implements(t, (*pulumi.StringInput)(nil), web.Build.Context) assert.Equal(t, "Dockerfile", *web.Build.Dockerfile) port := "8080" - assert.Equal(t, map[string]*string{"PORT": &port, "CONFIG": nil}, web.Environment) + assert.Equal(t, MapOrList[*string]{"PORT": &port, "CONFIG": nil}, web.Environment) require.NotNil(t, web.Deploy) assert.Equal(t, int32(2), *web.Deploy.Replicas) require.NotNil(t, web.Deploy.Resources) diff --git a/provider/defangaws/project.go b/provider/defangaws/project.go index 28117b3d..b90f3129 100644 --- a/provider/defangaws/project.go +++ b/provider/defangaws/project.go @@ -47,6 +47,19 @@ type ProjectOutputs struct { func (*Project) Construct( ctx *pulumi.Context, name, typ string, inputs ProjectInputs, opts pulumi.ResourceOption, ) (*ProjectOutputs, error) { + // Merge the default network's Compose labels into the Tags of all shared + // project infrastructure. Attaching the transformation to the Project + // component cascades it to every child (see provider/compose/labels.go), + // including the awsx VPC component — awsx propagates its Tags to the subnets, + // NAT gateways and route tables it creates internally, which an in-process + // transformation cannot reach across the multi-language-component boundary. + // The default network spans all services, so these labels also reach service + // resources; per-service labels (in newService) win on key collision. + netLabels := inputs.Networks[compose.DefaultNetwork].Labels + if t := compose.LabelTagsTransformation(netLabels, "aws", "Tags", nil); t != nil { + opts = pulumi.Composite(opts, pulumi.Transformations([]pulumi.ResourceTransformation{t})) + } + comp := &ProjectOutputs{} if err := ctx.RegisterComponentResource(typ, name, comp, opts); err != nil { return nil, err @@ -160,11 +173,22 @@ func newService( var endpoint pulumi.StringOutput var dependency pulumi.Resource var err error + + // Merge this service's Compose labels into the Tags of every AWS resource it + // creates. Attaching the transformation to the service component cascades it + // to all children (see provider/compose/labels.go). AWS tags accept the full + // Compose label character set, so labels are applied verbatim (nil normalize). + // Prefix "aws" matches both aws: and awsx: type tokens. + svcOpts := []pulumi.ResourceOption{parentOpt} + if t := compose.LabelTagsTransformation(svc.Labels, "aws", "Tags", nil); t != nil { + svcOpts = append(svcOpts, pulumi.Transformations([]pulumi.ResourceTransformation{t})) + } + switch { case svc.Postgres != nil: // Managed Postgres → RDS pgComp := &PostgresOutputs{} - if regErr := ctx.RegisterComponentResource(PostgresComponentType, svcName, pgComp, parentOpt); regErr != nil { + if regErr := ctx.RegisterComponentResource(PostgresComponentType, svcName, pgComp, svcOpts...); regErr != nil { return pulumi.StringOutput{}, nil, fmt.Errorf("registering postgres component %s: %w", svcName, regErr) } if err = createPostgres(ctx, pgComp, configProvider, svcName, svc, infra, deps); err == nil { @@ -174,7 +198,7 @@ func newService( case svc.Redis != nil: // Managed Redis → ElastiCache redisComp := &RedisOutputs{} - if regErr := ctx.RegisterComponentResource(RedisComponentType, svcName, redisComp, parentOpt); regErr != nil { + if regErr := ctx.RegisterComponentResource(RedisComponentType, svcName, redisComp, svcOpts...); regErr != nil { return pulumi.StringOutput{}, nil, fmt.Errorf("registering redis component %s: %w", svcName, regErr) } if err = createRedis(ctx, redisComp, svcName, svc, infra, deps); err == nil { @@ -188,7 +212,7 @@ func newService( // Container service → ECS svcComp := &ServiceOutputs{} - if regErr := ctx.RegisterComponentResource(ServiceComponentType, svcName, svcComp, parentOpt); regErr != nil { + if regErr := ctx.RegisterComponentResource(ServiceComponentType, svcName, svcComp, svcOpts...); regErr != nil { return pulumi.StringOutput{}, nil, fmt.Errorf("registering service component %s: %w", svcName, regErr) } imageURI, imgErr := provideraws.GetServiceImage(ctx, svcName, svc, infra.BuildInfra, pulumi.Parent(svcComp)) diff --git a/provider/defangazure/project.go b/provider/defangazure/project.go index 5d4d0a30..d6155605 100644 --- a/provider/defangazure/project.go +++ b/provider/defangazure/project.go @@ -195,6 +195,16 @@ func createServiceResources( comp := &serviceComponent{} var endpoint pulumi.StringOutput + // Merge this service's Compose labels into the Tags of every Azure resource it + // creates, on top of the project-wide BaseTags transformation. Attaching to the + // service component cascades to all children. Azure tags accept the full Compose + // label set, so labels are applied verbatim. A fresh slice avoids mutating the + // caller's childOpts backing array. + if t := compose.LabelTagsTransformation(svc.Labels, "azure-native:", "Tags", nil); t != nil { + childOpts = append(append([]pulumi.ResourceOption{}, childOpts...), + pulumi.Transformations([]pulumi.ResourceTransformation{t})) + } + switch { case svc.Postgres != nil: var err error @@ -454,9 +464,23 @@ func (*Project) Construct( // resource's Tags. azure-native has no DefaultTags, and pulumi-go-provider's // Construct ctx lacks a stack so RegisterStackTransformation panics — the // resource-level Transformations option is the supported cascade. - tagOpts := opts + transforms := []pulumi.ResourceTransformation{} if t := providerazure.DefaultTagsTransformation(baseTags); t != nil { - tagOpts = pulumi.Composite(opts, pulumi.Transformations([]pulumi.ResourceTransformation{t})) + transforms = append(transforms, t) + } + // Merge the default network's Compose labels into the Tags of shared project + // infrastructure (VNet, subnets, DNS zones, managed environment). The default + // network spans all services, so these also reach service resources; + // per-service labels (in createServiceResources) and base tags win on + // collision. Azure tags accept the full Compose label set (verbatim). + netLabels := inputs.Networks[compose.DefaultNetwork].Labels + if t := compose.LabelTagsTransformation(netLabels, "azure-native:", "Tags", nil); t != nil { + transforms = append(transforms, t) + } + + tagOpts := opts + if len(transforms) > 0 { + tagOpts = pulumi.Composite(opts, pulumi.Transformations(transforms)) } comp := &ProjectOutputs{} diff --git a/provider/defanggcp/gcp/labels.go b/provider/defanggcp/gcp/labels.go new file mode 100644 index 00000000..43efe004 --- /dev/null +++ b/provider/defanggcp/gcp/labels.go @@ -0,0 +1,54 @@ +package gcp + +import "strings" + +// SanitizeLabel rewrites a Compose label key/value into a form GCP accepts. +// +// GCP label keys and values must contain only lowercase letters, digits, +// underscores and hyphens (≤63 chars); keys must additionally start with a +// lowercase letter and be non-empty. Compose labels routinely use reverse-DNS +// keys with dots and mixed-case values (e.g. "com.acme.team"="Core"), which GCP +// rejects. We sanitize rather than pass through verbatim (unlike AWS) because an +// invalid label would otherwise fail the entire deploy. +// +// Sanitization is lossy and can collide (e.g. "a.b" and "a-b" both → "a_b"); the +// last value wins on collision. This is documented in docs/labels-as-tags.md. +func SanitizeLabel(k, v string) (string, string) { + return sanitizeLabelKey(k), sanitizeLabelChars(v) +} + +// sanitizeLabelChars lowercases s and replaces every character outside +// [a-z0-9_-] with an underscore, truncating to GCP's 63-char limit. +func sanitizeLabelChars(s string) string { + var b strings.Builder + for _, r := range strings.ToLower(s) { + switch { + case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-', r == '_': + b.WriteRune(r) + default: + b.WriteByte('_') + } + } + return truncate63(b.String()) +} + +// sanitizeLabelKey is sanitizeLabelChars plus GCP's key rules: a key must be +// non-empty and start with a lowercase letter (digits/_/- are not allowed as +// the first character). +func sanitizeLabelKey(s string) string { + out := sanitizeLabelChars(s) + if out == "" { + return "label" + } + if c := out[0]; c < 'a' || c > 'z' { + out = truncate63("k_" + out) + } + return out +} + +func truncate63(s string) string { + if len(s) > 63 { + return s[:63] + } + return s +} diff --git a/provider/defanggcp/gcp/labels_test.go b/provider/defanggcp/gcp/labels_test.go new file mode 100644 index 00000000..e5c08851 --- /dev/null +++ b/provider/defanggcp/gcp/labels_test.go @@ -0,0 +1,43 @@ +package gcp + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestSanitizeLabel(t *testing.T) { + tests := []struct { + name string + inKey, inVal string + wantKey, wantVal string + }{ + {"already valid", "team", "core", "team", "core"}, + {"dotted key, mixed-case value", "com.acme.team", "Core", "com_acme_team", "core"}, + {"uppercase and spaces", "My Label", "Hello World", "my_label", "hello_world"}, + {"value keeps dashes and underscores", "k", "a-b_c", "k", "a-b_c"}, + {"digit-leading key gets letter prefix", "1st", "v", "k_1st", "v"}, + {"empty key falls back", "...", "v", "k____", "v"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + k, v := SanitizeLabel(tt.inKey, tt.inVal) + assert.Equal(t, tt.wantKey, k) + assert.Equal(t, tt.wantVal, v) + }) + } +} + +func TestSanitizeLabel_Truncates(t *testing.T) { + long := strings.Repeat("a", 100) + k, v := SanitizeLabel(long, long) + assert.Len(t, k, 63) + assert.Len(t, v, 63) +} + +func TestSanitizeLabel_KeyStartsWithLetter(t *testing.T) { + // A key that sanitizes to all-invalid leading chars must still start with a letter. + k, _ := SanitizeLabel("9-9", "v") + assert.Regexp(t, `^[a-z]`, k) +} diff --git a/provider/defanggcp/project.go b/provider/defanggcp/project.go index 5a3336f3..8db97662 100644 --- a/provider/defanggcp/project.go +++ b/provider/defanggcp/project.go @@ -43,6 +43,17 @@ type ProjectOutputs struct { func (*Project) Construct( ctx *pulumi.Context, name, typ string, inputs ProjectInputs, opts pulumi.ResourceOption, ) (*ProjectOutputs, error) { + // Merge the default network's Compose labels into the Labels of shared project + // infrastructure that supports them (e.g. the reserved public IP, DNS zone, + // Artifact Registry). GCP networks/subnets/firewalls have no Labels field, so + // those are skipped by the reflection-based transformation. Sanitized like + // per-service labels. The default network spans all services, so these also + // reach service resources; per-service labels win on key collision. + netLabels := inputs.Networks[compose.DefaultNetwork].Labels + if t := compose.LabelTagsTransformation(netLabels, "gcp:", "Labels", providergcp.SanitizeLabel); t != nil { + opts = pulumi.Composite(opts, pulumi.Transformations([]pulumi.ResourceTransformation{t})) + } + comp := &ProjectOutputs{} if err := ctx.RegisterComponentResource(typ, name, comp, opts); err != nil { return nil, err @@ -170,10 +181,17 @@ func buildService( var lbEntry *providergcp.LBServiceEntry var svcComp pulumi.Resource - svcChildOpts := childOpts + // Start from a fresh slice so appends never mutate the shared childOpts backing array. + svcChildOpts := append([]pulumi.ResourceOption{}, childOpts...) if len(deps) > 0 { svcChildOpts = append(svcChildOpts, pulumi.DependsOn(deps)) } + // Merge this service's Compose labels into the Labels of every GCP resource it + // creates (Cloud Run service, Compute Engine instance template). GCP labels are + // validated strictly, so keys/values are sanitized (see gcp.SanitizeLabel). + if t := compose.LabelTagsTransformation(svc.Labels, "gcp:", "Labels", providergcp.SanitizeLabel); t != nil { + svcChildOpts = append(svcChildOpts, pulumi.Transformations([]pulumi.ResourceTransformation{t})) + } switch { case svc.Postgres != nil: diff --git a/sdk/v2/go/defang-aws/compose/pulumiTypes.go b/sdk/v2/go/defang-aws/compose/pulumiTypes.go index 0f6aa852..1ea5d4e3 100644 --- a/sdk/v2/go/defang-aws/compose/pulumiTypes.go +++ b/sdk/v2/go/defang-aws/compose/pulumiTypes.go @@ -666,7 +666,8 @@ func (o LlmConfigPtrOutput) Elem() LlmConfigOutput { } type NetworkConfig struct { - Internal *bool `pulumi:"internal"` + Internal *bool `pulumi:"internal"` + Labels map[string]string `pulumi:"labels"` } // NetworkConfigInput is an input type that accepts NetworkConfigArgs and NetworkConfigOutput values. @@ -681,7 +682,8 @@ type NetworkConfigInput interface { } type NetworkConfigArgs struct { - Internal pulumi.BoolPtrInput `pulumi:"internal"` + Internal pulumi.BoolPtrInput `pulumi:"internal"` + Labels pulumi.StringMapInput `pulumi:"labels"` } func (NetworkConfigArgs) ElementType() reflect.Type { @@ -739,6 +741,10 @@ func (o NetworkConfigOutput) Internal() pulumi.BoolPtrOutput { return o.ApplyT(func(v NetworkConfig) *bool { return v.Internal }).(pulumi.BoolPtrOutput) } +func (o NetworkConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v NetworkConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + type NetworkConfigMapOutput struct{ *pulumi.OutputState } func (NetworkConfigMapOutput) ElementType() reflect.Type { @@ -1346,6 +1352,7 @@ type ServiceConfig struct { Environment map[string]string `pulumi:"environment"` HealthCheck *HealthCheckConfig `pulumi:"healthCheck"` Image *string `pulumi:"image"` + Labels map[string]string `pulumi:"labels"` Llm *LlmConfig `pulumi:"llm"` Networks map[string]ServiceNetworkConfig `pulumi:"networks"` Platform *string `pulumi:"platform"` @@ -1376,6 +1383,7 @@ type ServiceConfigArgs struct { Environment pulumi.StringMapInput `pulumi:"environment"` HealthCheck HealthCheckConfigPtrInput `pulumi:"healthCheck"` Image pulumi.StringPtrInput `pulumi:"image"` + Labels pulumi.StringMapInput `pulumi:"labels"` Llm LlmConfigPtrInput `pulumi:"llm"` Networks ServiceNetworkConfigMapInput `pulumi:"networks"` Platform pulumi.StringPtrInput `pulumi:"platform"` @@ -1472,6 +1480,10 @@ func (o ServiceConfigOutput) Image() pulumi.StringPtrOutput { return o.ApplyT(func(v ServiceConfig) *string { return v.Image }).(pulumi.StringPtrOutput) } +func (o ServiceConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v ServiceConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + func (o ServiceConfigOutput) Llm() LlmConfigPtrOutput { return o.ApplyT(func(v ServiceConfig) *LlmConfig { return v.Llm }).(LlmConfigPtrOutput) } diff --git a/sdk/v2/go/defang-azure/compose/pulumiTypes.go b/sdk/v2/go/defang-azure/compose/pulumiTypes.go index 2d3aa3b1..aee5df38 100644 --- a/sdk/v2/go/defang-azure/compose/pulumiTypes.go +++ b/sdk/v2/go/defang-azure/compose/pulumiTypes.go @@ -666,7 +666,8 @@ func (o LlmConfigPtrOutput) Elem() LlmConfigOutput { } type NetworkConfig struct { - Internal *bool `pulumi:"internal"` + Internal *bool `pulumi:"internal"` + Labels map[string]string `pulumi:"labels"` } // NetworkConfigInput is an input type that accepts NetworkConfigArgs and NetworkConfigOutput values. @@ -681,7 +682,8 @@ type NetworkConfigInput interface { } type NetworkConfigArgs struct { - Internal pulumi.BoolPtrInput `pulumi:"internal"` + Internal pulumi.BoolPtrInput `pulumi:"internal"` + Labels pulumi.StringMapInput `pulumi:"labels"` } func (NetworkConfigArgs) ElementType() reflect.Type { @@ -739,6 +741,10 @@ func (o NetworkConfigOutput) Internal() pulumi.BoolPtrOutput { return o.ApplyT(func(v NetworkConfig) *bool { return v.Internal }).(pulumi.BoolPtrOutput) } +func (o NetworkConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v NetworkConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + type NetworkConfigMapOutput struct{ *pulumi.OutputState } func (NetworkConfigMapOutput) ElementType() reflect.Type { @@ -1346,6 +1352,7 @@ type ServiceConfig struct { Environment map[string]string `pulumi:"environment"` HealthCheck *HealthCheckConfig `pulumi:"healthCheck"` Image *string `pulumi:"image"` + Labels map[string]string `pulumi:"labels"` Llm *LlmConfig `pulumi:"llm"` Networks map[string]ServiceNetworkConfig `pulumi:"networks"` Platform *string `pulumi:"platform"` @@ -1376,6 +1383,7 @@ type ServiceConfigArgs struct { Environment pulumi.StringMapInput `pulumi:"environment"` HealthCheck HealthCheckConfigPtrInput `pulumi:"healthCheck"` Image pulumi.StringPtrInput `pulumi:"image"` + Labels pulumi.StringMapInput `pulumi:"labels"` Llm LlmConfigPtrInput `pulumi:"llm"` Networks ServiceNetworkConfigMapInput `pulumi:"networks"` Platform pulumi.StringPtrInput `pulumi:"platform"` @@ -1472,6 +1480,10 @@ func (o ServiceConfigOutput) Image() pulumi.StringPtrOutput { return o.ApplyT(func(v ServiceConfig) *string { return v.Image }).(pulumi.StringPtrOutput) } +func (o ServiceConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v ServiceConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + func (o ServiceConfigOutput) Llm() LlmConfigPtrOutput { return o.ApplyT(func(v ServiceConfig) *LlmConfig { return v.Llm }).(LlmConfigPtrOutput) } diff --git a/sdk/v2/go/defang-gcp/compose/pulumiTypes.go b/sdk/v2/go/defang-gcp/compose/pulumiTypes.go index 2796f956..88708cf0 100644 --- a/sdk/v2/go/defang-gcp/compose/pulumiTypes.go +++ b/sdk/v2/go/defang-gcp/compose/pulumiTypes.go @@ -666,7 +666,8 @@ func (o LlmConfigPtrOutput) Elem() LlmConfigOutput { } type NetworkConfig struct { - Internal *bool `pulumi:"internal"` + Internal *bool `pulumi:"internal"` + Labels map[string]string `pulumi:"labels"` } // NetworkConfigInput is an input type that accepts NetworkConfigArgs and NetworkConfigOutput values. @@ -681,7 +682,8 @@ type NetworkConfigInput interface { } type NetworkConfigArgs struct { - Internal pulumi.BoolPtrInput `pulumi:"internal"` + Internal pulumi.BoolPtrInput `pulumi:"internal"` + Labels pulumi.StringMapInput `pulumi:"labels"` } func (NetworkConfigArgs) ElementType() reflect.Type { @@ -739,6 +741,10 @@ func (o NetworkConfigOutput) Internal() pulumi.BoolPtrOutput { return o.ApplyT(func(v NetworkConfig) *bool { return v.Internal }).(pulumi.BoolPtrOutput) } +func (o NetworkConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v NetworkConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + type NetworkConfigMapOutput struct{ *pulumi.OutputState } func (NetworkConfigMapOutput) ElementType() reflect.Type { @@ -1346,6 +1352,7 @@ type ServiceConfig struct { Environment map[string]string `pulumi:"environment"` HealthCheck *HealthCheckConfig `pulumi:"healthCheck"` Image *string `pulumi:"image"` + Labels map[string]string `pulumi:"labels"` Llm *LlmConfig `pulumi:"llm"` Networks map[string]ServiceNetworkConfig `pulumi:"networks"` Platform *string `pulumi:"platform"` @@ -1376,6 +1383,7 @@ type ServiceConfigArgs struct { Environment pulumi.StringMapInput `pulumi:"environment"` HealthCheck HealthCheckConfigPtrInput `pulumi:"healthCheck"` Image pulumi.StringPtrInput `pulumi:"image"` + Labels pulumi.StringMapInput `pulumi:"labels"` Llm LlmConfigPtrInput `pulumi:"llm"` Networks ServiceNetworkConfigMapInput `pulumi:"networks"` Platform pulumi.StringPtrInput `pulumi:"platform"` @@ -1472,6 +1480,10 @@ func (o ServiceConfigOutput) Image() pulumi.StringPtrOutput { return o.ApplyT(func(v ServiceConfig) *string { return v.Image }).(pulumi.StringPtrOutput) } +func (o ServiceConfigOutput) Labels() pulumi.StringMapOutput { + return o.ApplyT(func(v ServiceConfig) map[string]string { return v.Labels }).(pulumi.StringMapOutput) +} + func (o ServiceConfigOutput) Llm() LlmConfigPtrOutput { return o.ApplyT(func(v ServiceConfig) *LlmConfig { return v.Llm }).(LlmConfigPtrOutput) }