feat(scaleway): add Scaleway provider with containers, postgres, and redis

[simple-agent-manager]

Summary

Adds the Scaleway Pulumi provider used by the Defang Scaleway BYOC CLI path. It maps Defang Compose projects to Scaleway Serverless Containers, Managed PostgreSQL, Managed Redis, private networking, Kaniko build jobs, and CD state artifacts.

Related CLI PR: DefangLabs/defang#2105.

Implemented

• Serverless Containers for Compose services, including ingress ports, resource mapping, health checks, scaling, env vars, and service metadata.
• Managed PostgreSQL and Managed Redis support with environment URL/hostname rewriting for dependent services.
• Shared private network for managed service connectivity.
• Kaniko builds through Scaleway Serverless Jobs v1alpha2.
• Secret Manager-backed build job secrets for S3 and Docker auth material.
• Cleanup of temporary build secrets after build jobs complete.
• Worker health shim for portless/background services and min_scale=1 for those workers.
• Scaleway CD image packaging for the local prerelease defang-scaleway provider plugin.
• CD project.pb upload to Scaleway Object Storage after successful deployment so CLI compose ps/services readback works, matching AWS/GCP/Azure behavior.

Scaleway platform quirks

• Serverless Containers require HTTP ingress and do not support host-mode ports.
• Private networks are egress-only for Serverless Containers. Containers can reach managed databases/Redis privately, but container-to-container traffic uses public HTTPS endpoints.
• Portless workers need a health shim because Scaleway requires a listening HTTP port. This currently assumes /bin/sh and one of node, python3, python, or nc; a TODO in code documents better future runtime paths.
• Portless workers are forced to min_scale=1 because scaled-to-zero workers cannot wake without inbound HTTP traffic.
• Redis is pinned to 8.4.0 because that is the available Scaleway version in current testing.
• Scaleway LLMs are handled in the CLI, not this Pulumi provider. The provider does not create LLM resources.

CD image notes

Pulumi's plugin install -f rejects prerelease provider versions such as 2.0.0-beta.5, while the CD program needs that exact version. The Scaleway CD image therefore installs the local provider binary into Pulumi's plugin cache layout manually and the CLI sets PULUMI_HOME=/root/.pulumi for the CD job.

Validated test image: rg.fr-par.scw.cloud/defang-cd/cd:sam-20260511d, digest sha256:f515f0387b57b001b5b91e08d4fbf854fac7847327efa6d9f08913c35b5f14aa.

Live validation

Validated on 2026-05-11 with the paired CLI draft branch:

Small app:

• Built and pushed the Scaleway CD image before testing.
• compose up completed for a Python ingress service with a Compose models: entry.
• The deployed native Scaleway endpoint returned HTTP 200.
• The app's /llm route successfully called Scaleway Generative API using CLI-created LLM auth config.
• project.pb upload restored compose ps; CLI reported DEPLOYMENT_COMPLETED and healthy.
• compose down removed the live Serverless Container and namespace.

Mastra Extended:

• Deployed the unmodified projects/samples/samples/mastra-extended sample on Scaleway stack mastraextended.
• Supplied required Defang config values only; no sample files were edited.
• Playwright clicked Generate sample items, verified 10 tasks, 10 events, and 20 classified items, waited 15 seconds, then asked the chat UI what to look at first.
• Playwright result: 1 passed.
• compose down removed the sample's Serverless Containers, namespace, managed Postgres, and managed Redis; native API checks confirmed cleanup.

Test plan

• go1.25.9 test ./provider/defangscaleway/...
• go1.25.9 test ./... from cd/
• Build and push Scaleway CD image
• Live Scaleway deploy, status, runtime endpoint, LLM route, logs query, and teardown validation
• Playwright validation of deployed mastra-extended UI
• Full CI pipeline

Known limitations

• Defang delegated DNS could not be validated with the available Scaleway credential because domain zone creation returned HTTP 403; native Scaleway domains worked.
• compose logs no longer fails on Cockpit endpoint/auth, but product log coverage/labels still need broader workload validation.
• One phantom Scaleway IPAM attachment from prior experiments still blocks deleting an old private network; all visible Serverless Containers/namespaces are gone.
• mastra-extended exposed two config polish issues: defang config set --random can generate a value that fails Scaleway Postgres password policy, and Scaleway Redis currently requires REDIS_PASSWORD even when the sample README does not mention it.

[lionello]

Looks good so far, although I would have done the resources in this order:

1. Service - test with known image
2. Build - test with sources
3. Redis - standalone test
4. Postgres - standalone test
5. Project - Compose integration

[lionello]

Perhaps we only need to watch the yaml examples, since most others are generated from those.

[simple-agent-manager]

Scaleway live-validation update pushed in 0f8707c.

What changed in the PR:

• Added examples/scaleway-e2e-demo/ with two small Go services:
  • api: connects to managed Postgres, creates demo_hits, inserts one row per request, and returns the count.
  • web: calls api and renders the API response from a public HTTP endpoint.
• Updated examples/scaleway-yaml/Pulumi.yaml into a reusable phased E2E demo template.
• Added examples/scaleway-e2e-demo/README.md with the exact flow used for validation: publish prebuilt images with ko, deploy DB, set DATABASE_URL, deploy API, set API_URL, deploy web, then destroy.
• Updated docs/scaleway-iteration-notes.md with the live findings and teardown status.
• Added comments explaining two important Scaleway-specific behaviors:
  • Serverless Container Private Networks are egress-only, so service-to-service traffic uses public HTTPS endpoints.
  • Scaleway reserves the default Postgres user/database name postgres, so the provider defaults to defang for managed Postgres unless explicitly configured.
• Adjusted dry-run config placeholders to be password-shaped (DryRun1!-...) so preview can pass provider-specific password validation.

Live validation performed:

• Destroyed the prior inspection deployment.
• Published temporary API/web images with ko because the workspace did not have Docker.
• Deployed a real chain: web -> api -> managed Postgres.
• Verified the public web endpoint returned API status: 200 and an incrementing database hit count from Postgres.
• Destroyed all 11 demo resources and removed local stack metadata.

Verification after docs/demo update:

• go test ./... in examples/scaleway-e2e-demo
• go test -v -timeout 5m ./provider/defangscaleway/... ./tests/scaleway
• make test_unit
• pulumi preview --non-interactive --diff for the updated YAML demo with temporary config, then removed the preview stack metadata

[simple-agent-manager]

Additional networking clarification pushed in f9eeb64.

Important distinction from the live E2E demo:

• We did confirm Scaleway Serverless Containers have no container-to-container internal networking path. Private Networks are egress-only for Serverless Containers, so web -> api must use the API service's public HTTPS endpoint. This is now called out more explicitly in the iteration notes and demo README.
• We did confirm the demo API could write to and read from managed Postgres.
• We did not confirm Docker Compose hostname parity for the database, i.e. using a cloud connection string like postgres://...@db:5432/... from the API container.
• The E2E demo used the provider-generated managed DB endpoint from db.connectionUrl, not hostname db.
• The Project component path does attach managed Postgres to shared private networking and postgresHostAndPort prefers Scaleway's private DB hostname/IP when available, but using a Compose-style db hostname in Scaleway remains unvalidated follow-up work.

I updated the docs so reviewers don't read the demo as proof that db hostname parity works on Scaleway yet.

[simple-agent-manager]

Corrected hostname validation summary (the prior comment was mangled by shell backtick substitution).

Validated the Scaleway DB hostname behavior live and pushed the docs update in f001940.

Results:

• Bare Compose-style DB hostname does not work from Serverless Containers: postgres://...@db:5432/... returned HTTP 500 with lookup db on 169.254.169.254:53: no such host.
• Scaleway internal DNS does work over the shared Private Network: postgres://...@db.demo.internal:5432/... returned HTTP 200 and the API inserted/read a row from managed Postgres.
• Test stack was destroyed and local Pulumi stack metadata was removed.

Docs updated to make the implication explicit: Scaleway supports private DB access, but not literal Docker Compose service-name DNS for managed DBs. The provider should preserve Defang's local/cloud connection-string experience by injecting or rewriting the cloud connection string to the provider-generated private DB endpoint (<resource>.<private-network>.internal or private IP), not by expecting db to resolve in Scaleway.

[simple-agent-manager]

Added a follow-up docs clarification in 969c675 to make the decision explicit:

• We should not try to make literal db DNS work on Scaleway Serverless Containers.
• Scaleway does not expose a platform-level Compose-style DNS alias mechanism for managed DB resources.
• Container-to-container private networking is unavailable, so a private DNS/proxy workaround is not a clean option.
• /etc/hosts or entrypoint mutation would be brittle and image-dependent.
• The intended compatibility layer is provider-managed connection string translation: keep the same app env var contract, but set the cloud value to Scaleway's private DB endpoint.

This is now documented in both the iteration notes and the E2E demo README.

[simple-agent-manager]

Scaleway mastra-extended live validation completed on 2026-05-11 using this PR's CI-built CD image retagged to rg.fr-par.scw.cloud/defang-cd/cd:sam-20260511e-pr234 and stack mastraextsam.\n\nEndpoint validated: https://mastraextended4036fb1c-app.functions.fnc.fr-par.scw.cloud\n\nProof screenshots are uploaded via the CLI PR branch:\n\n\n\n\n\nValidation details: generated 10 tasks + 10 events, dashboard reported 20 classified items and zero failed queue jobs, then asked a chat question and captured the answer in-browser with Playwright. The delegated Defang DNS name did not resolve immediately, so browser validation used Scaleway's native Serverless Container domain.