Quick start guide feedback

[ivanfeanor]

Salut. First of all, thanks for the great product. I've tried to provide some feedback based on my quick (no so quick) start experience to maybe help you improve your documentation and fix a few things.

Self-hosting on Kubernetes (Azure AKS) — Documentation & Chart Issues

I deployed Probo v0.192.0 to AKS with external Azure Postgres Flexible Server and (initially) Azure Blob Storage via S3Proxy, following the Kubernetes self-hosting docs. The deployment ultimately works, but I hit a number of blockers that took hours to diagnose — most of them avoidable with documentation or small chart fixes. Sharing in case it helps prioritize improvements.

Environment

• AKS, Kubernetes 1.35, Helm 4.2
• Azure Postgres Flexible Server (PostgreSQL 18.3, France Central)
• Azure Blob Storage → AWS S3 (after S3Proxy failed)
• Traefik 3.x + cert-manager (Let's Encrypt)
• Probo chart 0.1.0 / app 0.192.0

Issues encountered, roughly in order

1. OCI chart registry returns 401 for anonymous pulls

helm install probo oci://artifact.probo.inc/charts/probo (from the docs) returns 401 Unauthorized: project charts not found. Anonymous pulls don't work; no auth instructions in the docs. Worked around by cloning the repo and installing from contrib/helm/charts/probo.

Suggestion: either expose the chart anonymously, publish it to a public registry (Artifact Hub, etc.), or document the auth flow.

2. Docs --set postgresql.host=... doesn't actually wire up the database

The Kubernetes docs example uses:
--set postgresql.host="..." --set postgresql.password="..."
But the Deployment template reads .Values.probo.pg.addr, not .Values.postgresql.host. Result: chart fails at render time with nil pointer evaluating interface {}.addr. Two parallel postgres config blocks (postgresql: and probo.pg:) is confusing — only the second is read by the template.

Suggestion: unify the config, or at least document the probo.pg.* keys in the README.

3. Chart is missing OAUTH2_SERVER_SIGNING_KEY env var

The chart's secret.yaml and deployment.yaml don't include OAUTH2_SERVER_SIGNING_KEY, but probod-bootstrap requires it (along with AUTH_COOKIE_SECRET etc.). Pod crashlooped with missing required environment variables: OAUTH2_SERVER_SIGNING_KEY.

Worked around by patching the chart templates locally to add probo.oauth2.signingKey → Secret entry → valueFrom.secretKeyRef in the Deployment.

Suggestion: add probo.oauth2.signingKey to the chart, with --set-file instructions in docs (the value must be a PEM RSA private key — also undocumented).

4. encryption-key accepts the wrong format silently

Docs use openssl rand -base64 32. If you generate with openssl rand -hex 32 instead (common reflex), the pod crashes with key must be 32 bytes for AES-256, got 48 bytes after base64 decoding. Hex output is 64 chars → base64-decoded as 48 bytes. Same applies to OAUTH2_SERVER_SIGNING_KEY which must be a PEM RSA key, not a random string. Both errors are clear once they happen, but the chart's required validator only checks "non-empty" — not "valid".

Suggestion: add format validation in secret.yaml (length checks via gt, len, b64dec), and be explicit in the README about the format of each secret. Stating openssl rand -base64 32 would have saved me an hour.

5. No SSL configuration knob for external Postgres

Azure Postgres Flexible Server requires TLS. The chart only exposes postgresql.caBundle and postgresql.caBundlePath — no sslmode flag. Probo's driver apparently uses SSL only when a CA bundle is present, which is implicit and undocumented.

Compounding issue: passing the full Mozilla CA bundle (curl.se/ca/cacert.pem) via --set-file postgresql.caBundle=... causes exec /usr/local/bin/entrypoint.sh: argument list too long — the bundle gets injected as an env var that blows past ARG_MAX. Even a trimmed bundle (~6 KB) hit the same problem because env vars accumulate.

Worked around by creating a separate Secret with the CA bundle, mounting it as a volume, and using caBundlePath. Both caBundle and caBundlePath exist as values, but the chart only wires caBundle into a Secret it manages — for a CA mounted from your own Secret, there's no built-in path.

Suggestion: mount the CA via a Secret-backed volume in the chart by default, or at minimum document that caBundle only works for small bundles. Also expose sslmode / sslrootcert explicitly.

6. Schema migrations require ownership of public schema on PG ≥ 15

If you create a dedicated probod user, the first migration fails with permission denied for schema public. Required:

ALTER SCHEMA public OWNER TO probod;
GRANT ALL ON SCHEMA public TO probod;

Suggestion: call this out explicitly in the docs as part of the DB setup steps for managed PG providers.

7. Migrations create extensions that need cluster-level allow-listing on managed PG

Migration 20250202T141800Z runs CREATE EXTENSION citext. Migration 20250312T165900Z runs CREATE EXTENSION IF NOT EXISTS pg_stat_statements. On Azure Flexible Server (and most managed Postgres providers), these require server-level allow-listing via the azure.extensions parameter. pg_stat_statements additionally requires azure_pg_admin role membership.

The full list of extensions Probo uses: citext, pgcrypto, unaccent, pg_stat_statements.

Suggestion: document this list prominently in the "External PostgreSQL" section. Single biggest source of "why is my install failing" for managed-DB users.

8. Azure Blob Storage via S3Proxy is not actually viable

The S3 storage docs imply Azure Blob works via S3Proxy. In practice:

• azureblob jclouds provider (the default) hits "command is not replayable" errors on the SDK's chunked upload retries
• azureblob-sdk provider passes uploads through, but Azure rejects them with InvalidMetadata: characters not permitted — Probo sets metadata keys with hyphens (organization-id, trust-center-id, file-id, framework-id), which violate Azure's "valid C# identifier" rule
• S3Proxy has open issues for metadata key transformation since 2017 (gaul/s3proxy#243, #466) with no fix

S3Proxy fundamentally cannot work for Probo against Azure Blob Storage without either patching S3Proxy or patching Probo to use underscored metadata keys. Switched to AWS S3 and everything worked instantly.

Suggestion: either drop Azure Blob from the "S3-compatible storage" list (with a clear "not actually supported" note), or — much better — implement native Azure Blob Storage support in Probo. The Azure SDK for Go is mature and would let users keep data in their existing Azure tenancy without the S3 translation layer. Given how many users will arrive from Azure-shop environments, first-class Blob support would be a real differentiator.

9. Service selector matches both the main app and the Chrome pod

The chart's _helpers.tpl defines probo.selectorLabels with only name and instance. The Chrome Deployment correctly adds app.kubernetes.io/component: chrome to its template labels AND its Service selector. The main Probo Deployment does not add a corresponding component: server to its template labels.

Result: the main probo Service selects both pods. Traffic to ports 80, 443, 8080 occasionally hits the Chrome pod, which doesn't listen on those, causing intermittent 502s. Also affects kubectl port-forward svc/probo which fails with Pod 'probo-chrome-...' does not have a named port 'backoffice'.

Workaround: kubectl patch the Service selector to add component: server and the Deployment template to add a matching pod label. But Helm doesn't track those patches, so every helm upgrade reverts them and requires --force-conflicts + re-patching.

Suggestion: add app.kubernetes.io/component: server to the main Probo Deployment's pod labels and to the Service selector in the chart. One-line fix per file. Same convention the Chrome Deployment already uses.

10. Misc smaller things

• Chart.yaml only lists one dependency (haproxy-ingress) but the default values enable both haproxy-ingress and a cloudnative-pg-based Postgres demo — the postgresql-demo-app template lives directly in templates/ with no dependency hook, which means disabling postgresql.enabled is the only way to use external PG. Documented behavior, but the values structure makes it confusing.
• IP geolocation table is empty; run geoloc-import to populate it — warning at every startup with no mention in the docs of what geoloc-import is, where it lives, or whether it's needed.
• no account key provided, generating new ACME account - this will create a new account on each restart — for the trust-center custom-domains feature, the chart's customDomains.acme.accountKey should be wired through to a Secret so it persists, otherwise users get rate-limited by Let's Encrypt.
• i think somewhere i saw that there should be known third parties (like google/microsoft), maybe in the code, but for now i have to add everything by hand

What worked great

• Bootstrap entrypoint that generates the config from env vars at startup — very clean.
• probod-bootstrap failing loudly with a clear list of missing env vars is genuinely helpful.
• Migration system + clear logging of each migration as it runs.
• Once everything was configured correctly, the product itself starts up cleanly and the UI is solid.

[SachaProbo]

Thanks for your feedbacks. We are going to look into it.

[Radiergummi]

Also, has anyone noticed the quick start section on the website links to a missing compose file?

(1) Download the Docker Compose file

curl -sL https://raw.githubusercontent.com/getprobo/probo/refs/heads/main/deploy/docker-compose/docker-compose.yml -o docker-compose.yml

(2) Start the services

docker compose up -d

This starts all required services: the Probo server, PostgreSQL database, and SeaweedFS for file storage.

The repository seems to still contain compose files, but I don't know whether I'm supposed to use production or not, and it also requires a full checkout of the entire repository since it's referencing other files.

[gearnode]

Thank you for the detailed write-up, Ivan.

Most of the issues you raised have been addressed in the chart since this was filed. Here is a quick rundown:

Fixed in helm/v0.2.0 and later:

• Issue 2 (postgresql.host not wiring up): The chart now aligns postgresql.host/postgresql.port with the internal PG_ADDR env var. The parallel config blocks are gone.
• Issue 3 (missing OAUTH2_SERVER_SIGNING_KEY): The secret template and deployment now include OAUTH2_SERVER_SIGNING_KEY, with --set-file instructions in the README for the PEM key.
• Issue 4 (encryption-key format silently accepted): Early validation now checks that the base64-decoded key is exactly 32 bytes, and that the OAuth2 signing key is a valid PEM block. Bad values now fail loudly at render time rather than at pod startup.
• Issue 9 (service selector hitting the Chrome pod): The main Deployment now sets app.kubernetes.io/component: server on its pod labels, and the Service selector has been narrowed to match only those pods. The Chrome component already used this convention, so the chart is now consistent.

Documentation improved:

• Issue 5 (SSL for external Postgres): The README now documents the caBundle vs caBundlePath distinction, notes that large CA bundles must use the volume-mount path, and explains that SSL is enabled whenever a CA bundle is present.
• Issue 6 (public schema ownership on PG 15+): The external PostgreSQL setup section now calls out the ALTER SCHEMA public OWNER TO probod and GRANT steps explicitly.
• Issue 7 (extensions require allow-listing): The docs now list the full set of extensions Probo uses (citext, pgcrypto, unaccent, pg_stat_statements) and note the Azure Flexible Server allow-list requirement.
• Issue 8 (Azure Blob via S3Proxy): The storage section now includes a clear note that S3Proxy is not a viable path for Azure Blob Storage due to the metadata key restrictions, and recommends AWS S3 or a native-compatible S3 endpoint instead.
• Issue 10 / ACME key persistence: The chart values now include customDomains.acme.accountKey, wired through to a Secret, so restarts no longer generate a new ACME account.

Still open / tracking separately:

• Issue 1 (OCI registry 401): We are working on making the chart available on a public registry. For now, installing from the repo (contrib/helm/charts/probo) remains the workaround.
• Azure Blob native support: First-class Azure Blob Storage is on the roadmap. The S3Proxy path will stay documented as unsupported until then.
• Known third parties (Issue 10, last bullet): Pre-seeding well-known vendors (Google, Microsoft, etc.) is something we want to add; no ETA yet but we have it tracked.
• geoloc-import warning: We will add a note to the docs explaining what this command does and that it is optional (geo-based reporting only).

Thanks again for the thoroughness here. Issues like this make the self-hosting experience meaningfully better for everyone.

[confused-moniker]

Adding to this thread, but I wanted to try Probo with docker-compose following the quick start guide here: https://www.probo.com/docs/getting-started/quickstart.

I ran into the following issues:

• Some missing configuration like OAUTH2_SERVER_SIGNING_KEY mentioned above.

But also several of the images (forks of public docker images) are not public:

artifact.probo.inc/dockerhub/chrislusf/seaweedfs@sha256:cea8339d21dad1b200adce581dd7434d254b8f5975f142c3b4c930ba78647eef

artifact.probo.inc/dockerhub/chromedp/headless-shell@sha256:b24482ae166e2c67135f5a8ba9575c257efdd8e2fd6b2e931f9d88ede3d72f3b

Also, I would assume the postgres docker image hosted at artifact.probo.inc is also not publicly available.

Note: I was able to pull from the actual Probo Docker image at artifact.probo.inc/probo/probo; just all infrastructure images appear to be private from my testing.

[somedevchap]

@gearnode The docker issue persists. Is there a workaround or ETA on an install fix? Thanks!

[gearnode]

@somedevchap the compose issue should be fixed now.

[somedevchap]

@gearnode Thank you. Almost works, just getting:

✘ Container probo-postgres-1 Error dependency postgres failed to start

[ciscocollabguru]

@gearnode Thank you. Almost works, just getting:

✘ Container probo-postgres-1 Error dependency postgres failed to start

I had the same thing. Ran "docker compose down -v" to nuke the volumes and Postgres started up no issue

[ciscocollabguru]

I'm running into some other issues when trying to follow the Quickstart instructions with actually accessing the platform after everything appears to start properly. Looking at running containers though, probo doesn't appear in the list. No errors or indications that it failed for some reason. Not sure if there's a missing step in the guide or if this should be considered a separate issue.

[somedevchap]

@ciscocollabguru docker compose down -v doesn't seem to make a difference.

[ciscocollabguru]

@ciscocollabguru docker compose down -v doesn't seem to make a difference.

What are you seeing in the compose logs for Postgres?

[somedevchap]

Logs showed postgres was a memory issue. Resolved that, everything running now (thanks!) - however, localhost: connection refused.

Error in logs..

probo-1 | Generating configuration file from environment variables at: /etc/probod/config.yml
probo-1 | error: missing required environment variables:
probo-1 | - OAUTH2_SERVER_SIGNING_KEY

[FLWFTW]

I'm also running into issues following both the quick-start guide and the self-hosting docker compose guide.

Step 2-3 of the docker compose guide contains the following snippet:

export PROBOD_ENCRYPTION_KEY=$(openssl rand -base64 32)
export AUTH_COOKIE_SECRET=$(openssl rand -base64 32)
export AUTH_PASSWORD_PEPPER=$(openssl rand -base64 32)
export TRUST_AUTH_TOKEN_SECRET=$(openssl rand -base64 32)

cat > .env << EOF
PROBOD_ENCRYPTION_KEY=${PROBOD_ENCRYPTION_KEY}
AUTH_COOKIE_SECRET=${AUTH_COOKIE_SECRET}
AUTH_PASSWORD_PEPPER=${AUTH_PASSWORD_PEPPER}
TRUST_AUTH_TOKEN_SECRET=${TRUST_AUTH_TOKEN_SECRET}
PROBOD_BASE_URL=https://your-domain.com
API_ADDR=0.0.0.0:8080
API_CORS_ALLOWED_ORIGINS=https://your-domain.com
SMTP_ADDR=smtp.example.com:587
SMTP_USER=your-smtp-username
SMTP_PASSWORD=your-smtp-password
SMTP_TLS_REQUIRED=true
MAILER_SENDER_EMAIL=no-reply@your-domain.com
EOF

However the correct format for the AUTH_COOKIE_SECRET, AUTH_PASSWORD_PEPPER, TRUST_AUTH_TOKEN_SECRET, and API_CORS_ALLOWED_ORIGINS variables are PROBOD_AUTH_COOKIE_SECRET, PROBOD_AUTH_PASSWORD_PEPPER, PROBOD_TRUST_AUTH_TOKEN_SECRET, and PROBOD_API_CORS_ALLOWED_ORIGINS , respectively.

Additionally, probod currently errors out with the same logs mentioned by @somedevchap

probo-1 | Generating configuration file from environment variables at: /etc/probod/config.yml
probo-1 | error: missing required environment variables:
probo-1 | - OAUTH2_SERVER_SIGNING_KEY

Attempted to fix by setting OAUTH2_SERVER_SIGNING_KEY to the path to a PEM encoded private key and it still gives the same complaint.

Any advice on how to procede?