docs(dir): document zot storage config for remote object storage by conorbronsdon · Pull Request #1954 · agntcy/dir

conorbronsdon · 2026-07-31T18:33:53Z

What this changes

Documentation and chart-values comments only. No Go code, no behaviour change.

The chart ships a zot config.json that uses local filesystem storage and does not set dedupe. Zot defaults dedupe to true, and zot rejects dedupe: true on remote storage without a remote cache driver, so an operator who swaps the storage block for an S3 storageDriver gets a registry that will not boot. Nothing in the deployment docs covered which storage layouts are supported.

docs/content/dir/dir-prod-deployment.md — new Zot Storage Backend section: the two supported layouts, a config snippet for each S3 option (dedupe off, or dedupe on with a DynamoDB cacheDriver), credential guidance, and two new troubleshooting entries keyed on the exact error strings.
install/charts/dir/values.yaml — warning comment directly above the zot configFiles block. The config.json content itself is unchanged.
docs/content/dir/dir-component-store.md — cross-reference.

Verification

Run against zot v2.1.18 — the image tag shipped by the zot subchart (0.1.122, pinned in install/charts/dir/apiserver/Chart.lock), which matches ZOT_VERSION in Taskfile.vars.yml and the compose files after #1923 aligned them. The JSON in the two config snippets is extracted straight out of the committed markdown, so what is tested is what is documented.

Config	`zot verify`	`zot serve`
shipped `config.json`, local `rootDirectory`	valid	starts, `GET /v2/` → 200
shipped `config.json` + S3 `storageDriver`, `dedupe` unset	rejected	exit 1
documented Option A (`dedupe: false`)	valid	reaches S3 (config accepted)
documented Option B (`dedupe: true` + dynamodb `cacheDriver`)	valid	reaches the DynamoDB client (no DynamoDB in the harness)
Option A with `dedupe: false` removed	rejected	—
Option A with `storage.rootDirectory` removed	valid	exit 1, `no storage config provided`

The matrix was first run on v2.1.16 (the version in the bug report) and then re-run on v2.1.18. The zot verify column is identical on both versions; the serve results for the two documented failure modes (dedupe unset, rootDirectory missing) were re-confirmed on v2.1.18, the latter with the exact no storage config provided error. The MinIO push test was run on v2.1.16 only.

Three findings that differ from the report, and one limit on what was tested:

Dropping the search extension is not required, and gc: false was not needed to boot. Only dedupe is gated by config validation. Option A boots on S3 with GC at its default and search enabled. This is a startup result only — GC runs on an interval (gcInterval/gcDelay, one hour by default), so a clean boot does not exercise it, and it does not contradict the reporter's experience of needing gc: false in a running deployment. The doc says exactly this and keeps gc: false as the documented fallback.
zot verify is not sufficient. A config missing storage.rootDirectory passes verify and then crash-loops.
Keys inside storageDriver are pass-through and unvalidated. A misspelled credential key (accessKeyId for accesskey) still passes zot verify and fails only when zot contacts S3. Zot rejects unknown keys in storage but not in storageDriver — confirmed against v2.1.18 in both directions. The doc calls this out.

Credential parameter names (accesskey, secretkey, regionendpoint) and IAM-role support are per the zot storage planning docs; the environment-variable chain was confirmed in the harness.

helm lint ./install/charts/dir --with-subcharts → 2 charts linted, 0 failed. pymarkdown and codespell clean on both changed docs.

Minimal reproduction of the failing case:

# storage block = the shipped config.json with an S3 storageDriver added, dedupe left unset
docker run --rm -v "$PWD:/cfg:ro" ghcr.io/project-zot/zot-linux-amd64:v2.1.18 verify /cfg/config.json
# Error: invalid server config: invalid database config, dedupe set to true with
# remote storage and database, but no remote database configured

Zot defaults `dedupe` to true. The zot config.json shipped in the chart does not set it, so an operator who swaps `storage` for an S3 `storageDriver` gets a registry that refuses to start: invalid database config, dedupe set to true with remote storage and database, but no remote database configured Nothing in the deployment docs covered this, so it was only discovered at boot on a production node. Document the two supported storage layouts for a production node, with a verified config snippet for each S3 option (dedupe off, or dedupe on with a DynamoDB cacheDriver), and add the two corresponding startup failures to the troubleshooting section. Add the same warning next to the config in values.yaml, where an operator editing the storage block will see it. Verified against zot v2.1.16 (the version the chart pulls) with MinIO standing in for S3: the shipped config plus an S3 storageDriver exits 1 at startup; adding `dedupe: false` starts and serves, and a push lands in the bucket. Garbage collection and the search extension do not have to be disabled, contrary to the workaround in the report. Fixes agntcy#1885 Signed-off-by: Conor Bronsdon <cmbronsdon@gmail.com>

codecov · 2026-07-31T18:46:11Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…ntials Follow-up review fixes on the zot remote-storage docs: - The verify command pinned zot v2.1.16, which is the version from the bug report, not the version the chart deploys. The zot subchart (0.1.122, pinned in apiserver/Chart.lock) ships image tag v2.1.18, matching ZOT_VERSION in Taskfile.vars.yml and the compose files after agntcy#1923 aligned them. Re-ran the full config matrix on v2.1.18: every documented claim reproduces unchanged. - The gc/search note asserted that neither has to be disabled on remote storage, on the strength of a successful boot. GC runs on an interval (gcInterval/gcDelay, 1h by default), so a clean boot does not exercise it. Narrowed the claim to what was tested -- neither blocks startup -- and kept the reporter's gc: false as the fallback if GC errors appear later. - Neither S3 snippet mentioned credentials. Added the IAM-role-first guidance plus accesskey/secretkey and regionendpoint, and a note that storageDriver keys are pass-through and unvalidated, so a misspelled key still passes zot verify. - values.yaml pointed at a repo-relative docs path; the chart is published to OCI, so consumers never see that tree. Switched to the published URL. config.json is unchanged. Signed-off-by: Conor Bronsdon <cmbronsdon@gmail.com>

The Cross-Platform Test Linux job failed provisioning the e2e daemon testenv: the sentence-transformers/all-MiniLM-L6-v2 fetch from huggingface.co was reset by peer. Network flake, unrelated to this docs-only change. Signed-off-by: Conor Bronsdon <cmbronsdon@gmail.com>

paralta

LGTM 👍 Thank you so much for investigating this and for adding clear documentation to fix the reported issue!

keraron

Hi, some formatting changes and nits.

Co-authored-by: Aron Kerekes <arkereke@cisco.com> Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

Suggestions from reviewer applied Co-authored-by: Aron Kerekes <arkereke@cisco.com> Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

The Docs CI job failed on two lint errors in dir-prod-deployment.md: :136:1 MD009 Trailing spaces [Expected: 0 or 2; Actual: 4] :137:1 MD012 Multiple consecutive blank lines [Expected: 1, Actual: 3] Those were the symptom. Applying the code-review suggestions in 6b9da18 converted the two bold option headers into a mkdocs content tab, but the replacement kept only the first tab marker and dropped the body: both JSON config blocks and the entire Option B tab went with it. The section then read "Pick one of the following when moving Zot to S3:", offered an empty Option A, had no Option B, and closed with "Neither snippet sets S3 credentials" - referring to two snippets that were no longer there. storageDriver mentions went 10 -> 8 in that commit, which is what surfaced it. Restored both snippets as proper content tabs, which is what the suggestion was asking for, with the bodies indented four spaces so they render inside the tabs. Option A disables dedupe; Option B keeps it and adds the DynamoDB cacheDriver. The trailing-whitespace line that tripped MD009/MD012 is gone as a consequence. Verified with the project's own linter and config rather than by eye: pymarkdown --config docs/pymarkdown.yaml reproduces both errors at 136 and 137 on the previous commit and reports nothing here. Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

conorbronsdon · 2026-08-03T11:13:51Z

Thanks @paralta + @keraron - @keraron fixed your formatting changes/nits

Edit: I think the failing check is a flake?

github-actions Bot added the size/XS Denotes a PR that changes 0-49 lines label Jul 31, 2026

conorbronsdon marked this pull request as ready for review July 31, 2026 19:06

conorbronsdon requested a review from a team as a code owner July 31, 2026 19:06

conorbronsdon added 2 commits July 31, 2026 12:24

conorbronsdon mentioned this pull request Jul 31, 2026

[Bug]: bundled zot on remote (S3) storage fails to start with dedupe enabled — undocumented for dir operators #1885

Open

2 tasks

Merge branch 'main' into fix/issue-1885

587c52b

keraron self-requested a review August 3, 2026 07:52

paralta approved these changes Aug 3, 2026

View reviewed changes

keraron requested changes Aug 3, 2026

View reviewed changes

conorbronsdon and others added 2 commits August 3, 2026 04:02

Update docs/content/dir/dir-prod-deployment.md

75d6890

Co-authored-by: Aron Kerekes <arkereke@cisco.com> Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

Update docs/content/dir/dir-prod-deployment.md

7af41ac

Co-authored-by: Aron Kerekes <arkereke@cisco.com> Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

conorbronsdon commented Aug 3, 2026

View reviewed changes

Comment thread docs/content/dir/dir-prod-deployment.md Outdated

conorbronsdon and others added 4 commits August 3, 2026 04:03

Apply suggestion from @conorbronsdon

0a307f6

Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

Apply suggestions from code review

6b9da18

Suggestions from reviewer applied Co-authored-by: Aron Kerekes <arkereke@cisco.com> Signed-off-by: Conor Bronsdon <120674402+conorbronsdon@users.noreply.github.com>

Merge branch 'main' into fix/issue-1885

67a59f5

keraron approved these changes Aug 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(dir): document zot storage config for remote object storage - #1954

docs(dir): document zot storage config for remote object storage#1954
conorbronsdon wants to merge 10 commits into
agntcy:mainfrom
conorbronsdon:fix/issue-1885

conorbronsdon commented Jul 31, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 31, 2026

Uh oh!

paralta left a comment

Uh oh!

keraron left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

conorbronsdon commented Aug 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

conorbronsdon commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Verification

Uh oh!

codecov Bot commented Jul 31, 2026

Codecov Report

Uh oh!

paralta left a comment

Choose a reason for hiding this comment

Uh oh!

keraron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

conorbronsdon commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

conorbronsdon commented Jul 31, 2026 •

edited

Loading

conorbronsdon commented Aug 3, 2026 •

edited

Loading