Skip to content

mirror: improved channel handling, v3 -> v4 reencoding support - #268

Merged
cpu merged 4 commits into
mainfrom
ci/mirror-v4
Oct 21, 2026
Merged

cpu merged 4 commits into
mainfrom
ci/mirror-v4

Conversation

@cpu

@cpu cpu commented Oct 19, 2026

Copy link
Copy Markdown
Member

A few mirror fixes/improvements:

  1. The way the mozilla-crlite mirroring code was written would break deserialization when Mozilla adds a new channel type. The Channel enum would reject values other than default or compat.
  2. It'd be handy to be able to specify the channel via CLI flag, so that when a V4 enabled channel appears ahead of the default channel adopting that encoding, we could use it immediately.
  3. Since it's going to be quite a wait for the default channel to adopt v4, have the mirror re-encode v3 to v4. It's untenable to do that downstream for cache dir consumers not able to take a rust bincode dep, and the process is straight forward for the mirroring tool. Care is taken to avoid unnecessary digest churn from the re-encoding, and to preserve the v3 data format for clients reading from the legacy location (without the /revocation dir).

@cpu cpu self-assigned this Oct 19, 2026
@codspeed

codspeed Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 3 untouched benchmarks


Comparing ci/mirror-v4 (f0a4a68) with main (8a8d266)

Open in CodSpeed

@cpu

This comment was marked as outdated.

@cpu

This comment was marked as outdated.

@cpu

cpu commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

maybe just an artifact of the upstream revocation data changing relative to the baseline?

I re-ran the benchmark job on main to refresh the baseline, and then re-ran it on this PR branch, and as expected it now reports no regression 🤷

Comment thread upki-mirror/src/bin/mozilla-crlite.rs Resolved
Comment thread upki-mirror/src/bin/mozilla-crlite.rs Resolved
Comment thread upki-mirror/src/bin/mozilla-crlite.rs Resolved
@djc

djc commented Oct 19, 2026

Copy link
Copy Markdown
Member

Do we know how/when Canonical consumes our mirroring tooling?

@cpu
cpu marked this pull request as draft October 20, 2026 23:23
@cpu
cpu marked this pull request as ready for review October 21, 2026 00:08
@cpu

cpu commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Do we know how/when Canonical consumes our mirroring tooling?

I was hoping you would 😆 Should we reach out to ask?

quality lints / audit (pull_request) Failing after 24s

#269

cpu added 4 commits October 21, 2026 08:19
Mozilla is free to add new channel names at any time, and it shouldn't
break deser. This commit preserves unknown channel names in a distinct
variant to avoid that. Along the way I added test coverage for other
"the JSON might change" properties that are worth guarding against (new
fields, etc).
Add a `--channel` CLI flag that preserves the existing default, but
could be used to accept a different channel name. Limit the full filter
and incremental chain to the requested channel, and fail when its root
is missing.
Move the existing hash and size checks, file write, and manifest entry
construction into write_filter().
It's going to be some time (at least one ESR cycle) before the upstream
Mozilla clubcard data in the default channel is encoded in v4 format. To
aid ecosystem convergence on v4, re-encode the legacy filters before
mirroring them.

Repeated mirror runs should produce the same hashes for the same content,
including when converting v3 to v4, so sort the coverage entries during
conversion.

Lastly, the existing CI mirror job maintains the legacy data and
manifest format at the mirror root for clients that don't support v4.
A new `--encoding` flag lets CI explicitly request the older v3 encoding
for this part of the mirror.
@djc

djc commented Oct 21, 2026

Copy link
Copy Markdown
Member

I was hoping you would 😆 Should we reach out to ask?

I guess they'll holler if something breaks?

@cpu
cpu added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit c2ba5e3 Oct 21, 2026
27 checks passed
@cpu
cpu deleted the ci/mirror-v4 branch October 21, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants