Repository navigation
mirror: improved channel handling, v3 -> v4 reencoding support - #268
Conversation
Merging this PR will not alter performance1 warning
Comparing |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
I re-ran the benchmark job on |
|
Do we know how/when Canonical consumes our mirroring tooling? |
I was hoping you would 😆 Should we reach out to ask?
|
Mozilla is free to add new channel names at any time, and it shouldn't break deser. This commit preserves unknown channel names in a distinct variant to avoid that. Along the way I added test coverage for other "the JSON might change" properties that are worth guarding against (new fields, etc).
Add a `--channel` CLI flag that preserves the existing default, but could be used to accept a different channel name. Limit the full filter and incremental chain to the requested channel, and fail when its root is missing.
Move the existing hash and size checks, file write, and manifest entry construction into write_filter().
It's going to be some time (at least one ESR cycle) before the upstream Mozilla clubcard data in the default channel is encoded in v4 format. To aid ecosystem convergence on v4, re-encode the legacy filters before mirroring them. Repeated mirror runs should produce the same hashes for the same content, including when converting v3 to v4, so sort the coverage entries during conversion. Lastly, the existing CI mirror job maintains the legacy data and manifest format at the mirror root for clients that don't support v4. A new `--encoding` flag lets CI explicitly request the older v3 encoding for this part of the mirror.
I guess they'll holler if something breaks? |
A few mirror fixes/improvements:
mozilla-crlitemirroring code was written would break deserialization when Mozilla adds a new channel type. TheChannelenum would reject values other thandefaultorcompat.defaultchannel adopting that encoding, we could use it immediately.defaultchannel to adopt v4, have the mirror re-encode v3 to v4. It's untenable to do that downstream for cache dir consumers not able to take a rust bincode dep, and the process is straight forward for the mirroring tool. Care is taken to avoid unnecessary digest churn from the re-encoding, and to preserve the v3 data format for clients reading from the legacy location (without the/revocationdir).