VFS: add .sdzst support (zip with zstd / method-93 entries)#3055
Draft
tomjn wants to merge 3 commits into
Draft
Conversation
Adds a new .sdzst archive format: a zip container whose entries use Zstandard (PKWARE compression method 93). The vendored minizip reader is left almost untouched - the only change is whitelisting method 93 in its two method-validation guards so such an entry can be opened in raw mode. All zstd handling lives in CZipArchive::GetFileImpl, which reads the raw frame and decompresses it with libzstd, verifying the CRC manually (raw reads bypass minizip's CRC-on-close check). zstd decoding is global: existing .sdz archives that happen to contain method-93 entries now read too; .sdzst is a naming/signalling convention, registered as the same CZipArchive behind a new "sdzst" factory. The decoder reuses the already-vendored Tracy zstd sources, built as a decompress-only static lib (zstd_dec), so no new external dependency is introduced. Read-only; the zip writer is unchanged.
The engine links a prebuilt minizip (from spring-static-libs), so the earlier source patch to rts/lib/minizip was not used in CI: ZipArchive failed to build (Z_ZSTDED undeclared) and, even fixed, the prebuilt minizip would still reject method 93 on its open path at runtime. Revert the minizip changes and instead read zstd entries self-contained: minizip still enumerates method-93 entries fine (the method is only validated on the decode-open path), so CZipArchive::GetFileZstd locates the raw frame via the zip central-directory / local-file headers (from the unz_file_pos we already store per entry) and decompresses it with libzstd, verifying the CRC manually. Zip64 local-header offsets handled. Verified by round-tripping a real Python-authored method-93 .sdzst (text, lua, empty-file) byte-identically against a pristine minizip.
Contributor
Author
|
Some notes:
|
The previous commit built a zstd_dec static lib out of Tracy's internal vendored zstd sources, which couples the VFS build to Tracy's layout and would break on a Tracy update. Source zstd as a first-class dependency instead: add FindZstd.cmake (modelled on FindMiniZip.cmake) and resolve it with find_package_static, the same way zlib and minizip are found from the static-libs / mingwlibs bundle. SPRING_ZSTD_INCLUDE_DIR / SPRING_ZSTD_LIBRARY now point at the found package; the consuming targets are unchanged. Note: this requires libzstd to be present in the static-libs / mingwlibs bundles. Until that is added CI cannot link, which is expected.
Contributor
Author
|
As for why we might want zstd: 
It gives comparable compression to sd7 with performance better than sdz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new VFS archive format
.sdzst- a zip container whose entries are Zstandard-compressed (PKWARE compression method 93) - alongside the existing.sd7(7-Zip) and.sdz(zip/deflate) formats.Approach
The vendored minizip is not modified. minizip can already enumerate method-93 entries (the compression method is only validated on its decode-open path, not on info reads), so:
CZipArchive::GetFileZstdreads a zstd entry by locating its raw frame directly from the zip central-directory / local-file headers (using theunz_file_posminizip already gives us per entry) and decompresses it with libzstd, verifying the CRC manually. Zip64 local-header offsets are handled.rts/lib/minizipare not used - this is what an earlier patch-based attempt tripped over in CI).CZipStdArchiveFactory("sdzst")registers the sameCZipArchive;GetType()staysARCHIVE_TYPE_SDZ, so.sdzstis treated as a compressed archive everywhere for free.zstd_dec) built from the already-vendored Tracy zstd sources - no new external dependency.Decoding is global: an existing
.sdzcontaining method-93 entries will now also read. The zip writer is untouched (read-only feature).Verification
GetFileZstd), built with gcc-16 against a pristine, unpatched minizip (matching the prebuilt-lib build), round-trips 3/3 entries of a real method-93.sdzstauthored by Python 3.14'sZIP_ZSTANDARD- a multi-KB text file, a tiny Lua file, and an empty-file edge case - read back byte-identical with CRC validation.zstd_decsource set compiles under gcc-16 and clang.Not yet verified locally
.sdz/.sd7are unaffected: the zstd branch is gated oncompression_method == 93and the existing deflate/store path is unchanged.Notes
.sdzstfiles is left to external tooling (e.g. Python 3.14zipfile.ZIP_ZSTANDARD); this PR adds read support only.