fix: handle raw-deflate FlateDecode and drop unusable raw images (#615)#625
Closed
kh3rld wants to merge 2 commits into
Closed
fix: handle raw-deflate FlateDecode and drop unusable raw images (#615)#625kh3rld wants to merge 2 commits into
kh3rld wants to merge 2 commits into
Conversation
…xtractor Second occurrence of the class of issue fixed in #519. Two sites in docx/parser.rs matched attr.value.as_ref() without an explicit &[u8] cast, causing the compiler to lock the type to &[u8; 5] from the first arm and reject the other arms ("separate" = 8 bytes, "end" = 3 bytes). Two sites in fictionbook.rs called normalized.as_ref() on a Cow<str> without an explicit &str annotation, triggering E0282 on newer toolchains. Fixes #622
Two problems caused FlateDecode images to be returned as format='raw' with still-compressed bytes: 1. decode_flate_to_png only tried ZlibDecoder. PDFs that use raw deflate (no zlib header/footer) silently fell through to the raw fallback. Fix: try DeflateDecoder when ZlibDecoder fails. 2. When pdfium bitmap fallback also fails (e.g. pdfium not installed), images that are still 'raw' with data smaller than width*height*channels were returned to callers — unusable data with no indication of failure. Fix: filter those images out and surface a ProcessingWarning listing the count so callers know extraction was partial. Adds regression test: test_decode_flate_raw_deflate_fallback. Fixes #615
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two separate problems caused FlateDecode images to surface as
format=\"raw\"with still-compressed bytes that callers cannot use:1. ZlibDecoder-only — raw-deflate PDFs always fail
decode_flate_to_pngonly triedflate2::read::ZlibDecoder. Many PDF producers write FlateDecode streams as raw deflate (no zlib header/footer), which ZlibDecoder rejects. These fell straight through to the"raw"fallback regardless of whether pdfium was available. Fix: tryDeflateDecoderwhenZlibDecoderfails.2. Unusable raw images returned silently after all fallbacks
When pdfium is not installed (or
get_processed_imagefails), images still carryingformat="raw"withdata.len() < width * height * channelswere passed to callers — bytes that are definitively the original compressed stream, not pixel data. Fix: filter those images out of the result and surface aProcessingWarningwith the count so callers know image extraction was partial.Changes
pdf/images.rs: fallback fromZlibDecodertoDeflateDecoderindecode_flate_to_png; regression testtest_decode_flate_raw_deflate_fallbackextractors/pdf/mod.rs: afterreextract_raw_images_via_pdfium, drop images whereformat == "raw"anddata.len() < w*h*channels; consolidate fallback warning to cover both pdfium re-extractions and dropped imagesTest plan
cargo test -p kreuzberg --features pdf -- pdf::images::tests— 14 tests pass including new regression testProcessingWarningis presentFixes #615