ADFA-5552: Add media optimization space-savings report

Shareable writeup of what was optimized, the exact per-format settings, the
in-place (no-rename) design decision, safety properties, and how to reproduce.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: alexmmiller

MEDIA_OPTIMIZATION_REPORT.md

@@ -0,0 +1,74 @@
+# Documentation Database — Media Optimization Report
+
+**Ticket:** ADFA-5552 · **Branch:** `fix/ADFA-5552`
+
+## Summary
+
+Every image in `documentation.db` was optimized in place, reducing the database file by **26%** with no change to how any page references its media and no loss of any image.
+
+| Metric | Before | After | Reduction |
+|---|---:|---:|---:|
+| **Database file** | 249.1 MB | 183.6 MB | **−65.5 MB (−26.3%)** |
+| Optimizable image bytes | 130.2 MB | 65.7 MB | −64.5 MB (−49.5%) |
+| Images optimized | — | 1,081 | — |
+
+Integrity verified afterward: `PRAGMA integrity_check` and `foreign_key_check` both pass, and all 1,192 image records reassemble and decode correctly.
+
+## Scope
+
+Optimization was applied to **every media file in the database**, across all documentation sets (Android `a/`, IntelliJ `i/`, Kotlin `k/`, `p/`, Java `j/`) — not just the Kotlin website. It reuses the optimization pipeline from PR #24 (the Kotlin website's `optimize_media.py`) and applies it to the images already stored in the database.
+
+Videos (`mp4`, `quicktime`) and favicons (`x-icon`) were intentionally left untouched — the pipeline does not re-encode video.
+
+## Savings by media type
+
+| Type | Images optimized | Bytes saved | Method |
+|---|---:|---:|---|
+| PNG | 720 | 57.84 MB | pngquant + downscale |
+| GIF | 61 | 5.70 MB | per-frame downscale (animation preserved) |
+| WebP | 161 | 0.58 MB | re-encode + downscale |
+| SVG | 126 | 0.30 MB | Scour minification |
+| JPEG | 13 | 0.09 MB | re-encode + downscale |
+| **Total** | **1,081** | **64.51 MB** | |
+
+PNG downscaling is essentially the entire win. A further 109 images were already minimal and left as-is, and 14 were left untouched because re-encoding them would have *increased* their size (see "never enlarge" below).
+
+## Methods — exact settings
+
+All raster images are downscaled with a Lanczos filter to a **maximum width of 500px**, preserving aspect ratio, and **never upscaled** (an image already ≤500px wide keeps its dimensions). Then, per format:
+
+- **PNG** — quantized with **pngquant** (`--quality 65-95`, `--speed 4`, metadata stripped). pngquant is run at full resolution first (so its palette selection sees the original color detail), then the image is downscaled, then quantized again at the delivered size.
+- **JPEG** — re-encoded at **quality 82**, progressive, with optimized Huffman tables.
+- **GIF** — every frame downscaled individually; frame count, per-frame durations, and loop count are preserved so animations keep playing. Static GIFs are simply resized.
+- **WebP** — re-encoded at **quality 80**, method 6 (maximum compression effort). Animated WebP is left untouched.
+- **SVG** — minified with **Scour**: metadata/comments/editor cruft stripped, IDs shortened, styles converted to attributes, groups collapsed, and numbers rounded to **4 decimal places**. SVGs are never rasterized to PNG (that would rename the file — see below).
+
+## Key design decision: optimize *in place*, no format changes
+
+The Kotlin website pipeline (PR #24) converts images to WebP, which **renames** files (e.g. `foo.png` → `foo.webp`), and then rewrites the image references embedded in Kotlin's stored pages. That reference-rewriting step only understands Kotlin's page format.
+
+The other documentation sets don't share that format. For example, an Android page references its media by **absolute URL** (`https://developer.android.com/images/…`), which has no literal link to the path the file is stored under (`a/devsite/media/…`) that a text substitution could follow. **Renaming media outside Kotlin therefore can't be done safely by this tooling.**
+
+The solution is to optimize each image **in place** — same path, same file extension, same content type — so every reference, however it's written, keeps resolving to the same, now-smaller file. This is why the process never converts to WebP and never rasterizes an SVG. It trades a little additional savings (WebP would shrink things further) for correctness and uniform coverage across all doc sets.
+
+## Correctness and safety properties
+
+- **Never enlarges an image.** A stored image is only replaced when the optimized result is actually smaller. Images that would grow on re-encode (several already-small animated GIFs) are left exactly as they were.
+- **Backup first.** The database is copied to a timestamped file (`documentation.db.backup-YYYYMMDD-HHMMSS`) before any change.
+- **Single transaction.** All updates happen in one transaction and are rolled back on any error; the file is then `VACUUM`ed to reclaim freed space. A `--dry-run` mode does all the work and reports savings without writing anything.
+- **Handles chunked media.** Large files are stored split across multiple 1 MB database rows. These are reassembled into the whole image before optimizing and re-split on write, agreeing exactly with how the server reads them back. (Verified: every reassembled image — including 39 continuation rows folded into their bases — decodes correctly.)
+- **Handles both database generations.** Older databases store SVG/WebP as plain Brotli; newer ones compress them against a shared 256 KB Brotli dictionary (from the `CompressionDictionary` table). The tool detects which and, for dictionary databases, round-trips those rows through the `brotli` CLI using the dictionary read from the database itself. The decompress→optimize→recompress→decompress round-trip was confirmed lossless before any write.
+
+## Reproducing / running it
+
+```bash
+uv run --with-requirements requirements.txt scripts/optimize_db_media.py <path-to>.db --max-width 500
+```
+
+Add `--dry-run` to preview savings without modifying the database. Other tunables: `--jpeg-quality`, `--webp-quality`, `--pngquant-speed`, `--svg-precision`, `--verbose`. Requires the `pngquant` and `brotli` command-line tools on `PATH`.
+
+Code: branch **`fix/ADFA-5552`** — `scripts/optimize_db_media.py` (the optimizer) and `scripts/content_chunking.py` (the chunking protocol, shared with PR #24).
+
+## A note on image width (worth a group decision)
+
+A 500px max width was used uniformly, matching the Kotlin website's setting. For Android technical diagrams and screenshots this is fairly aggressive — most were originally wider. If preserving more detail in specific doc sets matters, the process can be re-run from the backup with a larger `--max-width` (globally or per doc set) at the cost of some of the savings above.