Add syntax highlighting with bundled Prism.js and GitHub theme - #6
Merged
Merged
Conversation
…e + language normalization The shipped prism-bundle.js threw during evaluation (sparql extends turtle, which was never bundled), so tokenizeCode — defined after the throw — never existed in the JSContext and every language silently fell back to plain text. Highlighting has been completely dead. - Recreate the missing Scripts/bundle-prism.sh (+ bundle-prism.js, prism-wrapper.js): pins prismjs 1.29.0, resolves component order from Prism's own components.json (no more hand-ordered dependency bugs), and refuses to write a bundle that fails a load/tokenize smoke test. - Regenerate the bundle: 69 components incl. new basic, pascal, vbnet, makefile, ini, r, batch, cmake, julia, matlab, protobuf, hcl, nasm, armasm (147 KB). Every language in the CodeBlocks.md demo now tokenizes. - Normalize fence info strings before grammar lookup (first word, lowercased, alias table: c++→cpp, golang→go, vb.net→vbnet, …) in new PrismLanguageNormalizer; previously the raw string was matched case-sensitively against grammar names. - Emit Prism token aliases from the JS wrapper and use them as a theme color fallback, so grammar-specific types (INI keys, diff signs, asm registers) pick up standard colors. - First tests for the subsystem: normalizer (all platforms) and JSC-gated tokenizer round-trip/coverage tests (Apple platforms). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
Fenced code re-renders on every streamed chunk, so an unbounded block re-tokenized its whole accumulated text through JavaScriptCore each time — O(n²) over a stream. Now: - CodeHighlightingPolicy: blocks over 16 KB render plain while the fence is open and get one full highlight pass on close (the existing blockEnded → refreshBlock path); blocks over 512 KB never highlight. - PrismTokenizer gains a 16-entry LRU keyed by (language, code), so verbatim re-renders of closed blocks (width changes, out-of-band refreshes) skip JavaScriptCore entirely. - Truth-table tests for the policy (all platforms) and end-to-end highlighter tests (Apple platforms). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
- CodeBlockTheme.gitHub(): Primer syntax colors for light and dark, covering every predefined PrismTokenType (except intentionally plain-foreground ones like punctuation), enforced by a coverage test. - New token types surfaced by the expanded grammar set: rule (CSS), section (INI headers), parameter (bash/PHP). - MarkdownRenderTheme.default() now uses .gitHub(); .prismDefault() remains available for the previous look. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
…lighter example - README: highlighting is on by default (bundled Prism via JavaScriptCore, GitHub palette, ~65 languages, alias normalization, streaming size guard), CodeBlockTheme presets, and the bundle regeneration workflow. - The Splash example didn't compile: MarkdownRenderTheme properties are let and the protocol method is async. Add MarkdownRenderTheme.withCodeHighlighting(codeBlockTheme:codeHighlighter:) so swapping palettes or engines is a one-liner, and fix the example. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a robust, bundled Prism.js syntax highlighting integration for code blocks. It includes scripts to automatically resolve dependencies and bundle Prism grammars, a new GitHub-flavored theme, a normalization utility for fence info strings, a caching mechanism for tokenized blocks, and a highlighting policy to prevent performance degradation on large streaming blocks. The review feedback suggests a minor performance optimization in PrismLanguageNormalizer.swift to limit the string splitting operation to only the first word, avoiding unnecessary allocations.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Review suggestion on #6: maxSplits: 1 avoids scanning/allocating past the first whitespace-delimited word in normalize(), which runs on every streamed chunk of a code block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
Both branches fixed the O(n²) streaming re-highlight in MarkdownAttributeBuilder's fencedCode path: main by highlighting only when the fence closes, this branch by highlighting live with a 16 KB size guard (CodeHighlightingPolicy) that defers oversized blocks to a single pass on close. Resolved by keeping the live+guard behavior — small chat-sized blocks stay colored while streaming with bounded per-chunk cost (and an LRU cache), and oversized blocks get main's render-plain-then-highlight-on-close treatment via the same blockEnded → refreshBlock path. Adopted main's structure of resolving theme fonts/colors before the highlight branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds production-ready syntax highlighting to code blocks using a curated Prism.js bundle running in JavaScriptCore. Includes a GitHub-flavored color palette that adapts to light/dark mode, normalization of fence info strings (e.g.,
C++→cpp,golang→go), and intelligent caching to handle large streaming blocks efficiently.Key Changes
Prism.js bundling infrastructure
Scripts/bundle-prism.shandScripts/bundle-prism.js: Automated build system that fetches Prism v1.29.0, resolves language dependencies topologically, concatenates components, and smoke-tests the bundle with sample code in all 65 supported languagesScripts/prism-wrapper.js: JavaScript bridge that flattens Prism's nested token tree into simple{content, type, alias}objects and provides thetokenizeCode()entry pointLanguage normalization
PrismLanguageNormalizer.swift: Maps fence info strings to bundled grammar names, handling casing, aliases (c++→cpp, objective-c→objc, golang→go, vb.net→vbnet, etc.), and explicit plain-text markersPrismLanguageNormalizerTests.swiftTokenizer improvements
PrismTokenizer: Added 16-entry LRU cache to avoid re-tokenizing identical (language, code) pairs during re-renders and width changesPrismToken: Now carries optionalaliasfield for grammar-specific type standardization (e.g., INIkey→attr-name)GitHub theme
CodeBlockTheme.gitHub(): New preset using Primer syntax colors (keyword red, string blue, function purple, type orange, tag green, etc.) with separate light/dark palettesCodeBlockThemingTests.swiftHighlighting policy
CodeHighlightingPolicy.swift: Prevents O(n²) re-tokenization of large streaming blocks by deferring highlighting until the fence closes (threshold: 16 KB, hard limit: 512 KB)CodeHighlightingPolicyTests.swiftIntegration
MarkdownRenderThemeto default to.gitHub()theme and addedwithCodeHighlighting()convenience methodMarkdownAttributeBuilderto use language normalization before tokenizationPrismCodeHighlighterTests.swiftandPrismTokenizerTests.swiftToken type additions
PrismTokenTypewith.rule,.section,.parameterand addedallPredefinedcollection for test assertionsDocumentation
Implementation Details
sparqlpulls inturtle)https://claude.ai/code/session_01NRUDZwowe1T836zcYLwQoo