fix(sidebar): read a linked SQL file's header in the encoding the whole file uses - #3083
Merged
Merged
Conversation
datlechin
force-pushed
the
fix/linked-header-utf8-boundary
branch
from
September 23, 2026 15:48
1bdf5d0 to
d7ed434
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The sidebar builds each linked SQL file's row from its first 4 KB. When a UTF-8 file had a multibyte character straddling byte 4096, TablePro labelled it ISO-8859-1 and garbled its frontmatter:
-- @name: Báo cáo doanh thushowed asBáo cáo doanh thu. Edit Metadata then seeded that garbled name and wrote it back into the file. BOM'd UTF-16 and UTF-32 files got the same label, and a big-endian UTF-32 file was garbled even when opened. Found while investigating #2505.Root cause
FileTextLoader.loadHeaderread exactly 4096 bytes and calledString(data:encoding: .utf8), falling back to.isoLatin1. Foundation has no notion of a partial character, so a character cut at the limit fails the whole prefix.FileTextLoader.loadrelied onString(contentsOf:usedEncoding:), which rejects a big-endian UTF-32 BOM (measured) and falls back to Latin-1. So the header and the full load disagreed, and the index kept the header's answer.Fix
ByteOrderMarkis the one place that recognises UTF-16 and UTF-32 marks.SQLChunkDecodernow uses it instead of its own copy of the tables; its own trimming at chunk boundaries is unchanged.TextPrefixDecoderdecodes a byte prefix: a mark's encoding first, then UTF-8, then ISO Latin-1. When the file is longer than the limit, it reads up to 3 bytes past it and completes a character the limit cut in half. Completing is stricter than trimming: a Latin-10xE9followed by ASCII still reads as Latin-1.FileTextLoader.loaddecodes a file with a mark through the same decoder, so the header and the full load agree for every mark, UTF-32 big-endian included. A mark now wins over thecom.apple.TextEncodingattribute, as it already did for the header. Files without a mark load exactly as before.Measured against the old
load()over 12,240 handmade and fuzzed files: 259 UTF-32 BE files now load as UTF-32 instead of Latin-1 (the fix); about 600 files startingFF FE 00 00whose length is not a multiple of 4 now load as Latin-1 keeping every byte, where Foundation used to drop the trailing bytes silently; files whose mark and attribute disagree follow the mark. No other file changed.SQLChunkDecodergave identical output to the old code over 2,588,124 chunked decodes.Tests
TextPrefixDecoderTests: 2-, 3- and 4-byte characters cut at every inner position, genuine Latin-1, a Latin-1 lookalike at the limit, UTF-16 and UTF-32 marks, a surrogate pair across the limit, cut code units, short and empty input.FileTextLoaderTests: real files at the 4096 limit, including the reported Vietnamese name; big-endian UTF-32; a cut marked file keeping every byte; a mark outranking the encoding attribute; and the header agreeing with the full load for every fixture.ByteOrderMarkTests, plus newSQLChunkDecoderTestscases that pass on both the old and new code and guard the refactor. Also run:SQLFileParserTests,SQLFileParserPLSQLTests,FileTabBaselineTests.Risks