fix: improve dataset processing / loading speeds

[Dogacel]

stage	before	after
load + normalize (pass 1, single-threaded)	40s	42s	—
tokenize (64 workers)	1436s @ 696/s	319s @ 3131/s	4.5x
collect results into entries	498s	23s	21.7x
torch.save	207s	94s	result of lower memory pressure
total build	2181s (36m)	480s (8m)	4.5x
cache size	42.8 GB	42.8 GB	byte-identical
cache reload (every run)	~74s	~74s	unchanged

Measured on the full 1M-conversation corpus, 64 workers, Qwen3-8B tokenizer,
max_seq_length=16384.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f4f3495f5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Restore the data directory ignore pattern

With this pattern, root-level data/ is no longer ignored: I checked the updated file with git check-ignore -v data/foo and it returns no match, while /data/ or the previous data/ pattern does match. This makes local training datasets under data/ show up as untracked files and increases the chance of accidentally staging large/private data.

Useful? React with 👍 / 👎.