Status: Phase 4 adds detached prompt capture and pinned native-space geometry measurement to the existing CPU training/replay model. Numerical model and training semantics are unchanged, with geometry loss zero. The 6,752-parameter tiny configuration has visible-fixture learning and descriptive geometry evidence; fresh capability evidence remains pending. See PHASE-3.md and PHASE-4.md.
Build a decoder-only causal Transformer from random initialization. The first task is controlled next-token learning on small, generated formal reasoning sequences. General conversation capability is outside the first experiment.
| Setting | First reference configuration |
|---|---|
| Tokenizer | UTF-8 bytes 0–255; BOS=256, EOS=257, PAD=258 |
| Vocabulary | 259 tokens; no tokenizer training corpus |
| Context | 256 tokens, including special tokens |
| Blocks / width | 6 blocks / 128 hidden dimensions |
| Attention | 4 heads; separate Q, K, V, and output projections; causal mask |
| Feed-forward | 128 → 512 → 128, GELU |
| Position | Learned 256 × 128 embedding |
| Normalization | Two pre-RMSNorm weights per block, plus final RMSNorm; epsilon 1e-5 |
| Linear bias / dropout | None / zero in the reference configuration |
| Output head | Tied to token embedding; floating point |
| Ternary scope | Attention and feed-forward linear matrices only |
| Reference numeric lane | CPU float32 activations, latent weights, scales, embeddings, normalization |
The configuration has 1,247,232 unique trainable parameters, counting tied embeddings once: 1,179,648 linear weights, 33,152 token-embedding weights, 32,768 positional weights, and 1,664 normalization weights. Phase 2 checks these counts against the constructed tensors and lists the tied embedding/output alias. It is a roughly 1.25M proof-of-concept, within the initial 1–5M research tier. The original 0.5B idea remains a later scaling option contingent on results and resources.
RMSNorm computes x / sqrt(mean(x²) + epsilon), multiplied elementwise by its learned gain. Each block applies pre-norm attention and pre-norm feed-forward branches with residual additions. Attention scales dot products by sqrt(head dimension), masks future positions, and applies softmax. The implemented order is value × rsqrt(mean(value²) + epsilon) × gain, attention QKᵀ / sqrt(head width), masked softmax, attention residual, exact GELU feed-forward residual, and final normalization/tied output. No fused attention or alternate GELU path is selected. Nonempty right-padded masks avoid all-masked softmax rows; masked states and logits are zeroed. Positions are absolute prefix positions; overlength and excessive attention-cell allocations fail. See PHASE-2.md for bounds and fixtures.
The byte tokenizer fixes source offsets without a learned vocabulary. Invalid UTF-8 generated byte sequences remain inspectable as bytes; human display must record its replacement/escaping policy. Padding is masked in both attention and loss. Overlength examples are rejected in the first lane rather than silently truncated or rechunked. BOS/EOS insertion and prompt/answer loss spans are explicit dataset transformations.
Maintain two otherwise matched models: a float32 dense baseline and a ternary-forward model. Their architecture, tokenizer, initial latent tensors, examples, batches, seeds, optimizer settings, and token budget are matched. TRAINING defines the quantizer and surrogate gradient. Higher-precision training state and non-ternary tensors remain visible.
The first ternary path materializes scaled floating-point weights in PyTorch. This validates semantics, not packed CPU acceleration. INFERENCE defines a later packed representation and equivalence gate.
An opt-in classical four-state lane is implemented in the comparison detour, preserving dense/ternary numerical paths. Inspection identifies its actual quantizer and {-3,-1,+1,+3} occupancy. ModelConfig's historical quantizer field remains the ternary identifier; the explicit lane selects the additional frozen codebook. Four-state has the same matrix scope/floating-point exceptions and establishes no packed savings.
Data admission and generation produce hash-bound examples and split manifests. The tokenizer produces IDs and source spans. Model/training consume these immutable artifacts. Inspection emits selected tensors and activations. Geometry analyzes capture artifacts; evaluation scores outputs. Artifact exports supply CPU inference. The observer records identities and evidence at each boundary.
Geometric losses default to disabled. The first geometry work measures controlled trajectories. A training intervention needs a separate frozen protocol and ablations. QSOL-GEO-REASON is the measurement reference; its current state does not establish that geometry training improves this model.
PROVENANCE surrounds runs as an observer. It is not an extra neural layer, optimizer, reward function, or source of training labels. Mandatory evidence failures make a run ineligible for a release claim while preserving the computation and failure record.
Implemented inspection lists all named modules, tensor shapes, dtypes, parameter sharing, trainability, quantizer scales/codes, and forward-operation settings. Implemented captures select layer and absolute token positions explicitly, retaining per-token float32 states without pooling. Context/pooling analysis is future geometry work. Attention dumps are not implemented. Selecting every hidden layer is explicit and subject to the aggregate capture limit. No default unbounded retention of the autograd graph.
The 8B parameter policy caps future native architectures at 8,000,000,000 unique total parameters, including frozen learned parameters and all experts, with genuine tied parameters counted once. It is a ceiling, not a scaling target or CPU feasibility claim. The current 5M constructor limit and smaller per-run budgets remain unchanged. The isolated prospective adapter requires explicit adoption; historical workflows are not silently retrofitted. Own-teacher distillation and quantization remain separately gated future work.