Lossless on any string (256-byte base vocab), regex pre-tokenization so merges never
cross word boundaries, incremental pair-count training with deterministic tie-breaks,
special tokens, and JSON persistence. The trainer gains --tokenizer bpe and reports
bits/byte so tokenizers are comparable; old character checkpoints still load.
Tests check the incremental trainer against a naive reference, round-trips on random
unicode, and merge-priority semantics; mutation checks confirmed they fail on broken code.
Draft proposes gamma tokens, target verifies them in one pass using rejection sampling
with residual resampling, and both KV caches roll back to the accepted prefix. Tests
check greedy equivalence and compare sampled output against the brute-force target
distribution; mutation checks confirm the test fails on deliberately broken variants.
jzzliu