A Token Effort

Before a language model reads a word, something has already decided what a word is. That something deserves more respect than it gets.

Ask an engineer to explain a large language model and the story almost always begins at the transformer. The input, in this telling, simply arrives. It does not. Every byte of training data must first pass through a single, unglamorous component that chops raw text into the discrete symbols the model actually sees—and the choices baked into that component are frozen long before the first gradient step. A model can only learn distinctions its tokenizer preserves. It inherits every bias the tokenizer smuggles in. That makes the encoding layer not a preprocessing footnote but a piece of training infrastructure, with the same operational weight as the data pipeline or the optimiser.

This post takes that infrastructure view. It walks through the algorithms in use today, the measurable ways an encoding choice changes what a model learns, the failure modes that lurk as latent bugs, and what it takes to build, test and monitor a tokenizer that must serve both pretraining and finetuning without drifting between them.

Three ways to slice a word

Nearly every production tokenizer runs one of three subword algorithms, and they differ chiefly in how they decide where to cut.

Byte pair encoding began life as a compression scheme. Adapted to text, it merges the most frequent adjacent symbol pair, over and over, until the vocabulary hits a target size (Sennrich et al., 2016). The merge list is the model: at inference you replay the same ordered merges on new text. WordPiece, the engine behind BERT and its descendants, takes a similar bottom-up route but picks the merge that most raises the likelihood of the training corpus under a unigram language model, rather than the one that is merely most frequent (Schuster and Nakajima, 2012). The Unigram model inverts the whole procedure. It starts from a bloated candidate vocabulary and prunes the tokens that contribute least to corpus likelihood, which lets it represent a single string several ways and weigh the probability of each segmentation (Kudo, 2018).

Beneath all three sits a quieter question: what counts as a base symbol. Early implementations worked on Unicode characters, which forces an awkward decision about the long tail of rare code points. GPT-2 switched to byte-level BPE, whose base alphabet is the 256 possible bytes, so any string in any language is representable with no out-of-vocabulary token whatsoever (Radford et al., 2019). SentencePiece, a language-agnostic library, then packaged these ideas by treating the input as a raw stream, whitespace and all, so the same tokenizer works without language-specific pre-segmentation (Kudo and Richardson, 2018). For an engineer the point is blunt: the algorithm and the base alphabet together fix the vocabulary's coverage, its robustness to strange input, and the compute it costs to run at scale.

Compression is not the only prize

The intuitive way to compare two tokenizers is fertility—the average number of tokens per word, or per byte. Lower fertility means each token carries more text. A fixed context window then holds more content, training throughput rises because there are fewer tokens to grind through for the same corpus, and inference gets cheaper. The temptation is to treat fertility as the one number to minimise.

Resist it. There are two reasons. First, compression and downstream quality are correlated but not identical, and tuning a tokenizer purely for bytes-per-token can leave performance on the table; the segmentation that compresses best is not always the one a model learns best from (Schmidt et al., 2024). Second, aggressive compression turns nasty in domains where the internal structure of a token matters. Arithmetic is the clearest case. A tokenizer that crams several digits into one token forces the model to memorise the behaviour of each multi-digit chunk instead of composing from single digits—and the alignment of digit groups between the operands and the answer turns out to drive accuracy. Work on frontier models found that enforcing right-to-left grouping of digits, so that place value lines up, measurably improved addition over the default left-to-right segmentation (Singh and Strouse, 2024). The moral generalises. The right tokenizer for a workload depends on what the model must compose, not merely on how few tokens it emits.

The multilingual tax

Tokenise the same passage in English and then in a morphologically rich or non-Latin-script language, and the second version is usually far longer in tokens. The gap is large and systematic. An analysis across seventeen tokenizers found that parallel text could differ in tokenised length by up to roughly fifteen times between languages, and the disparity persisted even for tokenizers built expressly for multilingual use (Petrov et al., 2023). This is sometimes called the token premium, and recent work frames it as a structural handicap: more tokens mean less effective context, slower training throughput and higher per-request cost—for precisely the language communities already underserved (Explaining and Mitigating Crosslingual Tokenizer Inequities, 2025).

The infrastructure implication is direct. A tokenizer trained on a corpus that is ninety-plus percent English will spend its merges and its vocabulary budget on English, and every other language pays the premium downstream. If a model is meant to serve many languages, the tokenizer's training mixture and vocabulary allocation are first-class design parameters—to be set deliberately and validated per language, not inherited from whatever corpus happened to be lying around.

Glitch tokens: bugs shipped in the vocabulary

The most operationally interesting failure mode springs from a simple mismatch: the tokenizer is trained on one corpus and the model on another. Any token that lives in the vocabulary but is rare or absent in the model's training data never receives a meaningful gradient signal. Its embedding loiters near its random initialisation. When such a token then shows up at inference, the model misbehaves—it may hallucinate, ignore the token, or emit unrelated text. The canonical specimen is the SolidGoldMagikarp token, a Reddit username that wormed its way into the vocabulary and produced bizarre completions when prompted.

What makes this an infrastructure concern rather than a party trick is that such tokens can be found systematically. A method for detecting under-trained tokens inspects the embedding matrix and the tokenizer configuration to flag candidates whose representations look untrained, then verifies them with targeted prompts; the authors report that such tokens are common across many widely used models (Land and Bartolo, 2024). For a team building tokenization systems this becomes a concrete pre-release check: scan the trained model for vocabulary entries that received negligible signal, because each is a latent bug and, in some cases, an exploitable input. The deeper lesson is that the tokenizer and the model's training corpus cannot be designed in isolation. The gap between them is exactly where these defects breed.

The tokenizer is a system, not a script

Once you accept that encoding choices are welded into the model, the engineering requirements follow. A tokenizer that serves real training workflows has to be fast, deterministic, observable and consistent across the boundary between pretraining and finetuning. The shape below is the part most teams underinvest in.

        corpus shards (many languages, code, math, structured data)
                                |
                                v
        +-----------------------------------------------+
        |  Tokenizer training                            |
        |   - algorithm: BPE / Unigram / WordPiece       |
        |   - base alphabet: bytes vs. characters        |
        |   - vocab size + per-language allocation       |
        |   - normalization + pre-tokenization rules     |
        +-----------------------------------------------+
                                |
                                v
        artifact: vocab + merges + config  (versioned, hashed)
                                |
        +-----------------------+-----------------------+
        |                                               |
        v                                               v
   Pretraining encode                             Finetuning encode
   (high-throughput, sharded)                     (same artifact, same rules)
        |                                               |
        v                                               v
   token-stream + fertility, OOV,                 identical normalization,
   coverage metrics per shard                     special-token handling
        |                                               |
        +-----------------------+-----------------------+
                                |
                                v
        validation gate:  fertility by language, digit/whitespace
        behavior, under-trained-token scan, round-trip decode test

A few properties of this pipeline are worth spelling out, because they are where the work of an encodings and tokenization engineer actually lives.

Determinism and versioning. The tokenizer artefact—the vocabulary, the merge rules, the normalisation configuration—is a versioned dependency, exactly like a model checkpoint. A silent change to normalisation or special-token handling between pretraining and finetuning will desynchronise the two stages and degrade the model in ways that are miserable to trace, because nothing crashes. Hashing the artefact and pinning it across stages is the cheapest insurance on the market.

Throughput. Encoding a multi-trillion-token corpus is a real data-processing job, and naive subword application is slow. Production systems lean on optimised implementations—byte-level BPE with precompiled merge tables and parallelised application—precisely so that tokenization does not become the bottleneck that starves expensive accelerators. This performance work is unglamorous and directly load-bearing.

Observability. Fertility per language, out-of-vocabulary or byte-fallback rate, and the distribution of token frequencies are the metrics that catch trouble before it reaches a training run. A spike in byte-fallback on a new data source, or a long tail of tokens that never appear, is a signal to interrogate the tokenizer rather than the model.

Testing across languages and data types. A robust validation suite checks round-trip encode-decode fidelity, confirms that whitespace and digits behave as intended, exercises scripts and code and structured formats, and re-runs the under-trained-token scan after training. This is the testing-framework requirement made concrete, and it is what lets a team change a tokenizer with confidence rather than hope.

The frontier: can we sack the tokenizer entirely?

Given how many problems originate at the encoding layer, a fair question is whether to drop the discrete tokenizer altogether and let the model chew on raw bytes. Early byte- and character-level models paid a steep efficiency cost, because sequences ballooned. Recent work narrows that gap. The Byte Latent Transformer groups bytes into dynamically sized patches, spending more compute where the next byte is hard to predict and less where it is easy, and reports matching the performance of tokenizer-based models at scale while improving robustness to malformed or unusual input (Pagnoni et al., 2024). This is a genuinely different answer to the very problem the tokenizer solves: how to turn a byte stream into units of computation at the right granularity.

The practical reading is not that tokenization is obsolete. It is that the function the tokenizer performs—deciding the granularity at which a model perceives text—is fundamental, and the field is busily testing whether a learned, dynamic boundary beats a fixed, precomputed one. Either way, the engineering questions do not budge: throughput, robustness, multilingual fairness, and the alignment between how data is encoded during training and how it is encoded in use.

Practical guidance

A handful of decisions carry most of the risk.

Co-design the tokenizer and the training mixture. The under-trained-token problem and the multilingual premium both trace back to a tokenizer whose training distribution wandered from the model's. Treat them as one design, not two.
Choose the base alphabet for robustness. Byte-level fallback guarantees that no input is unrepresentable, which matters the moment real-world data throws up scripts, emoji or corruption the tokenizer never saw.
Match segmentation to the task structure. If arithmetic, code or other compositional structure matters, validate digit and symbol handling explicitly rather than trusting the default merges (Singh and Strouse, 2024).
Pin and hash the tokenizer artefact across stages. The pretraining and finetuning encoders must be byte-for-byte identical in behaviour, normalisation and special tokens included.
Scan for under-trained tokens before release. Make it a gate, not an afterthought (Land and Bartolo, 2024).
Measure fertility per language and per data type, and track it over time. A change here is an early warning that something upstream has shifted.

Conclusion

The encoding layer is easy to overlook because it runs once, quietly, before the part of the system everyone watches. That is exactly why its mistakes are so expensive: they are compiled into the model and surface later as multilingual inequity, brittle arithmetic, or a glitch token that derails a completion. Treating tokenization as infrastructure—versioned, tested, monitored and co-designed with the training data—is what turns it from a source of latent bugs into a foundation researchers can build on. Whether the field keeps a fixed subword vocabulary or drifts toward learned byte patches, the same discipline applies, because the question the encoder answers—how a model perceives the data it learns from—never goes away. The transformer gets the applause. The tokenizer decides what there is to applaud.