Nimbus Systems https://nimbusarchitect.com/blog Solutions Architect at AWS. Writing about AI/ML, cloud architecture, cybersecurity, and quantitative research. en-us lilphd99@gmail.com (Benjamin Lee) lilphd99@gmail.com (Benjamin Lee) Sun, 21 Jun 2026 00:00:00 GMT https://nimbusarchitect.com/blog/tokenization-the-encoding-layer-that-decides-how-models-learn A Token Effort https://nimbusarchitect.com/blog/tokenization-the-encoding-layer-that-decides-how-models-learn The humble tokenizer decides what a model can ever learn. Why BPE, WordPiece and Unigram differ, and how to build, test and monitor one properly. Sun, 21 Jun 2026 00:00:00 GMT lilphd99@gmail.com (Benjamin Lee) tokenizationllmaiml-systemsnlppretraining