- Published on
The humble tokenizer decides what a model can ever learn. Why BPE, WordPiece and Unigram differ, and how to build, test and monitor one properly.
Benjamin Lee — Solutions Architect @ AWS
I'm Benjamin Lee, a Solutions Architect at AWS and data scientist. This is my personal corner of the web — where I share what I'm writing, building, and figuring out.
The humble tokenizer decides what a model can ever learn. Why BPE, WordPiece and Unigram differ, and how to build, test and monitor one properly.
A batch pipeline on AWS that puts a multimodal model at the centre, reading Korean and English marketing content for the offensive meaning that hides between the words.
Eight retrieval architectures, from Simple RAG to agentic swarms. The trick is not choosing the cleverest one, but the one that fixes your actual failure.
Tweaking a chatbot prompt is easy; knowing whether you improved it is not. How to A/B test LLM prompts in production—the mechanics, the metrics, the tooling.
Autonomous AI agents no longer just answer; they plan, act and fail in inventive ways. How to build them on AWS, from supervisors to event-driven swarms.