Tokenization: How LLMs Split Text

Before an LLM reads your text, it splits it into tokens. Try a live tokenizer, see why math is hard for AI, and watch BPE build a vocabulary.

This page is interactive and needs JavaScript. More reading: