Watch on YouTubeWatch on Instagram
About this video
Why AI can't spell "strawberry", yet knows a cherry sits next to a berry.
Episode 2 of Learn AI in Under 3 Minutes. Token splits and IDs are from o200k_base, the tokenizer OpenAI's GPT-5 uses (the older GPT-4 cl100k_base count is shown for Hindi); the maps are from the open embedding model BAAI/bge-small-en-v1.5, squashed with PCA; the token counts for English, Chinese, Hindi and Amharic were measured on one sentence (translations vary, so treat them as one example).
0:00 The strawberry test
0:22 Tokens
0:39 What tokens cost you
1:04 From ID to meaning
1:20 The map of meaning
2:15 Use it
Sources:
- Tiktokenizer, the free tokenizer shown in the video: https://tiktokenizer.vercel.app/
- How many Rs are in the word strawberry? (background on the strawberry problem): https://blog.legoktm.com/2024/08/10/how-many-rs-are-in-the-word-strawberry.html
- King - Man + Woman = King? (why the famous analogy needs the inputs excluded): https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85
- Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., NeurIPS 2023): https://arxiv.org/abs/2305.15425
- OpenAI gpt-oss-120b config: 2,880 numbers per token (hidden_size): https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json
- BAAI/bge-small-en-v1.5, the open embedding model used for the maps (384 numbers per text): https://huggingface.co/BAAI/bge-small-en-v1.5
- tiktoken's model table: gpt-5 uses o200k_base: https://github.com/openai/tiktoken/blob/main/tiktoken/model.py
- Measured for this video: o200k_base, the tokenizer GPT-5 uses, and the older cl100k_base (GPT-4), via tiktoken; BAAI/bge-small-en-v1.5 embeddings via fastembed: https://github.com/openai/tiktoken
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
Ask a chatbot how many R's are in strawberry, and for years, many said, totally confident: two. By the end, you'll know exactly why. First, guess: how many tokens does it see in strawberry? One, three, or ten? Lock it in. Three. In OpenAI's tokenizer it's S-T, raw, berry. The model never sees the letters at all. Last episode, the model predicted the next piece. Today: what a piece actually is, and how it turns into meaning. A tokenizer chops your text into tokens. Common words stay whole: the, cat, sat, on, the, mat. Six tokens. Tokens are the meter on everything: what you pay, how fast answers stream, and how much a model can read at once. And it isn't equal. We measured one sentence with OpenAI's tokenizer: eight tokens in English, eight in Chinese, seventeen in Hindi. And seventy-one in Amharic, from Ethiopia: nearly nine times the bill. If you build for other languages, check your token count. Back to strawberry: S-T, raw, berry. Each piece is just an ID number. Berry is nineteen thousand seven hundred seventy-two. An ID means nothing alone, so the model looks it up in a giant table. That row of numbers is an embedding. Now take a real embedding model, squash its numbers down to two, and plot them. Berry lands here. Cherry, grape and apple land nearby. Car, truck and bus, far away. Close on the map means close in meaning. Your turn. Which word sits closest to berry: bury, Barry, or cherry? Lock it in. Cherry, clearly. Bury sounds the same and Barry looks the same, but the map measures meaning, not spelling. And real maps aren't flat. This model uses 384 numbers per word; OpenAI's open model uses 2,880 per token. Here are just three. Even directions carry meaning. Man to woman points roughly the same way as king to queen, even in this small real model. But do the famous math in this model, and king minus man plus woman is still closest to king. Queen comes second. That's how search by meaning works: embed your documents, embed the question, grab the closest. It's the heart of RAG, episode eight. Use it this week: paste your next prompt into a free tokenizer like Tiktokenizer and watch the pieces. Fewer tokens, cheaper and faster. So that's why it can't spell, yet finds a cherry next to a berry. Next: attention, how 'it' finds the cat, not the mat.