About this video
In 11 minutes you'll understand the 5 ideas inside every modern AI model.
Which idea clicked for you: attention, query-key-value, or the eight heads?
We read "Attention Is All You Need" (Vaswani et al., 2017) page by page: the problem with recurrent networks, self-attention, queries, keys and values, multi-head attention, positional encoding, the full Transformer, its results, what happened next, and what it means if you build with AI in 2026.
Chapters
0:00 The paper behind every chatbot
0:29 The problem in 2017
1:29 Idea 1: Every word looks at every word
2:27 Idea 2: Query, Key, Value
3:48 Idea 3: Multi-head attention
4:32 Idea 4: Word order
5:26 Idea 5: The Transformer
6:34 Why it is faster
7:02 The results
8:10 What happened next
9:11 What it means for builders in 2026
10:25 Recap
Paper: Attention Is All You Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. NIPS 2017. https://arxiv.org/abs/1706.03762
Figures, tables and equations shown from the paper with attribution, under the permission printed on its first page (reproduction for journalistic or scholarly works).
Citation count (more than 250,000, 2026): https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Pranav Innovates: AI research, explained. Subscribe to stay up to date.
What the video says
Every AI chatbot you have ever used, ChatGPT, Claude, Gemini, runs on one idea from a fifteen-page paper. Eight researchers at Google wrote it in 2017, gave it a title borrowed from a Beatles song, and changed the world. Today we are reading the actual paper, page by page. What it says, how it works, and why it still matters in 2026. By the end, you will understand the five ideas inside every modern AI model. Let's open it. In 2017, the best translation systems read a sentence the way you would read through a keyhole. One word at a time. They were called recurrent neural networks. Each word updates a little memory, then passes it to the next word, like a note handed down a line. That causes two problems. First, the note gets smudged. By the end of a long sentence, the beginning is half forgotten. Second, it is slow. Word five has to wait for word four. Your GPU has thousands of cores, and most of them just sit there, waiting. The paper says it plainly. This inherently sequential nature precludes parallelization. Researchers already had a trick called attention, which let a model glance back at any earlier word. But it was always bolted onto those slow recurrent networks. So this team asked a wild question. What if we throw the recurrence away entirely, and keep only the attention? That is the title. Attention is all you need. Idea one. Every word looks at every other word, all at the same time. Take this sentence. The animal didn't cross the street because it was too tired. Quick, what does it mean here? You knew instantly. The animal. But a computer has to work that out. With self-attention, the word it looks across the whole sentence and scores every word. How relevant are you to me? Animal scores high. Street, not so much. So it pulls its meaning mostly from animal. Now change one word. Because it was too wide. Suddenly, street wins. Same sentence, one word different, and the attention flips. That is context, computed. And this is not just a cartoon. In the appendix, the authors show a real trained attention head doing exactly this, linking the word its back to Law and application. And because every word does this at once, in parallel, the GPU finally gets to use all of its cores. Idea two. How does a word decide what is relevant? The paper gives every word three roles, borrowed from search. Query, key and value. Think of searching on YouTube. Your query is what you type. Every video has a key, its title and tags. And the value is the video itself. Inside the model, every word makes all three. Its query asks, what am I looking for? Its key says, here is what I offer. Its value is what it actually passes along. To score a match, the model multiplies a query with each key. That is a dot product. Vectors pointing the same way get a big score. Then softmax turns those raw scores into percentages that add up to one hundred. Finally, it takes a weighted blend of everyone's values. Mostly animal, a little street, a pinch of the rest. That is equation one in the paper. Softmax of Q times K transposed, scaled down, times V. But look at that little square root of d k. Why is it there? Take a guess. The answer is in a footnote. Each vector has sixty-four numbers, so dot products grow huge. Softmax goes all or nothing, one word grabs everything, and learning stalls. Divide by the square root of sixty-four, which is eight, and the scores calm down. One tiny division, and the whole thing trains. Idea three. One look is not enough, so the Transformer looks eight times, in parallel. Each of the eight heads gets its own small query, key and value, sixty-four numbers instead of five hundred and twelve. So eight heads cost about the same as one big one, but now each head can specialise. One might track who it refers to. Another links a verb to its object. Another just watches the next word. You can see it in the paper's appendix. Several heads, each in its own colour, connect the word making to more difficult, many words away. The authors tested it. With a single head, the translation score dropped almost a full point. With too many heads, it dropped again. The sweet spot was in the middle. Idea four fixes a problem that attention creates. Attention has no idea about word order. To pure attention, dog bites man and man bites dog are the same bag of words. Not great. So before anything else, the model adds a position signal to every word, built from sine and cosine waves. Each dimension gets a wave of a different speed. Fast waves flip from one word to the next. Slow waves barely move. Like the second, minute and hour hands of a clock. Read all the hands together, and every position gets its own unique timestamp. The authors also tried simply learning the positions. Nearly identical results. They kept the waves, hoping the model could handle sentences longer than any it saw in training. Modern models swapped these waves for a newer trick called rotary embeddings, but the idea is the same. Stamp the order onto the words. Idea five puts it all together. This is figure one, maybe the most famous diagram in AI. On the left, the encoder reads the input sentence. On the right, the decoder writes the translation, one word at a time. Words become vectors of five hundred and twelve numbers, plus their position waves. Then comes a block. Multi-head attention, so words can talk to each other. Then a small feed-forward network, so each word can think on its own. Around each step there is a shortcut, called a residual connection, plus a normalisation layer, so signals flow cleanly through a deep stack. Stack that block six times. That is the encoder. The decoder has the same parts, plus two twists. First, a mask. While writing word three, it is not allowed to peek at word four. The future is hidden. Second, cross-attention. The decoder's queries look at the encoder's output, asking the input sentence, which of your words matter for what I am writing right now? At the top, the model predicts the next word, adds it to what it has written, and goes again, until the German translation is complete. Table one in the paper is the real argument. A recurrent network needs n steps in a row for n words. Self-attention needs one. Every word, at once. And the path between any two words, however far apart, is a single hop instead of a long chain. There is a catch, so keep it in mind for later. Every word compares itself with every word, so the work grows with the square of the length. Double the text, and attention costs four times as much. So, did it work? The team trained on four and a half million English and German sentence pairs, on one machine with eight GPUs. The base model trained in twelve hours. The big one, three and a half days. Here is table two. On English to German, the big Transformer scored twenty-eight point four, more than two points above the best result ever reported, even teams of models working together. And the small base model alone beat every earlier system, ensembles included, using about fifty times less training compute than Google's own best translation ensemble. On English to French, a new single-model record, forty-one point eight, at a fraction of the cost. A few training details everyone copied afterwards. A learning rate that warms up for four thousand steps, then slowly decays. Dropout. And label smoothing, which teaches the model to be a little less sure of itself. It made one score worse and the translations better. Then they tried a task it was never built for, mapping the grammar of English sentences. With almost no tuning, it beat nearly every specialised system. The last page says, we plan to extend the Transformer to images, audio and video. That turned out to be an understatement. In 2018, OpenAI built GPT from the decoder half, and Google built BERT from the encoder half. In 2020 came GPT-3, and vision transformers that read an image as patches, the way they read words. Transformers now transcribe speech in Whisper, generate video in Sora, and attention helps AlphaFold predict the shapes of proteins. And in 2022, ChatGPT. The world met the T in GPT. Generative Pre-trained Transformer. Today the paper has been cited more than two hundred and fifty thousand times, one of the most cited papers of this century. And every one of the eight authors has since left Google. Between them, they started companies like Cohere, Character AI, Sakana AI and Essential AI. One detail I love. The first page footnote says the author order was random. Equal contribution. Eight names, one idea. If you build with AI, here is what this paper still means for you in 2026. A modern model is still this block, with upgrades. Most chatbots keep only the decoder. Normalisation moved to the start of each step. Rotary embeddings replaced the waves. And the feed-forward layer often became a mixture of experts. But the n squared cost never left. It is why long prompts are slower and cost more, and why so much research goes into faster attention. FlashAttention, sharing keys and values across heads, and hybrids with state space models. The decoder loop also explains your API bill. Your prompt is read all at once, in parallel. The answer is written one token at a time, and each new token reads the stored keys and values of everything before it. That is the KV cache. That is why output tokens usually cost more than input tokens, and why prompt caching saves money. It reuses keys and values that were already computed. My take. When I build an agent, I treat the context window like this attention matrix. Everything I put in competes for attention, and the cost grows fast. So I keep it focused, put what matters where it is easy to find, and cache what does not change. So that is the paper. Throw out recurrence. Let every word look at every other word. Ask with queries, match with keys, pass along values. Look eight ways at once. Stamp on the order. Stack it six times. Fifteen pages, and the foundation of modern AI. Next in this series, we jump to the newest research, papers from this month that build on this one. Subscribe so you stay up to date. Because it turns out, attention really was all we needed.