Watch on YouTubeWatch on Instagram
About this video
I asked a small AI about 24 things that don't exist. It answered all 24. Why models make things up, a free check that catches many of them, and how to test a model properly.
Episode 10 of Learn AI in Under 3 Minutes, level 2: using models well.
Measured on a 2020 MacBook Air with Qwen3.5-0.8B (open weights, Feb 2026, thinking off): 48 short questions written for this video, 24 real facts from around the world and 24 about books, people and places made up for it. Plain prompt: 13/24 real facts right; all 24 made-up questions answered (no "I don't know"). Adding "If you are not sure, say I don't know": 0 of 48. Asked for the made-up novel's year, its first token was a digit 99.8% of the time. Five sampled answers per question (temperature 0.7): on all 28 questions where fewer than 4 of 5 agreed, its answer was wrong; agreeing was no proof (Almaty, 5 of 5). Skipping those: still 13 right; scored +1 right, -1 wrong, 0 skipped, -22 always answering vs +6 skipping. Grading its own answers: with no key it passed 27 of its 35 wrong answers; with the key, 2, agreeing with my program on 46 of 48. One small model and 48 questions: an example, not a benchmark.
Sources: Kalai, Nachum, Vempala, Zhang, "Why Language Models Hallucinate" (OpenAI; arXiv:2509.04664, Sept 2025, CC BY 4.0): training and evaluation reward guessing over admitting uncertainty; fix the scoring of existing benchmarks. Farquhar et al., "Detecting hallucinations in large language models using semantic entropy" (Nature 630, 2024): disagreement across sampled answers flags confabulations, not consistent mistakes. Artificial Analysis, AA-Omniscience (Nov 2025, v1.1 Oct 2026): +1 correct, -1 incorrect, 0 abstain; leader Gemini 4 Carbon (High), 54.
0:00 It answered everything
0:18 Why it makes things up
0:39 Real facts
0:53 Ask it five times
1:21 Score it like an exam
1:55 Who grades the answers
2:19 My take
Music: Special Spotlight by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Special_Spotlight_(ISRC_USUAN1600067).mp3)
Follow @PranavInnovates for the tech story that matters, every day.
What the video says
I asked a small AI about twenty-four books, people and places that don't exist. It answered every single one. Then I added: if you're not sure, say I don't know. It said it zero times. Today: why models make things up, a free check that catches many of them, and how to test a model properly. Remember episode one: it predicts the next likely piece. Asked for a year, its first token was a digit ninety-nine point eight percent of the time. I don't know: almost zero. Training adds to it. OpenAI's 20 25 paper: facts seen only once are hard to learn, and tests reward guessing. A blank scores zero; a guess sometimes scores. On twenty-four real facts, it got thirteen right. The eleven it got wrong sounded just as sure. Africa's highest mountain: Everest. The Four Seasons: Bach. Kazakhstan's capital: Almaty, which it used to be. Here's the free check: ask the same question five times. Bhutan's capital? Thimphu, every time. The Salt Orchard? Two thousand five, ninety-four, twenty eleven, twenty eleven, ninety-six. Made-up answers wander. Across all forty-eight questions, whenever its five answers disagreed, it was wrong. Twenty-eight out of twenty-eight. But agreeing isn't proof. Almaty came back five times out of five. A steady mistake looks just like knowledge. So let it skip a question when its answers disagree. Graded on accuracy alone, does skipping score higher? Exactly the same: thirteen right either way. Accuracy can't see a bluff. Now plus one if right, minus one if wrong, zero for a skip. Always answering: minus twenty-two. Skipping when unsure: plus six. That's the fix the paper asks for. Artificial Analysis now scores knowledge this way, and even its leader, Gemini 4 Carbon, nets just fifty-four out of a hundred. Last: who grades? Teams often use a model as the judge. I asked mine to grade its own answers. Everest for Africa? It said yes. With no answer key, it passed twenty-seven of its wrong answers. Given the key, just two. A judge needs a key. That list of questions with known answers, plus a grader, is an eval. It's how labs compare models, and how you should test your prompts. My take: a demo only shows the questions you picked. I keep real questions with answers, and every change has to beat that list. Try it this week: write twenty questions from your own work, with answers. Ask each five times, and score plus one, minus one, zero. Next: small models and quantization, AI on your own laptop. Episode eleven.