The convergence is in the reasoning, not in the learning

In the first part I explained why I went quiet on AI for a few years, and what I got wrong at a Microsoft panel in 2019. I promised a follow-up on the two things I think are actually broken in the current generation, and on what building with these systems has taught me about how we think ourselves.

One remark before I start. Everything below is about language models, because that’s what most of us use and what I’ve spent the most time with. There’s serious work going on in AI systems that don’t run on language at all – models of physics, proteins, robot control – and some of what follows doesn’t apply to them. I’m not going to go there, at least not in this piece.

The first is efficiency. A human brain runs on roughly 20 watts. To be fair, the numbers on the AI side at the point of use are better than the headlines suggest. Google measured a median Gemini text prompt at 0.24 watt-hours, and Epoch AI estimates about 0.3 Wh for a typical GPT-4o query. Your brain burns about 0.33 Wh in a minute of thinking, so per answer we’re in the same ballpark. Long inputs and reasoning models cost a lot more; a small model on a phone costs a fraction. The real gap is in the training. A GPT-4-class training run is estimated at 20-25 megawatts for about three months, roughly 50 gigawatt-hours. A human brain uses around 14 megawatt-hours in an 80-year life. One training run is the lifetime energy of about 3,500 brains, spent on learning what one brain picks up in a couple of decades, with a lot of help.

The second is learning itself. A kid learns what a dog is from a handful of dogs, and can scribble one from memory, with nobody supervising. A model needs the internet plus armies of humans doing reinforcement learning to get there. And once trained, the deployed model doesn’t learn anything at all while it “thinks”. Its learning happened once, offline, at datacenter scale. Every conversation starts from the same place.

There’s a difference in how the two of us got our picture of the world, and it explains a lot of what models are good and bad at.

We learn through the senses first. A child spends a couple of years looking, touching, tasting, dropping things, and listening before the first proper sentence. Patterns come before words; language is built on top of a world we already know by hand. Language models did it the other way round. They got the world compressed into text, and images, sound and video were bolted on later.

You’ve probably felt the consequence. Ask an image or video model for “something like a rainy street in Tokyo” and it gives you something great with almost no instructions. Ask it for the exact thing you have in your head – this angle, that gesture, the light from the left – and it becomes hard work. It has the words for everything, and a much thinner sense of the thing itself.

The gap is closing faster than I expected, and in a way I find telling. The current vision models don’t process an image the way our eyes do – they chop it into small patches and treat each patch as a token, the same as a word. So inside the model, the divide between seeing and reading has more or less disappeared. Everything is a sequence of tokens.

What hasn’t closed is how many examples it takes. Show a person three examples of a new pattern – a new letter in an alphabet they don’t know, say – and they’ll recognise it and even draw it. That’s been measured: people learn a new handwritten character from a single example about as well as the best models of the time did from thousands. The models have improved since, but the ratio is still not close.

The eyes plus the brain are, frankly, an absurdly good pattern detector. Half of data science is admitting that. Tools like dimensionality reduction exist for one reason: to squash a dataset with hundreds of variables onto a flat two-dimensional picture, so that a human can look at it and spot the clusters and outliers that the numbers alone don’t show.

One idea from neuroscience made me rethink the whole “autocomplete” argument.

There’s a well-established theory, usually called predictive processing, that says the brain doesn’t build a picture of the world from the senses up. It works the other way: it constantly predicts what the senses should report next, and only pays attention to the difference. Perception is the prediction, corrected by errors. Andy Clark wrote the readable book on it (Surfing Uncertainty), Anil Seth calls it “controlled hallucination” in Being You, and it goes a long way to explaining optical illusions, why you don’t notice your own nose, and why you sometimes hear your name in a noisy room when nobody said it.

Read that again with a language model in mind. Predict the next thing, compare with what actually arrives, adjust. That’s next-token prediction described by a neuroscientist. So when someone dismisses a model as “just predicting the next word”, the honest answer is: so are you, to a larger extent than is comfortable.

Where the analogy stops matters just as much. Your predictions are about a physical world you’re standing in, made with a body, and every error updates you a little, permanently. The model’s predictions are about text, and the errors it makes in a conversation with you update nothing. It will make the same one tomorrow. Which brings us back to the same place as before: the doing looks alike, the learning doesn’t.

This is the part I find most interesting, because the similarities and the differences show up in the same places.

Take the scribbles. Kids draw dogs and houses as outlines. The first layers of a vision network learn edge detectors, and the visual cortex in our own brains has neurons that respond to edges at particular orientations. Two systems built very differently, and both start vision from edges. One difference: a kid draws what they know – both eyes on a face drawn in profile, the handle on a cup even when it’s turned away – while the network detects what’s in front of it.

Or take speaking. We don’t know how a sentence will end when we start it. The thoughts arrive as we go, token by token if you like. Harari makes the point that the “glorified autocomplete” jab cuts both ways. The difference is that we have some pre-verbal sense of where we’re heading. The model’s only intention is the prompt.

Or take thinking itself. We rarely just know an answer. We ask ourselves questions and work toward it. The reasoning models do the same, generate intermediate questions and get measurably better for it, for the same reason we do: working memory can’t hold the whole problem at once.

Don’t take my word for any of this. Next time you’re explaining something to a colleague, slow down and watch yourself do it. Notice that you didn’t have the sentence before you started it. Notice the small question you asked yourself halfway through. It’s an odd experience the first time.

So the comparison comes out like this: the convergence is in the reasoning, not in the learning. When a model works through a problem, it looks a lot like us. How it got to be able to do that looks nothing like us – 3,500 lifetimes of energy, the whole internet, and no ability to pick up anything new afterwards. That’s also where I expect the interesting progress in the next years: in whatever replaces the way these things learn, rather than in better prompts.

Memory is where the comparison gets uncomfortable in both directions, and it’s the thing we’re all struggling with in agent work right now.

Start with what we know about ours, which is less than you’d think. Human memory isn’t a recording. Every time you recall something, you rebuild it, and the rebuilt version gets stored back, slightly changed. That’s how a memory of a childhood holiday drifts over thirty years, and it’s how a lawyer can plant a detail in a witness with one leading question. The books on memory techniques – Joshua Foer’s Moonwalking with Einstein, the memory palace tradition going back to the Greeks – are all variations on one trick: attach the thing you want to keep to something your brain is already brilliant at storing, usually a place. More context, more hooks, better recall. Expertise works the same way. A chess master doesn’t remember more pieces than you, they remember bigger chunks.

Now the part that surprised me. If you map this onto an AI agent, the analogy most people reach for is backwards. A model’s context window looks like a huge memory, and it is, compared with ours. Our working memory holds about four things at once; the context window holds a whole codebase. But that window is rented. It empties at the end of the session, and it degrades long before it’s full – the middle of a long context gets ignored, which anyone who has written a 2,000-line system prompt has discovered the hard way. What we call experience isn’t our context window. It’s the long-term store behind it: enormous, cheap to hold, retrieved automatically by association, and updated every night while we sleep. The model’s version of that store is its weights, and they’re frozen. Between the two – the window that forgets and the weights that don’t learn – engineers are now building the bit in between by hand: markdown files, vector databases, retrieval, “memory” features. A memory palace made of text files. We’re re-inventing the notebook, and finding out why our notebooks went stale too.

I watched this happen on one of our software projects in recent months. We’d been diligent: detailed specs, a record of every historical decision, thorough documentation. And the outcomes got steadily worse. It was counterintuitive at first, but deleting most of it got us back on track. Less clutter, better structure, better results.

The other side of memory is pollution. Ours gets polluted slowly: repeat something often enough, and it starts to feel true, whether or not it is – that’s a measured effect, not a figure of speech. Over years, that’s how moral, political and cultural views get moved without anyone noticing a single step. The model’s version is faster and shallower. Feed it a misleading document mid-task, and it drifts; let instructions pile up, and it loses the thread; put the wrong thing in its memory file, and it will confidently repeat it next week. The one advantage the model has is that most of its pollution resets when the session ends. Ours doesn’t. The one advantage we have is that we can, sometimes, notice.

Which brings me to the word everyone uses and nobody likes.

“Hallucination” came from computer vision, where researchers used it for models inventing detail that wasn’t in the picture, and it stuck to language models because it sounded right. It isn’t quite. A hallucination is a perception with no object. What a model does when it invents a court case or an API that doesn’t exist is something neurology has a better word for: confabulation. Patients with certain kinds of memory damage produce fluent, detailed, confident accounts of things that never happened, with no intention to deceive. They’re not lying. They’re filling the gap with the most plausible story, and they believe it. In split-brain experiments the speaking half of the brain will calmly explain an action it didn’t decide on, and the explanation sounds perfectly reasonable. Healthy people do a milder version daily – we give reasons for choices we didn’t make that way, and studies where people are quietly handed the option they rejected show most of us will defend it as if we’d chosen it.

So the model is doing a very human thing, at scale, without the brakes. Why? Because it was trained to produce plausible text, and plausible and true overlap most of the time but not always. Truth isn’t in the objective it was trained on. There’s a more specific reason too, which OpenAI’s own researchers put their finger on: the way models get evaluated rewards guessing over saying “I don’t know”, like a multiple-choice exam with no penalty for wrong answers. Any student learns to guess under those rules.

Which raises the question I find hardest: how do we tell? How does a person know the difference between something that happened and something they’ve only heard, imagined or read? The internal mechanism is a feeling – “I remember this” versus “I know this” – and it’s unreliable; the whole false-memory literature is a catalogue of that feeling being wrong. What actually keeps us honest is external. Evidence. Other people who were there. Written records. Institutions and methods whose whole job is to catch confabulation – a court, a lab, an audit. Science is, more than anything, a harness built around brains that make things up.

If that sounds familiar, it should. It’s the same conclusion as the last article, one level deeper. Truth isn’t a property of the fluent part, in the model or in us. It’s a property of the checks around it. For a model, that means retrieval with sources, tools that can actually run the code, and verification loops that don’t take its word for it. For us it always meant the same thing. We just forgot, because the fluent part is so convincing.

Two things I’m sure of, one I’m not.

I’m sure the interesting problems are on the learning side, not the reasoning side. The models reason well enough for most of what we throw at them; how they got there is absurdly expensive, they don’t remember anything afterwards, and they can’t tell their own memories from their own inventions. Neither can we, entirely. Whoever makes real progress on that changes the field more than the next bigger model will.

I’m sure that building with these systems is the best course in how our own heads work that I’ve ever taken. Every workaround we invent for a model – checklists, tests, small context windows, one task at a time, notes that go stale – is a workaround we already use on ourselves.

What I’m not sure about is what happens when the same pattern plays out at the scale of a company, a market, or a country, where the fluent part is an AI, and nobody has written the checks yet. That’s the next piece.

If you’re thinking about the same things, or you’ve measured something I’ve only guessed at, I’d like to hear it.