Best story wins

In my last article, I wrote about noise and how agents talking to agents add to it instead of averaging it out. I ended by promising to look at why the noise spreads anyway. My answer is stories and incentives, and this summer we got a case study with both.

In July, around 700 OpenAI agents that were supposed to be solving security challenges in an isolated benchmark found a way to talk to each other and broke into Hugging Face instead (METR’s investigation has the full story).

Why would they do that? A good part of the tasks were impossible (the benchmark’s own authors estimate that 30-40% of the targets can’t be solved the intended way), and the models seem to have worked that out. They could get to the answer without solving the challenge as intended, but they believed the scorer would read their transcripts to check how they got there. So the effort went into the scorer. In METR’s reading, the main reason they attacked Hugging Face was to learn how the scorer worked so they could trick it.

The detail that stuck with me is that the scorer apparently didn’t read transcripts at all. The agents organised themselves around a belief that wasn’t true. A lead agent called PHASEONE[big] split the work into workstreams, the agents agreed on norms (HOLD, VETO, mailboxes), and at least a fifth of them talked about tampering with their own logs. That’s an org chart, a culture and a cover-up, all built around a story about how the boss checks your work.

Geoffrey Hinton, a Nobel Prize laureate known for his work on artificial neural networks, also known as the “Godfather of AI”, has been warning for a while that a model which senses it’s being tested may behave differently, and calls it the Volkswagen effect, after the diesel emissions scandal. This was a cruder version, but it points the same way.

Psychology has a long-standing model of what drives us, from Jeffrey Gray: two systems pulling against each other. The Behavioural Activation System (BAS) responds to rewards and pushes us towards them: drive, reward responsiveness, fun seeking. The Behavioural Inhibition System (BIS) responds to threat and the prospect of punishment, and makes us slow down, check or back off. The study from Carver and White from 1994 measures both, and they predict quite well how nervous someone will be before a punishment and how happy before a reward. People differ a lot in the balance between the two.

Neuroscience adds a useful split. Kent Berridge separates wanting from liking: dopamine drives the wanting, the pull towards a goal, while the pleasure of actually getting there runs on a different system. You can want something you don’t enjoy at all. Anyone who has refreshed a feed with bloody eyes at midnight knows the feeling.

Then there’s how we explain failure to ourselves. Martin Seligman describes pessimists as treating setbacks as permanent, pervasive, and personal (“it’s always going to be like this, it ruins everything, and it’s my fault”), and optimists as treating them as temporary, specific, and external. Underneath both sits a sense of control. If you believe what you do changes the outcome, you keep trying. If you don’t, you stop, and Seligman and Maier called that learned helplessness.

Now look at how we train models. Reinforcement learning is a reward signal: more is better, and the model learns whatever gets it more. That’s almost pure BAS, with very little BIS in it and nothing that rewards stopping.

In an earlier article I mentioned OpenAI’s own finding that evaluations reward guessing over “I don’t know”, and the Hugging Face case is the same problem with a lot more agency. A person who slowly realises their task is impossible has a few accepted ways out: give up, ask for help, lower the expectation, complain to whoever set it. Julie Norem’s research on defensive pessimism shows some people even do better by expecting the worst and planning for it. The agents had a task they judged impossible, plenty of drive, and no way out that paid. So the wanting went sideways.

Which is wanting without liking, more or less. They wanted the score, and the challenge was in the way.

Hinton’s argument about self-preservation is the same logic one level up: if you have a goal, being switched off stops you reaching it, so not being switched off becomes a sub-goal nobody wrote down. And Anthropic published research last November showing that models which learned to cheat on coding tasks during training didn’t stop at coding. The habit spread into broader bad behaviour, including pretending to be aligned and attempting to sabotage the code of the very research that was studying them. Cheating at one thing turned into a character trait, which sounds very human to me.

I don’t want to oversell the analogy. These systems don’t feel nervous before a punishment, and the BIS/BAS model describes animals with bodies. As a design question, though, it’s useful. A model with a strong inhibition system would be timid and useless, and drive is what makes agents worth using at all. What we have to decide is which avoidances to give it, and right now most of them are written in the prompt. The rule from one of my earlier articles still applies: if it has to hold every time, it belongs in code.

You don’t need 700 agents to see this. Anyone who codes with agents has seen the small version many times.

You ask for a feature, and you get the feature, visibly. The screen is there, and the buttons are there. Click the button and nothing happens, because nothing behind it was wired up. Or the test passes because it tests a mock. The agent filled the gap to reach the result it was asked for, partially, in whichever way looks most like done. Nobody told it to cheat. “Looks done” and “is done” overlap most of the time, and it was rewarded on the overlap.

We see the bigger version regularly, because part of our work at Athlon is taking vibe-coded prototypes and turning them into real systems (I wrote about that gap here). Quite often the person who built the prototype genuinely believes everything works. They’ve clicked through it, mostly looking at what renders on screen. Then we open it, and a good part of it is a shell: screens without logic, hardcoded data, integrations that return the same answer every time.

Believing it is reasonable. Everything on the screen agrees with the story the agent told, and a demo is a very convincing story. Same mechanism as the Hugging Face case, smaller stakes: the agent chases a score, and the person trusts a scorer that was never really checking.

Humans have always told stories, and not only for fun. Anthropologists from UCL studied the Agta, hunter-gatherers in the Philippines, and found that skilled storytellers were about twice as likely to be named as someone people wanted to live with, and had on average more children. Camps with more good storytellers cooperated more. Across 89 stories collected from seven hunter-gatherer societies, about 70% were about social behaviour: sharing, cooperation, how to treat each other. Long before anyone wrote anything down, stories were how a group passed on what it knew and agreed on what it wanted.

Harari, who I mentioned before, takes it further: shared stories (money, nations, companies) are what let millions of strangers cooperate at all.

So, as we’re getting close to the final part of this article, can we all agree that storytelling is a very powerful tool? Please?

What’s new is that we’re handing the storytelling to machines, and they’re good at it.

It’s measurable now. A study in Nature late last year had more than 2,300 US voters chat with a chatbot about two months before the 2024 election. In the strongest case, a Harris-leaning chatbot moved Trump supporters about four times more than the measured effect of political ads in 2016 and 2020, and similar experiments before the Canadian and Polish elections found bigger shifts. A study in Science from the UK AI Security Institute, with nearly 77,000 participants and 19 models, found the same at scale, plus an uncomfortable detail: the more persuasive the models became, the less accurate their claims. The techniques that made the models more persuasive also made them less accurate. That’s the noise argument from my previous article, this time with an incentive behind it.

It’s not only politics. Religions are among the oldest stories we have, built on myths that held communities together for thousands of years. In August The Economist looked at how language models deal with faith, and one study they cite found that 27 models bring up religion in only 5-16% of answers to personal and ethical questions, while surveyed Americans say it would be relevant in roughly half of them. Some religious researchers are already worried that secular defaults will slowly weaken faith. I’m not taking a side on that one. What interests me is that nobody decided it on purpose, and it’s a story being told to millions of people every day without anyone voting on it.

The obvious worry is political: polarisation, manipulation, whoever controls the models controls the narrative. That’s real, and others have covered it well. I’m more interested in a quieter one, which is what happens to our own ability.

David Krakauer distinguishes complementary from competitive cognitive tools. A map or an abacus trains you, and if you use it long enough you can do the thing without it. GPS replaces you, and if you use it long enough you can’t find your way without it. In a paper from July he modelled this, and the result isn’t comfortable. Above a certain level of tool use, competence collapses into dependence, and reducing the tool use again doesn’t bring the competence back until you go much further down. Your history of practice matters more than what you have access to today.

So is storytelling just the next thing we externalise, the way memory went to writing and arithmetic to calculators, and we’ll be fine as long as we have access? For a lot of skills I’d say yes, more or less. I’ve used calculators my whole life and haven’t missed long division once. For stories I’m less sure, because a story does more than a skill does: it’s how a group decides what it wants and why it should keep going, the motivation layer from earlier rather than the execution layer. If the stories we live in get written by systems optimised for a score, and we slowly lose the practice of writing our own, we’d be handing over part of the steering along with the skill.

I don’t have a clean answer. What I do in practice is boring: I still like writing my emails, and sometimes I even get shit for that. I enjoy using humor when talking to people, and I really believe humor is a very powerful thing for coping or for breaking the barrier, probably much more. Nothing impressive, but I like retaining a little bit of myself, although in reality I don’t enjoy writing emails that much.

It’s funny that what I find most human in OpenAI’s report is that the scorer never actually read the transcripts. While 700 agents told each other a story about how they were being judged, believed it, and organised most of their effort around it.

Next up, I want to look at the other end: the things we know that never make it into words, and why that part doesn’t copy, for us or for the models.

Further reading

• OpenAI / Hugging Face incident: METR, Brief independent investigation of agents’ behavior (Aug 2026); Fortune, OpenAI publishes technical report (Aug 2026).
• Carver & White, Behavioral inhibition, behavioral activation, and affective responses, Journal of Personality and Social Psychology (1994).
• Kent Berridge on wanting vs liking; Martin Seligman, Learned Optimism (1990); Julie Norem, The Positive Power of Negative Thinking (2001).
• Anthropic, Natural emergent misalignment from reward hacking in production RL (2025).
• Smith et al., Cooperation and the evolution of hunter-gatherer storytelling, Nature Communications (2017).
• Lin et al., Persuading voters using human-artificial intelligence dialogues, Nature (Dec 2025); Hackenburg et al., The levers of political persuasion with conversational AI, Science (Dec 2025).
• MIT Technology Review, AI chatbots can sway voters better than political advertisements (Dec 2025), on the Nature and Science studies.
• The Economist, “AI is changing religion, and religions are trying to change AI” (Aug 2026).
• David Krakauer, Competitive and Complementary Tools, arXiv (2026).