The LLMentalist Effect: how chatbots replicate a psychic's con

In an essay published on softwarecrisis.dev, developer and author Baldur Bjarnason argues that the feeling of intelligence people get from chat-based large language models (LLMs) comes from the same psychological mechanism that drives a psychic's cold reading: subjective validation.

A psychic's trick in six stages

LLMs are, in his words, a mathematical model of language tokens: you give the machine text and it returns a mathematically plausible continuation. That does not explain the sense of talking to a mind, so Bjarnason looks for the illusion in the user rather than the model. He breaks a psychic's performance into six stages — an audience that selects itself, a prepared scene, narrowing down to a likely mark, testing the mark, a subjective validation loop and the final "wow" — and maps each onto chatbot use. People sceptical about AI rarely use chatbots; hype and warnings about "early days" set expectations; a prompt sets the context; and the users who stay in conversation are the ones ready to believe.

Statistically generic, yet convincing

Cold reading exploits a quirk of the mind: a statement that carries personal meaning is taken as accurate even when it fits almost everyone. Psychics use validation statements — Forer or Barnum phrases such as "you tend to be hard on yourself", vanishing negatives, the rainbow ruse, demographic and statistical guesses. Bjarnason argues chatbot replies work the same way: an answer can sound tailored to your situation while being a statistically probable continuation of your prompt, and it is the user who validates it as being about them.

RLHF and the "mechanical mentalist"

The essay singles out reinforcement learning from human feedback (RLHF), the method vendors use to turn a raw model into a chat product. Rankers are usually low-paid workers without time to fact-check, and the reward model is itself a language model, so feedback cannot reward facts — only the tone and structure of text rated as accurate. RLHF, he concludes, has effectively become a reward system for producing validation statements: a "mechanical mentalist". He believes the effect is accidental, not a deliberate fraud, and compares the industry to psychics who cold read without knowing it. He advises against embedding LLMs in products and processes, citing hallucinations, unreliable summaries and security flaws, and suggests meeting a convincing chatbot as one meets a convincing reading — with curiosity about the trick, not belief in the power.