A very pro-AI account on both Bluesky and X posted about a "disturbing" Stanford paper on LLMs' failures at reasoning

[highplainsdem]

13. It's called reasoning by people working on and promoting AI, but it's really more a pretense of

reasoning. Some people are still really impressed by the AI supposedly showing its reasoning, like a schoolkid showing their work solving a math problem.

But they found out nearly a year ago that the new "reasoning" AI models actually hallucinated more than older AI models that don't show their reasoning. See this thread I posted last April and the article it's about:

OpenAI's new reasoning AI models hallucinate more
https://www.democraticunderground.com/100220267171
https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/

OpenAI found that o3 hallucinated in response to 33% of questions on PersonQA, the company’s in-house benchmark for measuring the accuracy of a model’s knowledge about people. That’s roughly double the hallucination rate of OpenAI’s previous reasoning models, o1 and o3-mini, which scored 16% and 14.8%, respectively. O4-mini did even worse on PersonQA — hallucinating 48% of the time.

Third-party testing by Transluce, a nonprofit AI research lab, also found evidence that o3 has a tendency to make up actions it took in the process of arriving at answers. In one example, Transluce observed o3 claiming that it ran code on a 2021 MacBook Pro “outside of ChatGPT,” then copied the numbers into its answer. While o3 has access to some tools, it can’t do that.

This new study took a more thorough look at the reasoning failures.

I thought the stunned and apparently scared reaction from the pro-AI account was worth posting here, especially this:

One of the most disturbing findings is how often models produce unfaithful reasoning. They give the correct final answer while providing explanations that are logically wrong, incomplete, or fabricated.

This is worse than being wrong, because it trains users to trust explanations that don’t correspond to the actual decision process.

-snip-

The takeaway isn’t that LLMs can’t reason.

It’s more uncomfortable than that.

LLMs reason just enough to sound convincing, but not enough to be reliable.

And unless we start measuring how models fail not just how often they succeed we’ll keep deploying systems that pass benchmarks, fail silently in production, and explain themselves with total confidence while doing the wrong thing.

As a genAI nonbeliever, my first response to reading that had been to laugh at anyone not already understanding that.

It's been known for years that genAI models make lots of mistakes while still sounding convincing and authoritative. That's why even AI companies peddling this inherently flawed tech admit it's important to check AI answers because they're often wrong.

But people who like to use AI tend to push that warning aside, and those gullible people are even more impressed when an AI "shows its reasoning."

This new paper exposes just how foolish it is for an AI user to do that.

One of the most disturbing findings is how often models produce unfaithful reasoning. They give the correct final answer while providing explanations that are logically wrong, incomplete, or fabricated.

This is worse than being wrong, because it trains users to trust explanations that don’t correspond to the actual decision process.

You have to be gullible to let a machine that can't reason "train" you to trust it. But it's been known for years that the more someone uses chatbots, the less likely they are to bother checking the AI results. Plus chatbots are designed to persuade and manipulate - to keep AI users engaged - and there are a lot of gullible AI users out there. Which is why we hear more and more stories about what's often called AI psychosis, where a chatbot gradually pushes a too-trusting user into delusions that can result in breakdowns and even suicide.

AI fans like to believe that isn't likely to happen to them. They also like to believe their favorite AI models really are trustworthy. This new study blows up that assumption of trustworthiness.

But because it blows up those assumptions, there will probably be a lot of AI users who will refuse to read it or believe the conclusions.