Validate those AI answers, people

Ana Maria Constantin: AI advice made people three times less accurate but twice as confident, researchers found

Researchers from three French and Italian universities found that access to AI advice collapsed people’s willingness to say “ I don’t know ” from 44% to 3%. Accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. “People became much worse, the accuracy was only one third, but they were twice as confident,” said Valerio Capraro, associate professor at the University of Milano-Bicocca.

I think this is an interesting study that is definitely important to recognize. AI models do acknowledge when they don't know things more than they used to, but it's undeniable that they still sometimes get things wrong. Just like how you can't trust everything you read online, you can't trust everything an LLM says either, and you need to use your human faculties to validate things.

I will also note that this study does not test how often LLMs are wrong. They specifically chose a model that would get things wrong, and they specifically chose questions they knew it would get wrong. The study was specifically about when an AI gives a wrong answer, do people trust it? From that perspective, it would be interesting to see a similar study where people are presented regular blog posts written by humans that are inaccurate, and how many people get it wrong in that case. Are we particularly susceptible to LLMs getting things wrong, or do we just trust everything we read, no matter what entity wrote it?

For what it's worth, this is why I think LLMs are such a powerful tool for coding. Yes, sometimes the coding agent gets things wrong, but that is revealed immediately when the software doesn't do what it's supposed to do.