Artificial intelligence models can give users the wrong answer and do so with great confidence. They can also hedge and warn that they are unsure — even when they get the answer right.
A study led by UC Riverside computer scientists helps explain why. Researchers found that confidence and correctness can arise from different internal features within large language models, challenging the assumption that a model’s confidence is a reliable indication of whether its answer is accurate.
Their discovery could help build more reliable AI models. As large-language models are increasingly used to inform decisions and complete tasks, developers need better ways to determine when their answers can be trusted.
By identifying internal features associated separately with confidence and correctness, the UCR-led research points toward ways AI systems could be adjusted so they are more confident when they are right and more cautious when they are likely to be wrong.
“The main assumption in the field is that when the model is confident, it is likely to be correct, and when the model is unsure, it is more likely to be incorrect,” said Het Patel, a UCR computer science doctoral student and lead author of the study. “But we often see the counterexamples that are well documented. A model can answer with certainty, but also be wrong, or it can answer while being less confident and can be correct.”
The researchers went beyond documenting that mismatch. They identified internal features associated with confidence and correctness and showed that altering some of them could change model behavior without the costly process of retraining an entire model.
Patel explained that modern AI models are created by training enormous networks of mathematical units on vast amounts of data. During training, the network repeatedly adjusts billions of numerical parameters, called weights, as it learns patterns in the data. In a language model, those learned patterns allow it to predict which words are likely to follow others and ultimately generate responses to questions.
Patel and his colleagues wanted to know what was happening inside the models when confidence and correctness did not match.
They studied two “open-weight” large-language models — Meta’s Llama-3.1-8B and Google’s Gemma-2-9B — whose internal workings can be examined by researchers. Using multiple-choice questions, they separated responses into four groups based on whether answers were correct or incorrect and whether the models were confident or uncertain.
They then used tools called “sparse autoencoders” to examine the models’ internal activity. That allowed them to determine which features became active with particular behaviors.
The analysis identified three kinds of features: those associated primarily with uncertainty, those associated primarily with incorrect answers, and “confounded” features associated with both. The researchers then suppressed selected features as the models answered questions. Patel compared the process to “turning knobs” inside a model to see how its behavior changed.
The differences were striking. Turning off features associated purely with uncertainty sharply reduced accuracy, suggesting those features play an important role in producing good answers. By contrast, suppressing most features associated solely with incorrect answers had little effect.
The confounded features produced a different result. Suppressing features associated with both uncertainty and incorrectness improved accuracy by up to 1.1% while reducing the models’ uncertainty by up to 75%. Similar effects appeared across different question-answering benchmarks.
The interventions were made when an already-trained model is answering questions, and do not require retraining it.
“In a sense, it’s kind of like adjusting or modifying the values of these activations or features after the fact to kind of get the behavior you want,” Patel said.
Another experiment suggests the internal signals could eventually help AI systems decide when not to answer. Using just three of the confounded features from a single middle layer of the Llama model, the researchers could predict whether the model was about to answer incorrectly. Having the model decline to answer the questions flagged this way raised its accuracy from 62% to 81% while it still answered about 53% of the questions.
By comparison, giving the model an “I don’t know” option and letting it abstain on its own raised accuracy only to about 64%. The results show the findings are not only observational; the same internal signal points to a practical step developers can take to make AI systems more reliable.
The researchers also found evidence that these internal features were not tied narrowly to individual benchmarks. Features identified using one benchmark produced similar effects when applied to others, suggesting they reflected more general characteristics of the models.
Patel said the approach could extend beyond confidence and correctness. Researchers could search for internal features associated with other desirable or undesirable AI behaviors and test whether manipulating them changes how models perform.
“You could pick another behavior you want or don’t want, find the features related to it the same way, and then work on those internal features to drive that behavior or reduce it,” Patel said.
The study, “Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders,” has been accepted for presentation at the SIAM International Conference on Data Mining in Salt Lake City in November.
In addition to Patel, the co-authors are UCR Professor Evangelos E. Papalexakis, the Ross Family Endowed Term Chair in Computer Science; Jia Chen, a UCR associate professor of teaching; and Arizona State University researchers Tiejin Chen and Hua Wei.
