Can AI know what it doesn’t know?

A language model can generate smooth language about something it cannot verify. In a 2024 ACL study, Kaitlyn Zhou and colleagues reported that models often failed to signal uncertainty appropriately and that people could rely on outputs even when they were wrong. Yona and colleagues (2024) also found gaps between a model’s internal uncertainty and the decisiveness of its wording.

A better design question is what the system can show about the claim. Does a source support it? Is it about the right person? Does it still apply? Uncertainty language should communicate an evidence assessment, not substitute for one.

A record may never have been saved, may exist but not be retrieved, may conflict with another, may be superseded, may be ambiguous about who is speaking, or may describe a future possibility. Each calls for a different response: search further, fix retrieval, keep accounts distinct, add a time boundary, or label a prediction as a prediction.

Finding a record of one fishing trip establishes that a trip occurred, not that it was the first. A single confidence score cannot explain all these limitations or identify the right next action.

Sebastian Farquhar and colleagues explored semantic entropy in a 2024 Nature paper: comparing the meanings of several sampled answers. When meanings diverge, that instability can flag some confabulations, and withholding those answers improved accuracy.

But agreement is not proof. If an incorrect moving date has entered an archive, every answer may reproduce it consistently. Semantic consistency should supplement source checking, not replace it.

As an engineering analogy to monitoring and control, an AI could inspect the quality of retrieved evidence and then choose whether to answer directly, qualify, search again, or ask. Imagine three messages about a cabin: thinking about selling, a tentative plan, then a clear decision to keep it. Counting statements favors selling; careful reading recognizes that the later decision changes the current answer, and states that no later update was found.

A proposed monitoring layer would check evidence, attribution, time, independence of sources, and the cost of being wrong. These are design principles, not claims that AI has feelings.

Withholding every answer would avoid false claims but defeat the purpose of a memory assistant. Evaluation should examine accuracy and coverage together, and difficult questions should not vanish from evaluation because the system declines them.

A proposed experiment would compare answer-first, evidence-aware abstention, and extra checks on difficult cases, measuring unsupported personal claims, false abstentions, changed-fact errors, cost, and reasons for withholding. An honest uncertainty statement should make the next step clearer, not conceal a system failure.

From the outside, an AI that knows its limits sounds different. It answers plainly when its memory clearly supports an answer, says what it found when the evidence is partial, and says it does not have a record when it truly does not, instead of guessing.

A good habit for users is to ask where an answer came from. If the assistant can point to something you said or a file you shared, the answer rests on evidence. If it cannot, treat the answer as a suggestion to verify.

The goal is not an assistant that hedges every sentence. It is one whose confidence tracks its evidence, so that when it sounds sure, it usually is right.