Three new tests ask whether AI agents can use memory, not just store it

An AI assistant can appear to remember you while still failing at the thing memory is supposed to help it do. It might have the right detail stored but fail to retrieve it during a task. It might retrieve it and ignore it. Or it might correct an outdated detail while leaving an old plan built on that detail untouched.

Three research papers released between October 30 and November 2 approach those failures from different directions. MemU-Bench examines memory during long-running mobile-app tasks. MemUse looks at what happens after an agent receives a memory. StateGraph addresses what an agent should revise when a remembered premise changes. These are research proposals and evaluations, not announcements that consumer assistants have solved the problems.

What changed: Three places memory can fail

The first test starts with a familiar difficulty: a request can make sense to a person who knows the user’s history but be vague to an agent seeing only the latest instruction. MemU-Bench, introduced by Yizhou Liu and coauthors, contains 118 vague tasks drawn from 29 long-horizon user trajectories. It tests whether an agent can turn earlier interactions into useful memory and retrieve missing context when a task calls for it.

Its setting matters. The benchmark concerns mobile-app interactions over time, rather than only asking an assistant to recite a previously supplied fact. The accompanying method, AppMem, is described as training-free: it organizes memory around apps, tracks intentions interrupted by switches between apps, and selectively retrieves relevant information. That is a proposed approach, not proof that every app-switching problem is solved.

Imagine asking an assistant to finish something you began earlier without naming the app or repeating the details. The useful test is not whether a database contains those details. It is whether the assistant recognizes that the request depends on earlier context, finds the appropriate part, and uses it to continue the task. That is the distinction MemU-Bench is designed to probe. Source: MemU-Bench.

The second paper moves one step later in the process. A memory may be retrieved and placed in front of an agent, yet make no useful difference. MemUse, from Changran Xu, Zeju Li, and Qiang Xu, examines individual memory deliveries in recorded agent trajectories. It assesses relevance, accuracy, novelty, influence, and the observed effect on later actions, rather than treating retrieval or final task success alone as the answer.

That distinction produces an important caution. In the paper’s evaluation, success rates remained close to a no-memory baseline, while the authors identified many delivered memories that were irrelevant, went unused, or repeated information already visible to the agent. Source: MemUse. The result does not establish that memory is generally useless. It shows why a system can look busy retrieving memories without demonstrating that those deliveries helped.

The third paper considers what happens after a correction. StateGraph, proposed by Yiqi Wang and coauthors, tracks dependencies among remembered beliefs, plans, and actions. Instead of correcting only a fact that has been directly contradicted, its method checks whether conclusions that depended on that fact still have support. The paper also introduces StateChangeBench, a benchmark of 50 cases.

This is a different problem from finding the right old detail. Suppose, hypothetically, an assistant has recorded that an appointment is on Tuesday and made a plan around that date. Changing the appointment entry to Thursday would not, by itself, fix every Tuesday-based step. StateGraph is aimed at this kind of downstream revision. Its authors report larger gains over their comparison system on cases with deeper dependency chains. That reported result is specific to their evaluation; it is not a guarantee about deployed assistants. Source: StateGraph.

Why it matters: Recall is only the beginning

These papers describe different points in a single chain. An agent must preserve useful context, notice when a new request needs it, deliver the right memory, make use of it, and update its conclusions when circumstances change. MemU-Bench emphasizes the connection between past interactions and present action; MemUse examines the delivered memory’s contribution; StateGraph focuses on the consequences of revisions. None of the three, on its own, measures every link.

That matters because an impressive memory demo can conceal several kinds of failure. A system might answer a direct question about a past conversation yet be unable to apply that information when a user makes an indirect request. It might surface a correct preference at exactly the wrong moment, adding clutter rather than guidance. And it might accept a correction while continuing to act on an inference that is no longer justified. The three papers offer ways to examine those possibilities more closely, each within its stated research setting. Sources: MemU-Bench, MemUse, StateGraph.

They also suggest why a single score can be misleading. If a task succeeds, did memory help, or could the agent have succeeded using information already on screen? If it fails, was the needed fact never stored, retrieved too late, ignored after delivery, or undermined by an obsolete assumption? MemUse explicitly investigates delivered memories that repeat visible information or do not influence subsequent actions; MemU-Bench and StateGraph test different failure points before and after that moment. Sources: MemUse, MemU-Bench, StateGraph.

What it means for assistants

For people using an assistant, the practical question is not whether a company says it has memory. It is whether that memory improves the next task without allowing a stale detail to steer the result. A good evaluation would need to distinguish having information from using it appropriately, and accepting a correction from actually updating the decisions that depended on it. These papers make those distinctions more concrete, but their benchmarks should not be mistaken for independent evidence about the performance of any particular consumer product. Sources: MemU-Bench, MemUse, StateGraph.

For developers, the work points toward narrower, more informative questions to ask of a memory system. When a vague request arrives, can the agent find the earlier context that matters? When it receives a memory, does that item change an action for the better? When a fact changes, can it identify which conclusions must be reconsidered—and which should be left alone? The papers do not supply one universal test or one settled design. They do provide three concrete research settings in which the answer can be examined rather than assumed. Sources: MemU-Bench, MemUse, StateGraph.

The immediate news is therefore not a new assistant that remembers everything. It is a sharper set of questions for judging the claim. Memory becomes valuable when it helps an agent do the right thing now—and remains trustworthy when what it knew before is no longer true.