Large language models and personal memory

Speaker: Paul Smart

Smart's argument was that language models are becoming a genuinely new kind of personal memory technology, not passive storage, but an active participant in encoding, consolidating, retrieving, and reconstructing someone's own memories. He pointed to existing and emerging systems as evidence this is already underway. Tools that support context-sensitive access to personal information, a system called OS-1 explicitly modeled on the AI companion from the film Her, combining multimodal sensing with persistent memory and conversation, and the memory feature now built into ChatGPT, which stores discrete facts about a user and uses them to shape later responses.

Four mnemonic processes

Drawing on a recent position paper of his, he laid out four ways a model can participate in personal memory:

Encoding. Turning raw multimodal input (images, speech, location data) into structured, semantic records, including actively asking the user for clarification when the input is incomplete or ambiguous.
Elaboration and consolidation. Summarizing what's stored, linking records semantically and temporally, and binding fragments into coherent events that are easier to recall later.
Retrieval. Memories can surface through an explicit query, through current context, or through a progressive back-and-forth where the user offers a partial cue and the model narrows in, with either the model or the person ultimately landing on the right memory.
Joint reminiscing. The model helps revisit, refine, and re-narrate past experience, potentially co-creating something new, like a multimedia storybook, out of it.

Across all four, he argued, the model isn't functioning like a filing cabinet. It's actively involved in constructing and reconstructing the memory, closer to a conversational partner than a passive archive. That matters most for autobiographical memory specifically, which is itself organized across multiple timescales (whole life periods, recurring themes, individual episodes), and a model could plausibly generate narratives at any of those scales, from a single event up to something like an overarching life story. That matters because those narratives aren't just records of what happened. They shape a person's sense of who they are and where they're headed.

Is hallucination actually a problem here?

Human memory is itself reconstructive and imperfect. If a model's elaboration, even one that isn't strictly accurate, helps produce a more coherent personal narrative, that might be a virtue rather than a vice for this particular use, in a way it clearly wouldn't be for, say, factual question answering.

He closed by noting this kind of memory work is deeply tied to language, which raises real questions about whose narratives get well served by a system built primarily around linguistic fluency. And that LLM-based memory systems give philosophers a concrete, testable case for older questions about extended memory, whether and how memory can be constituted partly outside the biological brain, in the tradition of the extended mind thesis.

I came into this talk partway through, so the framing of the earlier points is reconstructed a bit and might not be exactly how he put it.