The Problem with AI Memory Living Inside the AI

In 1945, Vannevar Bush published "As We May Think" in The Atlantic — an essay about the coming problem of too much information and too little recall. His solution was the Memex: a desk-sized device that would store all of a person's books, records, and communications on microfilm, searchable and annotatable, with trails of association you could share with colleagues. He called it "an enlarged intimate supplement to memory."

The thing Bush got right, and that we've since managed to get wrong, is that memory you can see is fundamentally different from memory you can't. His Memex was made of physical microfilm. You could pull a reel, hold it up to the light, see exactly what was on it. If something was wrong, you could fix it. If something was private, you could remove it. Memory, in Bush's model, was a filing cabinet — organized, legible, yours.

Eighty years later, the AI tools we use every day have memory that works almost exactly the opposite way.

What's actually happening

When you use ChatGPT with memory enabled, it stores facts about you in a system you can partially inspect — there's a settings panel where you can see a list of saved memories — but the full picture is harder to see. Your conversation history is summarized and re-injected into future context windows in ways you can't directly audit. Charles Packer, founder of MemGPT, put it plainly in early 2025: "memory features in consumer chat apps mess with the context window in an opaque way — while they have the potential to raise the ceiling, they also can lower the floor."

The floor-lowering is the part that deserves more attention than it gets. If an AI has a faulty memory of you — you mentioned once that you were learning guitar and it now injects that into every technical conversation — you might not even notice why responses feel slightly off. The degradation is invisible by design.

Claude's approach is architecturally different: memory lives in markdown files you can open, read, and edit. But both approaches share a common problem: the memory lives in someone else's system. You can read it sometimes. You can't move it. You can't take it with you when you switch tools.

Why this matters more than it seems

There's a word for memory you can't verify: faith. You're trusting the AI to remember you accurately, update those memories correctly, and not let stale or wrong facts quietly shape every subsequent interaction.

That's an odd kind of faith to extend to a system designed to be useful rather than honest about its own state. Models hallucinate. They update context in ways that compound subtly over time. A memory that noted "user works in healthcare" after a single offhand comment might be accurate, or it might quietly miscalibrate every response about your actual work in fintech for the next six months. There's no way to check without already knowing something is wrong — and most of the time, you don't know.

Parametric memory — knowledge baked into model weights during training — has the same problem at a larger scale. The model doesn't know what it knows, exactly, and neither do you. Academic researchers have documented this for years: facts must be observed many times during pre-training for reliable memorization, and during inference it's essentially impossible to surgically unlearn outdated or incorrect facts from weights alone. A model trained through 2023 might know that a company's CEO was someone who left in 2024. It will still answer confidently.

The invisible error is always worse than the visible one.

What inspectable memory looks like

The alternative isn't particularly exotic. It's closer to Bush's filing cabinet than to neural network architecture: memory as structured, human-readable records that you control.

A person record with actual fields. A preferences file with entries like "prefers direct answers, source: user note, confidence: high." An audit trail showing when the AI added something, what it read before adding it, and a diff you can approve or reject before it sticks. Not a black box silently injecting opaque summaries — a set of readable files you can open in a text editor on a Tuesday afternoon when you're wondering why the AI started treating you like a morning person.

The reason this matters is trust, but trust in a specific technical sense: legibility. You can only genuinely trust systems whose behavior you can trace. Not every user wants to audit their AI memory manually. But the option needs to exist. When something feels wrong, you should be able to go look.

There's also the portability problem. Memory that lives inside a company's proprietary system is memory you lose when you switch products, or when the service shuts down. That isn't a hypothetical risk — it's the default outcome for every productivity tool that has ever closed its doors. Memory as exportable files, or files on your own disk, is memory that travels with you.

The Memex was never built. Bush's microfilm desk remained a thought experiment, and we got the web instead, which is better in almost every way. But his core intuition — that personal memory should be something you can hold, inspect, and traverse — kept resurfacing. Hypertext, wikis, personal knowledge bases. Now AI assistants that can query that knowledge, update it, reason over it.

The question is whether the memory underneath all of that will be yours in any meaningful sense. Right now, for most people, it isn't.

---

Asgeir Albretsen is the founder of Harbor.