Agent Memory Is Four Kinds of Files

Working with a coding agent keeps raising the same small question. I just learned something, so where do I write it down? It might be a build quirk, a decision, or a procedure that finally worked. My setup has at least four plausible places for it and no obvious rule for choosing.

An IBM Technology explainer this week gave me better vocabulary. It walks through CoALA, short for Cognitive Architectures for Language Agents, which it attributes to a Princeton research team. The framework sorts agent memory into four types: working, semantic, procedural and episodic.

The taxonomy is not the interesting part. The interesting part is IBM's observation that in production systems each type ends up as a file.

The four types

Working memory is the context window: the current conversation, the system instructions, whatever files are loaded. The video says even windows past a million tokens lose track of material buried in the middle, and gives no number for that.

Semantic memory is facts, rules and conventions. The academic framing is vector databases and knowledge graphs. IBM's point is that many production agents implement it as markdown loaded at the start of every session, with Claude Code's CLAUDE.md as the named example.

Procedural memory is how to do things. Here the video points at the agent skills standard: a folder with a SKILL.md. The agent sees a light index of names and descriptions, about 100 tokens per skill by IBM's figure, and loads the full instructions only when a task matches.

Episodic memory is what happened before. Saving every transcript technically counts and is rarely useful. What works, in IBM's telling, is distillation. Their example: a note that the last auth bug was in the middleware layer is worth more than the 45-minute transcript of finding it.

Here is how that lands on my own vault.

Naming the type picks the file

Once the four names exist, the where-does-this-go question turns into a short checklist.

• True regardless of what I am doing today: semantic.
• A sequence of steps I want repeated the same way: procedural.
• Something that happened, with a lesson that may matter later: episodic.
• Only relevant until this task is done: leave it in the context window and write nothing.

Reysu's walkthrough of his Claude-managed Obsidian vault is a good test, because he never uses the vocabulary and the pieces still sort cleanly. His daily folder is a journal the agent writes itself. He has Claude append an entry each time it finishes a task so the next session can read what was done. That is episodic memory as a dated file. Inside one project folder he keeps a guide for creating flashcards, with sync rules and classification logic, which Claude reads whenever it picks that project back up. That is procedural memory stored next to the project.

The second example showed me a gap in my own setup. My CONTEXT.md files describe what a project is, which is semantic. They carry no procedure for the recurring work inside it.

Reysu also keeps AGENTS.md as a shorter mirror of CLAUDE.md for subagents. Nobody in the video measures the saving, and it promotes a paid setup guide, so I am treating that as a design to test.

Stable facts go in the file that loads first

A second IBM video, on cache augmented generation, sharpens the semantic row. It describes provider prompt caching as a service that lets requests sharing a prefix skip reprocessing it, at something like a 90 percent discount on those tokens. The catch is that any change to the cached material forces a recompute. The speedups it quotes, ten to forty times, are credited to unnamed research, so I read them as order of magnitude.

What I take from it is a placement rule. Semantic files that load at session start should hold facts that stay put. A detail that changes every week does not belong in CLAUDE.md, because it goes stale and because it churns the part of the context that is cheapest when it stays the same. That is my inference. The video is about knowledge bases, not agent config files.

Forgetting is the unsolved part

IBM calls episodic memory the hardest type, because something has to decide what gets deleted and when a fact has expired. The video stops there. It also says nothing about when an episodic lesson has earned promotion into semantic memory.

Chris Raroque's personal agent has one answer to the first question. He runs a nightly job with three agents. A consolidator proposes pruning, merging or promoting each memory. An adversary argues for keeping it. They get up to two rounds, and a stronger model breaks any remaining tie. He puts the cost near 50 dollars per nightly run, absorbed by a flat-rate subscription. That figure is spoken, not shown, and not being billed for it is different from it being cheap.

My own state is uneven. Compiled articles in my vault carry a staleness_signal field that says what would make the article wrong. My memory files have nothing like it. Laid against the table, that reads as a forgetting policy for semantic memory and none for episodic.

Key Takeaways

• CoALA names four kinds of agent memory. IBM's useful addition is that production systems implement each one as plain files.
• Naming the kind picks the file: stable facts in context files, repeatable steps in skills, distilled lessons in memory notes, everything else left in the session.
• A project needs both a description and a procedure, and those are usually two files.
• Forgetting is still open. Adversarial pruning is one design, and its reported cost is high.

Sources

• IBM Technology, "The Four Types of Memory Every AI Agent Needs"
• IBM Technology, "CAG vs Long Context: How AI Models Use and Remember Information"
• Reysu, "A look inside my LLM second-brain setup (Full walkthrough)"
• Chris Raroque, "I didn't like OpenClaw so I built a better version"
• CoALA, "Cognitive Architectures for Language Agents" (Princeton), as described in the IBM video. I have not read the paper itself.

Next: a staleness field on memory entries, and a rule for when one of them graduates into a compiled article.