Agent Persistent Memory Graph: Hands-On Measurements and Limits
In short: After actually running an agent memory graph in our own operations, the verdict is one line: "remembering is easy; forgetting and verifying are hard," and the measurements say it plainly: 8,882 facts have accumulated, but about 76% are unverified stale candidates (8,086 awaiting verification). The graph writes faster than it verifies and prunes.
After actually running an agent memory graph in our own operations, the verdict is one line: "remembering is easy; forgetting and verifying are hard," and the measurements say it plainly: 8,882 facts have accumulated, but about 76% are unverified stale candidates (8,086 awaiting verification). The graph writes faster than it verifies and prunes. And retrieval (recall) does not give the "relevant" we expect but the embedding-"similar" - in fact, asking for "voice cloning reference quality," the top hit was a schema note rather than the quality guidance, and an unrelated 3D model's "quality report" slipped in on the shared word "quality" alone.
In plain terms: a memory graph is a vast box of notes. Putting notes in is easy; the real work is (1) continually verifying whether old notes are still true and (2) pulling out the "truly relevant," not the "similar-worded," when you search. Pruning and searching well is harder than growing the box.
What state is the graph in when you actually run it?#
Large, but skewed by type. Read-only, there are 8,882 facts (8,667 active), by type semantic 5,207, episodic 2,888, procedural 787. That is, "what" (facts) is plentiful but "how" (procedures) is only 787, so reproducible know-how is relatively thin. Projects are also fragmented across 200+ (custom-game 1,081 vs game-research 90), so density varies widely by area. Average confidence is 0.735, higher in well-worked areas (e.g. tts-cloning 0.848).
First, unpack the terms. Semantic memory is a fact like "X is Y," episodic memory is an event record of "what happened when," and procedural memory is reproducible know-how, "do this, this way." For people, procedural memory is the most valuable (it automates repeated work), yet in our graph it is the thinnest.
| Item | Measured | Meaning |
|---|---|---|
| Total facts | 8,882 (8,667 active) | well accumulated |
| Verify debt | ~76% stale, 8,086 pending | writing > verifying |
| Type skew | semantic5207, episodic2888, procedural787 | much "what," thin "how" |
| Confidence | average 0.735 | varies by area |
| Recall | by similarity (not relevance) | needs re-ranking |
- 표본
- 2 measured metrics (Hax /data curated)
- 수집일
- 2026-07-04
- 방법
- bench_harness.probe_curator (curator stats 실측)
What is the biggest limit?#
Verify debt and the decay of truth over time. A fact true "when written" can be false next month (code, prices, policy change), and if verification lags the graph keeps believing old facts. Our measured 76% stale and 8,086 pending are exactly that debt. So a memory graph's core engine is not "insert" but forgetting (decay), invalidation, and re-verification; weak at that, it becomes large but untrustworthy memory. In live search, many high-relevance candidates showing ==verified=never== is the face of this debt.
A common misconception here: it is not "the more you store, the smarter it gets." Store without verifying and wrong facts rise to the top of search, actively confusing the agent. So when the balance between "write speed" and "verify/prune speed" breaks, the trustworthiness of the whole graph drops.
Can you trust recall?#
Do not take the top results at face value. Semantic search pulls by embedding similarity, not meaning, so a fact that merely shares words with the query can rank high. As measured above, an unrelated 3D report slipped in on the shared word "quality." So recall becomes usable only after re-ranking and filtering by confidence, verification time, and project scope (trust top-k blindly and you mistake noise for fact). Pinning and project-scoped search sharply raise precision.
Embedding similarity turns a sentence into coordinates in hundreds of dimensions and measures how close those coordinates are. Closeness can come from overlapping surface words or context rather than shared meaning, so one word like "quality" can drag in a completely different domain. That is why, instead of trusting results as-is, you layer on filters for confidence, recency, and scope.
So how do you use it well?#
The key is investing in "verify and search," not "store."
- Hygiene: re-verify stale facts regularly and invalidate wrong ones (prune as much as you add).
- Search: re-rank recall by confidence, recency, and scope, and pin the core facts.
- Balance: deliberately keep reproducible "procedural" facts, not just "what" (the most valuable yet the thinnest).
Reference links
- Mem0 (agent memory layer)
- Graphiti (temporal knowledge-graph memory)
- Letta (formerly MemGPT, memory management)
- MemGPT paper (LLM-as-OS memory)
- LlamaIndex (knowledge graph/RAG)
Note: figures are a read-only live snapshot of our graph in 2026 and shift every moment with operation and pruning cadence (not permanent numbers). Stale ratio and recall quality depend on embeddings and verification policy, so measure exact precision on your own data and queries (these numbers are only a start). Memory frameworks move fast, so this is reviewed quarterly.
Responses
No responses yet. Be the first to respond.