CatMemory: Designing an Agent Memory System That Isn't Just Retrieval

I. Memory Is Not Just Storage

The common approach to building a memory system, in most mainstream implementations, is fairly simple:

1. Chunk memory records and session information;
2. Encode via an embedding model and store in a vector database;
3. At retrieval time, use multi-path recall (e.g. BM25 + vector search), reranked via RRF (Reciprocal Rank Fusion);
4. For systems with a higher quality bar, layer on a reranking model for a second, finer pass.

⚠️ The core problem: this is a retrieval system, not a memory system.

Plain chunking-plus-vector-search only solves “find related content” — a real memory system also needs to solve for:

• Temporal evolution of memory: it can’t distinguish which version of a fact is current as of a given point in time (e.g. it can’t tell whether “used MySQL in March” or “migrated to PostgreSQL in April” reflects the present state).
• Causal linkage between memories: it can’t establish logical, causal, or sequential relationships between fragments stored in isolation.
• Adaptive compression of memory: it can’t converge and compress recurring information across a long-running conversation, so redundant information piles up without bound.

A typical open-source agent’s default file-system-plus-index memory setup illustrates the gap well — even in production, it still can’t cleanly solve for isolated fragments or dynamic evolution over time.

---

II. The Core Problems a Real Memory System Must Solve

2.1 Memory Has a Timeline

• Scenario: a user says in March, “Project X runs on MySQL,” then in April says, “Project X has migrated to PostgreSQL.”
• System requirement: when the LLM is later asked about Project X’s database, the memory system needs time-awareness — it must accurately retrieve and surface the current PostgreSQL fact, rather than returning both conflicting historical statements side by side and confusing the model.

2.2 Memory Isn’t Isolated

• Scenario: a user’s separate conversations may logically supplement, correct, or causally extend each other.
• System requirement: retrieval based purely on semantic similarity can easily miss context that’s logically highly relevant but has low surface-level similarity. The memory system needs some mechanism (like a graph or entity linkage) to preserve and reconstruct the deeper semantic relationships between memories.

2.3 Memory Needs Compression

• Scenario: as a user’s interaction with an agent deepens, raw conversation logs and the discrete facts extracted from them grow at a roughly exponential rate.
• System requirement: without merging and pruning redundant, duplicate, or stale memories, retrieval efficiency drops, storage cost balloons, and — more urgently — the LLM’s precious context window fills up fast.

2.4 Memory Needs Filtering

• Scenario: everyday interaction is full of low-value content (“hi,” “thanks,” small talk, idle chatter).
• System requirement: a memory system needs a “junk filter” that only converts genuinely meaningful content — preferences, facts, decisions — into memory, so junk data doesn’t pollute the retrieval candidate pool.

---

III. CatMemory System Design

3.1 System Overview

CatMemory is a distributed, remote memory service. It breaks free of the usual client-local-storage constraint, providing a unified, centralized long-term memory capability across multiple client products.

CatMemory isn’t just engineered for operability — its algorithm layer integrates time-awareness, relational reasoning, automatic compression, and offline “Dreaming” consolidation. All memory data ultimately converges on the CatMemory server, which handles extraction, linkage, vectorization, storage, retrieval, and consolidation end to end.

---

3.2 Memory Level Classification

CatMemory abstracts memory into five levels, from shallow to deep:

Level	Type	Business Meaning	Storage & Lifecycle
L0	Conversation Memory	Raw conversation content	Full raw session-level text, preserving complete context.
L1	Memory Unit	Atomic facts/topics extracted from a session	The smallest retrievable unit, extracted in real time from a single turn or session (e.g. “user prefers writing scripts in Go”).
L2	Project Memory	Memory clustered around a fixed topic	Spans multiple sessions, clustered and condensed around a specific project or topic.
L2	Time Memory	Entity-state records strongly tied to time	Carries an absolute timestamp and a state-transition chain (e.g. tracking how a user’s job status changes over time).
L3	Global Memory	The final persistent memory layer (user profile)	Long-term, rarely-changing traits distilled via offline computation (e.g. a user’s name, core tech stack, behavioral preferences).

---

3.3 Memory Extraction and Lifecycle Flow

CatMemory runs a dual-track lifecycle model: “real-time extraction + offline consolidation”:

 Agent interaction ends (turn_end)
       │
       ▼
 [1. Server-side sync]      ───► Sync this turn's session and context to the CatMemory server
       │
       ▼
 [2. Sliding-window detection] ───► Identify topics within the session, extract content in real time, generate L1 memory units
       │
       ▼
 [3. Conflict & linkage check] ───► Search existing L1 units across the store, automatically [update / mark stale / link]
       │
       ▼
 [4. Offline batch processing] ───► (background job) Aggregate related memory units into L2 Project/Time Memory
       │
       ▼
 [5. Dreaming consolidation]  ───► (offline "Dreaming" job) Periodically scan, prune low-value noise, and crystallize high-value memory into L3 Global Memory

---

3.4 Multi-Stage Retrieval

To balance speed and accuracy, CatMemory skips single-pass vector search in favor of a progressive, multi-stage retrieval chain:

 User query
       │
       ▼
 ┌───────────────────┐
 │ Stage 1: Coarse pass │ ───► Fast multi-path recall (BM25 + vector search), pulling the top candidates from L1–L3
 └───────────────────┘
       │
       ▼
 ┌───────────────────┐
 │ Stage 2: Fine rerank │ ───► Deep cross-encoding reranker, blending [semantic score + recency-decay score + base importance score]
 └───────────────────┘
       │
       ▼
 ┌───────────────────┐
 │ Stage 3: Agentic  │ ───► (optional Agentic Search) if the logic is complex, triggers tool calls for multi-step reasoning retrieval
 └───────────────────┘
       │
       ▼
 Injected at the top of the LLM prompt (a small handful of the highest-quality memories)

---

3.5 Logical Isolation Mechanism

CatMemory implements a flexible, multi-dimension multi-tenant architecture to keep data isolated and secure across contexts:

Isolation Dimension	Business Meaning	Notes / Use Case
User	User-level isolation	Hard multi-tenant isolation at the lowest level, keyed by employee ID — memory never crosses between users.
Source	Source-system isolation	Isolated by which client product the memory came from (e.g. different chat clients, desktop tools, IDE integrations).
Agent instance	Agent-instance isolation	Supports multi-agent architectures and dedicated sandboxes within a single client, preventing cross-agent contamination.
Type/channel	Scenario-channel isolation	Fine-grained isolation by conversation type (group chat, one-on-one, project channel).
Metadata	Custom dimension	Supports arbitrary custom key-value pairs (e.g. {"project_id": "X"}) for application-defined isolation.

---

3.6 Data Security

• Fully on-premises: the embedding model, reranking model, and LLM used throughout CatMemory all run on internally-hosted or self-fine-tuned foundation models.
• No external exposure: all external API dependencies are cut off, ensuring absolute data security for proprietary code, project business data, and sensitive merchant information during processing.

---

IV. Rollout Progress and Roadmap

Milestone	Progress	Notes
Memory server goes live	Gray-launched as a plugin for the primary chat client	Validated the multi-tenant and L1 fact-extraction pipeline
Packaged and shipped with the client’s gray release	Rolling out to full availability within about a week	Focused monitoring on extraction latency and vector-store QPS
Next iteration phase	Gray rollout to the desktop client	Kicked off the first round of offline “Dreaming” memory consolidation

---

V. Roadmap

Planned Capability	Technical Description and Goal	Expected Value
Distributed knowledge graph support	Introduce a graph database or lightweight graph-structured index, linking scattered memory units via “entity-edge-relationship” chains.	Breaks past the limits of pure vector retrieval, strengthening deep logical linkage across multiple spans and entities.
Multi-stage retrieval (agentic search)	Introduce autonomous agent judgment and multi-round retrieval, letting the retriever decide on its own whether to issue a follow-up query based on the prior round’s fragments.	Meaningfully improves retrieval precision for complex, ambiguous, or long-tail queries.
Visual memory-management console	Build a visual web front end giving developers and power users memory search, CRUD operations, and lineage views.	Makes it easier for people to perceive and intervene in an agent’s memory, improving human-AI trust.
Offline “Dreaming” consolidation	Establish a scheduled batch job that runs during idle hours (e.g. overnight), clustering offline, filtering high-frequency noise, and distilling knowledge.	Smoothly and safely converts short-term memory into a long-term profile, guarding against unbounded storage growth.
Systematic benchmarking	Build a regression-test pipeline against a standard long-context memory-evaluation benchmark.	Quantifies a baseline and drives continued iteration, targeting a very high, stable accuracy score.