Async memory tiers for long-running LLM agents
Flat vector stores treat every memory equally. Tiered memory with promotion rules keeps hot context fast and archival recall bounded.
LLM · Agents · Python
Agents that run across multiple sessions need memory beyond the context window. The default approach — dump everything into a vector store and retrieve top-k on each turn — works until it doesn't.
Why flat memory fails
Three problems emerge as agents accumulate memories:
- Latency scales with corpus size. Even with HNSW indexes, hybrid retrieval over thousands of chunks adds milliseconds per turn. In tight agent loops, that compounds.
- Recency gets diluted. A fact from five minutes ago competes equally with a fact from five weeks ago. Recency-weighted scoring helps but still scans the full index.
- Working context bloats. Agents re-retrieve stable facts every turn instead of keeping hot context in-process.
A tiered model
Sawtooth Memory uses three tiers modeled after human recall patterns:
| Tier | Scope | Storage | Promotion trigger |
|---|---|---|---|
| Working | Current turn context | In-process | Access count ≥ 2 |
| Session | Current agent run | Fast local store | Access count ≥ 4 |
| Long-term | Cross-session | Pluggable backend | Manual or policy |
New memories enter the working tier. On repeated access, they promote upward. Each tier has a fixed capacity with LRU eviction — oldest accessed memories get dropped first.
Async-first by design
Memory operations must not block the agent event loop. Every store, recall, and promote call is async. Backends (in-memory, Redis, SQLite) implement the same interface, so development stays zero-config and production can swap in Redis without changing agent code.
The API surface stays minimal: store, recall, promote, evict. Four methods. Agents don't need to understand tier internals.
Predictable latency bounds
Working tier recall is sub-millisecond — it's an in-process dict lookup. Session tier adds a local I/O hop. Long-term tier hits the pluggable backend.
The key property: hot context never pays the cost of scanning the full long-term corpus. Recall queries check working first, then session, then long-term — short-circuiting as soon as enough results are found.
Pluggable backends without ceremony
Requiring Redis for development kills adoption. The default in-memory backend handles local agent development. When you need persistence, swap the backend config — the agent code doesn't change.
This is the same principle as SQLite in development, Postgres in production — but for agent memory instead of relational data.
What I'd measure before shipping
- Promotion latency at each tier boundary
- Recall hit rate per tier (what percentage of queries resolve without hitting long-term?)
- Eviction impact on answer quality over long sessions
Memory layers are infrastructure. If you can't measure them, you can't trust them in production.