Problem
Standard LLM memory systems (like LangChain's ConversationSummaryMemory) process conversation history sequentially on the main thread — the application freezes for 5–10 seconds while an LLM generates a new summary on every turn. These summaries suffer from the "Lost in the Middle" effect, frequently deleting UUIDs, names, and rules to save tokens.
Approach
- — Designed a four-layer memory stack (L0 system, L1 working, L1.5 entity ledger, L2 archive) stitched by ContextManager at prompt-build time.
- — Offloaded summarization to an async background worker — messages append to L1 instantly and control returns in milliseconds.
- — Extracted critical facts into an immutable entity ledger via deterministic regex NER before LLM compression runs.
- — Built dual compression backends (Ollama local, Cloud with OpenAI/Anthropic/Gemini) with turn-based batching and graceful hard-limit truncation.
- — Shipped native integrations for LangGraph (ToolMessage sanitization), LangChain LCEL, and a sync wrapper for Flask/Django.
Outcome
- 11.3× faster main-thread execution on 20-message conversations (5.7s vs 64.15s, RTX 5060 / phi4-mini).
- 100% UUID and fact retention via L1.5 entity ledger vs variable recall with summary-only memory.
- 10% lower final prompt token cost (454 vs 506 tokens) while preserving more context.
- Published to PyPI with Redis and Postgres distributed storage adapters for horizontal scaling.
Metrics
- Main thread
- 11.3×
- Faster than standard summary memory (20-msg benchmark)
- Fact recall
- 100%
- UUIDs retained via L1.5 entity ledger
- Token cost
- -10%
- 454 vs 506 tokens in final prompt
- Ingestion
- Instant
- Non-blocking write, async compression
Architecture
Trade-offs
Compression
Chosen
Async background worker
Alternative
Inline LLM summarization on main thread
Async compression keeps the main thread responsive. Inline summarization blocks the user for 5–10 seconds per turn.
Fact retention
Chosen
Entity ledger with deterministic NER
Alternative
LLM-only summarization
Regex extraction guarantees UUIDs and IDs survive compression. Summarization alone hallucinates and drops critical identifiers.
Storage default
Chosen
In-memory with pluggable Redis/Postgres
Alternative
Mandatory distributed store
Zero-config for development. Redis and Postgres adapters enable multi-container deployments without API changes.
Live demo
Loading demo…
Next project
0%