lucimem

Lucimem

---

Why Lucimem?

AI agents forget everything between sessions.

An agent can spend hours working on a project — discovering facts, making decisions, learning patterns — then shut down and lose it all by tomorrow. Existing memory solutions make this worse by hiding behind opaque databases that you can't read, edit, or debug.

Lucimem is different. It stores memory as plain markdown files with YAML frontmatter. You can read it with a text editor, search it with grep, version it with git, and open it in Obsidian for graph view and backlinks. The agent's memory is yours to inspect and control.

• File over database — every memory is a markdown file, not a hidden row in SQLite
• Obsidian native — [[wikilinks]], graph view, backlinks — all work out of the box
• Unix-friendly — grep, awk, jq, git log — your memory speaks your tools
• Transparent retrieval — you always see what gets stored and how it gets found

---

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                            LUCIMEM ENGINE                               │
│                                                                         │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │                     CLI / MCP / Python API                        │  │
│  │              (lucimem cli  │  mcp server  │  LucimemClient)       │  │
│  └───────────────────────────────┬───────────────────────────────────┘  │
│                                  │                                      │
│  ┌───────────────────────────────▼───────────────────────────────────┐  │
│  │                      Consolidation Engine                         │  │
│  │                                                                   │  │
│  │   ┌──────────┐  promote  ┌──────────┐  promote  ┌──────────┐     │  │
│  │   │ Working  ├──────────►│ Episodic ├──────────►│ Semantic │     │  │
│  │   │ (seconds)│           │(sessions)│           │(knowledge)│     │  │
│  │   └──────────┘           └──────────┘           └────┬─────┘     │  │
│  │        ▲                       ▲                    │           │  │
│  │        │                       │               ┌────▼─────┐     │  │
│  │        └─────── decay ─────────┴──────────────►│Procedure │     │  │
│  │                                                │  (how-to)│     │  │
│  │                                                └──────────┘     │  │
│  └───────────────────────────────────┬─────────────────────────────┘  │
│                                      │                                │
│  ┌───────────────────────────────▼───────────────────────────────────┐  │
│  │                      Hybrid Retriever                             │  │
│  │                                                                   │  │
│  │   ┌──────────┐     ┌──────────────┐     ┌──────────────────┐     │  │
│  │   │  BM25      │     │ Embeddings   │     │ Knowledge Graph  │     │  │
│  │   │  (lexical) │     │ (semantic)   │     │  (relations)     │     │  │
│  │   └─────┬─────┘     └──────┬───────┘     └────────┬─────────┘     │  │
│  │         │                  │                       │              │  │
│  │         └──────────────────┼───────────────────────┘              │  │
│  │                            │                                      │  │
│  │                    ┌───────▼────────┐                              │  │
│  │                    │  Hybrid Score  │◄─ TTL decay                  │  │
│  │                    │  & Ranking     │◄─ strength tracking          │  │
│  │                    └────────────────┘                              │  │
│  └───────────────────────────────┬───────────────────────────────────┘  │
│                                  │                                      │
│  ┌───────────────────────────────▼───────────────────────────────────┐  │
│  │                        Storage Layer                              │  │
│  │                                                                   │  │
│  │   ┌────────────────────┐  ┌─────────┐  ┌──────────────────────┐   │  │
│  │   │  Markdown Files    │  │ SQLite  │  │  Embedding Indexes   │   │  │
│  │   │  (yaml frontmatter)│  │(meta/   │  │  (npz / on-disk)     │   │  │
│  │   │  [[wikilinks]])    │  │graph/obs)│  │                      │   │  │
│  │   └────────────────────┘  └─────────┘  └──────────────────────┘   │  │
│  └───────────────────────────────────────────────────────────────────┘  │
│                                    ▲                                    │
│  ┌─────────────────────────────────┴─────────────────────────────────┐  │
│  │                      Obsidian Sync                                │  │
│  │              (markdown + wikilinks + graph view)                  │  │
│  └───────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

---

The 4-Tier Memory System

Lucimem models memory after cognitive science — not everything deserves to last forever. Memories flow through four tiers, promoted by relevance and decayed by neglect.

Tier	Lifespan	Purpose	Example
Working	Seconds–minutes	Immediate context for the current task	"The user just asked about the deployment config"
Episodic	Session-level	Events and decisions from a session	"Chose PostgreSQL over MySQL because of JSONB support"
Semantic	Persistent	Generalized facts and knowledge	"The project uses FastAPI with SQLAlchemy"
Procedural	Highly persistent	Skills, patterns, and how-to knowledge	"Always run migrations before deploying to staging"

How consolidation works:

• Memories start in Working memory as raw observations
• Periodic consolidation promotes relevant items to Episodic, then Semantic
• Frequently reinforced Semantic memories become Procedural (actionable knowledge)
• Low-strength memories decay and auto-forget — preventing bloat
• Contradiction detection flags conflicting memories for review

---

Hybrid Retrieval

Lucimem combines three retrieval strategies into a single scored result set:

Retriever	What it finds	Strength
BM25	Lexical keyword matches	Exact terminology, filenames, IDs
Embeddings	Semantic similarity	Conceptual matches, paraphrases
Knowledge Graph	Related entities	Connected concepts, multi-hop reasoning

Results are fused with a hybrid scoring function that weights each retriever and applies TTL decay — fresher and stronger memories rank higher.

final_score = w₁·bm25 + w₂·embedding + w₃·graph − λ·decay(t)

---

Obsidian Compatibility

Open your lucimem vault in Obsidian and everything just works:

• [[wikilinks]] — memories link to each other natively
• Graph view — visualize how concepts connect
• YAML frontmatter — structured metadata on every file
• Backlinks — see which memories reference a concept
• Tags — #tag syntax for categorization

---
title: FastAPI Deployment Guide
type: procedural
strength: 0.85
created: 2026-04-01
tags: [deployment, fastapi, staging]
---

The staging environment requires running `alembic upgrade head` before deploying the FastAPI app to staging. Related to [[Database Migrations]] and [[Staging Environment]].

---

Quick Start

# Install
pip install lucimem

# Initialize a memory vault
lucimem init ~/my-agent-memory

# Ingest a fact
lucimem ingest "The project uses FastAPI with SQLAlchemy on PostgreSQL"

# Search your memory
lucimem search "database setup"

# Check status
lucimem status

# Sync to Obsidian
lucimem sync --vault ~/MyObsidianVault

That's it. Your agent now has persistent memory that lives in plain markdown files.

---

Features

Feature	Description
File-based storage	All memories stored as markdown with YAML frontmatter
Obsidian native	Wikilinks, graph view, backlinks work out of the box
4-tier consolidation	Working → Episodic → Semantic → Procedural lifecycle
Hybrid retrieval	BM25 + embeddings + knowledge graph fusion
Strength tracking	Memories gain/lose strength based on use and recency
Auto-forgetting	Low-strength memories decay automatically
Contradiction detection	Identifies and flags conflicting information
MCP server	41+ tools for agent integration
CLI interface	Full command-line control with interactive TUI (textual)
Python API	LucimemClient for programmatic access
Git-friendly	Track memory changes with version control
Unix pipes	grep, jq, awk — your memory is just text

---

How It Compares

	lucimem	agentmemory	Flat files
Storage	Markdown files + SQLite	ChromaDB vector store	Plain text files
Transparency	✨ Read/edit any file	Opaque database	Manual only
Obsidian support	Native ([[wikilinks]], graph)	No	Partial (manual)
Retrieval	BM25 + embeddings + graph	Vector similarity only	None (manual)
Consolidation	4-tier lifecycle with decay	None	None
Auto-forgetting	✅ Strength-based decay	❌	N/A
CLI	Full CLI with TUI	Limited	None
MCP tools	41+ tools	41+ tools	None
Version control	Git tracks all memory	Requires export	Manual
Contradiction detection	✅	❌	❌
Grep/jq/awk	✅ It's all text	❌	✅

---

Benchmarks

Benchmarks are coming soon. We're building a comprehensive evaluation suite covering:

• Retrieval accuracy (nDCG, recall@k) across hybrid retrieval modes
• Consolidation quality (promotion accuracy, decay curves)
• End-to-end agent task performance with/without memory
• Obsidian sync performance on large vaults
• MCP tool latency under concurrent load

Benchmark          | Config           | Result    | Baseline (none)
-------------------|------------------|-----------|----------------
Retrieval nDCG@10  | BM25 only        | —         |
Retrieval nDCG@10  | Hybrid           | —         |
RAG Task Accuracy  | With lucimem     | —         |
RAG Task Accuracy  | Without memory   | —         |
Sync latency       | 10k files        | —         |
MCP latency (p50)  | 41 tools         | —         |

---

Installation

# From PyPI (recommended)
pip install lucimem

# With embeddings support (recommended)
pip install lucimem[embeddings]

# From source
git clone https://github.com/sriram369/lucimem.git
cd lucimem
pip install -e ".[embeddings]"

Requirements: Python 3.11 or higher.

---

MCP Integration

Lucimem ships with a fully-featured MCP (Model Context Protocol) server that exposes the entire memory system as tools to any MCP-compatible agent host (Claude Desktop, Cursor, Windsurf, etc.).

Configuration

Add to your MCP config (claude_desktop_config.json or equivalent):

{
  "mcpServers": {
    "lucimem": {
      "command": "lucimem",
      "args": ["mcp", "--vault", "/path/to/your/memory"]
    }
  }
}

Available Tool Categories

Category	Count	Description
Core	12	remember, recall, search, forget, summarize
Observation	8	Tool usage capture, error logging, session events
Fact	6	Extract, store, and query concrete facts
Concept	6	Track abstract patterns and themes
Narrative	4	Generate coherent session summaries
Procedure	3	Store and retrieve how-to knowledge
Graph	4	Knowledge graph queries and BFS traversal
Sync	3	Obsidian vault sync operations
Maintenance	5	Consolidation trigger, status, cleanup

All 41+ tools are documented with descriptions and schemas for automatic function calling.

---

API Reference

Python API

from lucimem import LucimemClient

# Initialize client on a local vault
client = LucimemClient(db_path="~/.my-memory.db")

# Ingest memories
client.ingest("The API server runs on port 8000", tags=["api", "config"])
client.ingest("PostgreSQL requires JSONB for flexible schemas", category="fact")

# Search memories with hybrid retrieval (BM25 + graph + embeddings)
results = client.search("database configuration")

for r in results:
    print(f"[{r.score:.3f}] {r.content}")

# Get stats
stats = client.get_stats()
print(f"{stats['total_items']} memories, {stats['entities']} entities")

# Consolidate (4-tier lifecycle)
client.consolidate()

# Knowledge graph
entities = client.graph.get_top_entities(10)
related = client.graph.get_related_entities("postgresql")

All memories persist on disk as SQLite + markdown. Close the process, reopen — everything is there.

CLI Commands

lucimem init       Initialize a new memory vault
lucimem ingest     Add a memory entry
lucimem search     Search memories (full-text, semantic, hybrid)
lucimem status     Show vault statistics and tier summary
lucimem list       List memories with optional filters
lucimem get        Show a specific memory
lucimem forget     Remove a memory entry
lucimem consolidate Run the consolidation engine
lucimem sync       Sync to Obsidian vault
lucimem serve      Start the MCP server
lucimem ui         Launch the interactive TUI
lucimem export     Export vault to JSON/CSV
lucimem import     Import memories from JSON/CSV

CLI Options

lucimem search <query> [OPTIONS]
  --mode    STRATEGIES   bm25 | embeddings | graph | hybrid (default)
  --tier    TIER_NAME    working | episodic | semantic | procedural
  --top-k   INT          Number of results (default: 10)
  --json                 Output as JSON
  --explain              Show scoring breakdown

---

Package Structure

lucimem/
├── pyproject.toml          # Package config
├── README.md               # Documentation
├── src/lucimem/
│   ├── __init__.py
│   ├── cli.py              # CLI & entry points
│   ├── store.py            # SQLite metadata layer
│   ├── indexer.py          # BM25 index + embeddings
│   ├── retriever.py        # Hybrid search
│   ├── graph.py            # Knowledge graph
│   ├── consolidate.py      # 4-tier consolidation
│   ├── obsidian/
│   │   ├── sync.py         # Markdown sync
│   │   └── wikilinks.py    # Wikilink handling
│   ├── mcp_server.py       # MCP server
│   └── hooks/
│       ├── base.py         # Hook interface
│       └── capture.py      # Auto-capture hooks
├── tests/
└── examples/

---

License

MIT — see LICENSE.

---

Links

• GitHub — Source code & issues
• PyPI — Package releases
• Obsidian — Recommended vault viewer