memkeeper-ingest

memkeeper-ingest

Part of the Memkeeper family.

Memkeeper is the open-source, local-first control plane that AI agents run on: durable memory, project context, coordinated task handoffs, and deny-by-default permissions, all deterministic and on your own machine.

Bring an existing document library into memkeeper for retrieval. memkeeper-ingest walks a folder, chunks each file in a structure-aware way, and feeds the chunks into memkeeper's isolated document store — without polluting your curated memories.

It is an optional add-on. memkeeper works without it; install this only if you want to point memkeeper at a notes/docs folder.

Status: pre-release (v0.1). Flags and behavior may change before 1.0.

ℹ️ Generated release mirror. This repo is published from a private development repo as releases. The main branch may be regenerated, so pin to tagged releases (or the release artifacts) rather than to arbitrary main commits — tagged releases are stable. See CONTRIBUTING.md for how to contribute; issues, security reports, and design feedback are the best paths today.

What it does (and doesn't)

• Write-side only. It walks, parses, chunks, and calls memkeeper's public ingest command. Retrieval stays in memkeeper (memkeeper document-search).
• No LLM required. Ingestion is faithful RAG: your documents are chunked and embedded as-is, not summarized. (Turning high-traffic chunks into distilled memories is a separate, opt-in, memkeeper-side step.)
• Isolated. Chunks land in a dedicated documents space and never receive the curated memory tier's supersession / dedup / graph / promotion treatment.
• Incremental. With --state, unchanged files are skipped on re-runs, so recurring syncs are cheap.
• Markdown first. v1 ingests .md (headings → breadcrumb context, fenced code kept intact, YAML front matter stripped). PDF/HTML/code are future work.

Requirements

• Rust toolchain (stable, via rustup) — cargo install builds the memkeeper-ingest binary. The crate is edition 2021, so Rust 1.56 or newer.
• The memkeeper binary on your PATH (or pass --binary /path/to/memkeeper). This add-on shells out to it; embeddings are computed by memkeeper, so chunk vectors always match the model used at query time.
• Only for promote LLM distillation: a running OpenAI-compatible endpoint. By default this is a local Ollama server at http://localhost:11434/v1 with the --llm-model you pass already pulled; or point --llm-base-url / MEMKEEPER_LLM_BASE_URL (and --llm-api-key / MEMKEEPER_LLM_API_KEY) at a remote provider. Plain ingest needs none of this — only promote in distill mode calls an LLM.

Install

# Install the binary onto your PATH:
cargo install --path .

# ...or just build it locally without touching ~/.cargo/bin
# (then run ./target/release/memkeeper-ingest):
cargo build --release

Usage

ingest — import a folder

# Initial import of a notes folder
memkeeper-ingest ingest --root ~/notes --state ~/.memkeeper/ingest-sync.json

# Re-run later: only changed files are re-ingested
memkeeper-ingest ingest --root ~/notes --state ~/.memkeeper/ingest-sync.json

# Preview chunking without writing
memkeeper-ingest ingest --root ~/notes --dry-run

Then search from memkeeper:

memkeeper document-search --json '{"query":"how did we decide on spaces vs silos"}'

Flag	Default	Description
--root <dir>	(required)	Folder to walk for documents.
--store <path>	memkeeper default	memkeeper store to ingest into.
--space <name>	documents	Target space (kept isolated from curated memory).
--binary <name>	memkeeper	memkeeper binary to invoke.
--ext <ext>	md	File extension to ingest.
--chunk-size <chars>	1500	Target chunk size.
--overlap <chars>	200	Overlap between chunks within a section.
--state <path>	(none)	Sync-state file for incremental re-ingest.
--dry-run	off	Parse and chunk but do not write.

promote — turn document chunks into memories

Promotion distills (or pins) document chunks into curated memkeeper memories that join the graph and proactive recall. By default it submits to the review queue; approve with memkeeper candidate-approve. Promoted memories keep a provenance link to their source chunk in source_ref_json.

# Remember a whole document (LLM-distilled into atomic memories), to the queue
memkeeper-ingest promote --doc ~/notes/decisions.md --llm-model qwen2.5

# Remember one chunk verbatim (no LLM), straight to memory
memkeeper-ingest promote --chunk src_... --verbatim --direct

# Usage-driven: promote chunks that earned retrieval traffic
memkeeper-ingest promote --auto --min-distinct-queries 3 --llm-model qwen2.5

Selection (one required): --doc <path> | --chunk <source_episode_id> | --auto. Mode: LLM distillation (default) or --verbatim (no-LLM pin). Routing: review queue (default) or --direct. The distiller speaks the OpenAI-compatible chat API and defaults to a local Ollama endpoint (MEMKEEPER_LLM_BASE_URL, default http://localhost:11434/v1); set --llm-model / MEMKEEPER_LLM_MODEL (required unless --verbatim) and, for off-box providers, --llm-api-key / MEMKEEPER_LLM_API_KEY. The LLM is used only for distillation; ingest and verbatim promotion never call it.

Both subcommands print a JSON summary and exit non-zero on any failure.

Benchmark

bench/doc_rag_bench.py measures retrieval quality (hit-rate@k, MRR) over a doc corpus to tune chunking. See bench/README.md. The bundled corpus is a template; plug in your own held-out set for real numbers.

Contributing

Issues and pull requests are welcome. Contributions require signing the project Contributor License Agreement — the CLA bot prompts you on your first pull request; you keep the copyright to your contributions. See CONTRIBUTING.md.

License

Dual-licensed under either of MIT or Apache-2.0 at your option.