{"id":"fb7990c9-a1e4-45d1-bf08-f8e4188badf4","slug":"mathews-tom-searchat","kind":"mcp","name":"searchat","summary":"Searchat - Semantic search for AI coding agent conversations","owner":"Mathews-Tom","repoUrl":"https://github.com/Mathews-Tom/searchat","license":"MIT","licenseTier":"permissive","trust":"community","stars":2,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-01-24T20:53:27.000Z","addedAt":"2026-10-21T16:44:14.786Z","logoUrl":"https://avatars.githubusercontent.com/u/9562152?u=f1304e4fc82287ba7b28c1782b5e08076d985a68&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["search","databases"],"tags":["coding-agents","developer-tools","duckdb","python","rag","semantic-search"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T21:51:42.308Z","sample":false,"url":"https://conduits.dev/tools/mathews-tom-searchat","language":"Python","transports":["stdio"],"install":{"pypi":"searchat"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Searchat\n\nSemantic search and RAG-powered Q&A for AI coding agent conversations. Find past solutions by meaning, not just keywords, and ask questions about your conversation history.\n\n## Project Lineage\n\nThis repository is inspired by and built on the original [Process-Point-Technologies-Corporation/searchat](https://github.com/Process-Point-Technologies-Corporation/searchat).\n\nThe original repository is also accompanied by and stems from the paper [Structured Distillation for Personalized Agent Memory: 11× Token Reduction with Retrieval Preservation](https://arxiv.org/html/2603.13017) (arXiv:2603.13017, March 13, 2026).\n\nCredit to the original repository and its authors for the foundation:\n\n- local-first semantic search for AI coding conversations\n- append-only indexing and live file watching\n- the initial FastAPI + web UI shape\n- the original Claude Code and Mistral Vibe support\n- the structured distillation and personalized agent memory framing described in the paper\n\nThis fork has since expanded substantially beyond that baseline. It now includes:\n\n- 8 agent connectors across local coding tools and chat clients\n- RAG chat, session chat, MCP integration, and embedded local LLM support\n- an expertise layer for extraction, validation, priming, and onboarding workflows\n- an L3 knowledge graph for lineage, contradiction detection, and resolution\n- bookmarks, saved queries, dashboards, exports, backups, snapshots, and analytics\n- a materially larger API, CLI, frontend, documentation, and test surface\n\n## Supported Agents\n\n| Agent        | Location                                                     | Format   |\n| ------------ | ------------------------------------------------------------ | -------- |\n| Claude Code  | `~/.claude/projects/**/*.jsonl`                              | JSONL    |\n| Mistral Vibe | `~/.vibe/logs/session/*.json`                                | JSON     |\n| OpenCode     | `~/.local/share/opencode/storage/session/*/*.json`           | JSON     |\n| OpenAI Codex | `~/.codex/sessions/**/rollout-*.jsonl`                       | JSONL    |\n| Gemini CLI   | `~/.gemini/tmp/<project_hash>/chats/*.json`                  | JSON     |\n| Continue     | `~/.continue/sessions/*.json`                                | JSON     |\n| Cursor       | `.../Cursor/User/.../*.vscdb`                                | SQLite   |\n| Aider        | `.aider.chat.history.md` (set `SEARCHAT_AIDER_PROJECT_DIRS`) | Markdown |\n\n## Features\n\n### Core Search\n\n- **Hybrid Search** — DuckDB FTS keyword + FAISS semantic vectors with RRF fusion\n- **Query Synonyms** — Automatic expansion of common search terms\n- **Cross-Encoder Re-ranking** — Optional re-ranking with cross-encoder models\n- **Multi-Agent** — Search across Claude Code, Mistral Vibe, and OpenCode sessions\n- **Tool Filters** — Filter results by specific agent (Claude, Vibe, or OpenCode)\n- **Autocomplete** — Smart search suggestions as you type\n- **Search History** — Persistent search history with LocalStorage\n\n### AI-Powered Features\n\n- **RAG Chat** — Ask questions about your conversation history with AI-powered answers\n- **Session Chat** — Multi-turn RAG conversations with session persistence (30-min TTL)\n- **Pattern Mining** — Extract recurring coding patterns from conversation archives\n- **Embedded LLM** — Run RAG chat locally with a GGUF model (llama-cpp-python)\n- **Semantic Highlights** — Optional LLM-generated highlight terms for search results\n- **Conversation Similarity** — Discover related conversations using semantic similarity\n- **Code Extraction** — Extract and view code snippets with syntax highlighting\n\n### Organization & Management\n\n- **Bookmarks** — Save and annotate favorite conversations\n- **Saved Queries** — Save reusable searches (query + filters + mode)\n- **Dashboards** — Build dashboards from saved queries (widgets + auto-refresh)\n- **Search Analytics** — Track search patterns and usage statistics\n- **Export** — Export conversations in JSON/Markdown/Text (optional PDF + Jupyter notebook)\n- **Bulk Export** — Export multiple conversations at once\n- **Agent Config Export** — Generate CLAUDE.md, copilot-instructions, or cursorrules from patterns\n- **Pagination** — Navigate large result sets efficiently\n\n### Data Safety & Performance\n\n- **Live Indexing** — Auto-indexes new/modified files (5min debounce)\n- **Append-Only** — Never deletes existing data, safe for long-term use\n- **Backups** — Create and restore backups from UI or API\n- **Snapshots** — Browse backups as read-only datasets (\"snapshot\" mode)\n- **Safe Shutdown** — Detects ongoing indexing, prevents data corruption\n- **DuckDB Storage** — Efficient Parquet-based storage with fast queries\n- **FAISS Vectors** — High-performance semantic search\n\n### User Experience\n\n- **Keyboard Shortcuts** — Power user navigation and commands\n- **Cross-Platform** — Windows, WSL, Linux, macOS\n- **Local-First** — All data stays on your machine\n- **Self-Search** — Agents can search their own history via API\n- **MCP Server** — Let MCP clients (Claude Desktop, Cursor, etc.) query your local index\n- **Terminal Resume** — Resume conversations directly in terminal\n\n## 📊 Interactive Documentation\n\nVisual architecture diagrams and flow charts:\n\n- [System Overview](docs/architecture.md) - Current end-to-end architecture across connectors, indexing, storage, retrieval, knowledge layers, and interfaces\n- [RAG Chat Pipeline](docs/infographics/rag-chat-pipeline.html) - AI-powered Q&A over conversations\n- [Backup & Restore](docs/infographics/backup-restore-flow.html) - Data safety and recovery procedures\n- [File Watching](docs/infographics/file-watching-indexing.html) - Live indexing and event handling\n- [Multi-Agent Connectors](docs/infographics/multi-agent-connectors.html) - Agent support architecture\n\nUse the HTML diagrams in a browser for interactive walkthroughs, and use `docs/architecture.md` for the canonical Mermaid system map. [See all infographics →](docs/infographics/README.md)\n\n## Quick Start\n\n### Install And Run (Standalone)\n\nInstall Searchat and build the initial index:\n\n```bash\npip install searchat\n\n# Optional: create ~/.searchat/config/settings.toml interactively\npython -m searchat.setup\n\n# Build the initial search index\nsearchat setup-index\n\n# Start the web server\nsearchat web\n```\n\nOpen http://localhost:8000 (searches ports up to 8010 if 8000 is busy; override with `SEARCHAT_PORT`).\n\n### Install From Source\n\n```bash\ngit clone https://github.com/Mathews-Tom/searchat.git\ncd searchat\npip install -e .\n\n# First-time setup: build search index\npython scripts/setup-index\n\n# Start web server\nsearchat web\n```\n\nOpen http://localhost:8000 (searches ports up to 8010 if 8000 is busy; override with `SEARCHAT_PORT`).\n\nThe setup script indexes all conversations from supported agents. On subsequent runs, the web server automatically indexes new conversations via live file watching.\n\n### Rebuilding The Index\n\nTwo different operations can be confused — know which one you need:\n\n- **`searchat rebuild-derived [--force]`** — always safe. Rebuilds the\n  keyword and vector search indexes from conversations already stored in\n  the local database. Never opens a source conversation file, so it's safe\n  to run at any time, including after a compaction or backup restore, or\n  with no source files present at all.\n- **`searchat reingest-sources [--force]`** — guarded and dangerous if\n  source files are incomplete. Re-scans source JSONL/session files and\n  replaces the existing index with what it finds there. If source files are\n  missing or incomplete, indexed conversations absent from those files are\n  permanently lost. Refuses with an error unless an existing index is\n  absent or `--force` is explicitly passed.\n\nFor routine maintenance (fixing a corrupted index, restoring after backup,\nrecovering from disk compaction) use `rebuild-derived`. Only use\n`reingest-sources --force` if you have personally verified every source\nfile is present and complete.\n\n## MCP Server (Claude Desktop, Cursor, ...)\n\n```bash\npip install \"searchat[mcp]\"\nsearchat mcp\n```\n\nSee `docs/mcp-setup.md` for client configuration.\n\nAvailable MCP tools:\n\n- `search_conversations`\n- `get_conversation`\n- `find_similar_conversations`\n- `ask_about_history`\n- `list_projects`\n- `get_statistics`\n- `extract_patterns`\n- `generate_agent_config`\n\n## Embedded LLM (Local GGUF)\n\nInstall the optional embedded dependency and download the default GGUF model:\n\n```bash\npip install \"searchat[embedded]\"\nsearchat download-model --activate\n```\n\nWhen `llm.default_provider = \"embedded\"`, the server will auto-download the default model if `embedded_model_path` is not set.\n\n## Enable Claude Self-Search\n\nAdd to `~/.claude/CLAUDE.md`:\n\n````markdown\n## Conversation History Search\n\nSearch past Claude Code conversations via local API (requires server running).\n\n**Search:**\n\n```bash\ncurl -s \"http://localhost:8000/api/search?q=QUERY&limit=5\" | jq '.results[] | {id: .conversation_id, title, snippet}'\n```\n````\n\n**Get full conversation:**\n\n```bash\ncurl -s \"http://localhost:8000/api/conversation/CONVERSATION_ID\" | jq '.messages[] | {role, content: .content[:500]}'\n```\n\n**Ask questions (RAG):**\n\n```bash\ncurl -X POST \"http://localhost:8000/api/chat\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"How did we implement authentication?\", \"model_provider\": \"openai\", \"model_name\": \"gpt-4.1-mini\"}'\n```\n\n**Ask questions (RAG, embedded/local):**\n\n```bash\ncurl -X POST \"http://localhost:8000/api/chat\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"How did we implement authentication?\", \"model_provider\": \"embedded\"}'\n```\n\n**When to use:**\n\n- User asks \"did we discuss X before\" or \"find that conversation about Y\"\n- Looking for previous solutions to similar problems\n- Checking how something was implemented in past sessions\n- Asking questions about past work (use RAG chat)\n\n**Start server:** `searchat web` from the searchat directory\n\n````\n\nSee `CLAUDE.example.md` for the full template.\n\n## Usage\n\n### Web UI\n\n```bash\nsearchat web\n````\n\nFeatures:\n\n- **Search Modes:** hybrid/semantic/keyword with autocomplete\n- **Filters:** project, date range, tool (agent), similarity search\n- **View Conversations:** Full message history with code extraction\n- **Bookmarks:** Save and annotate favorite conversations\n- **RAG Chat:** Ask questions about your conversation history\n- **Analytics:** View search patterns and statistics\n- **Saved Queries:** Save and re-run complex searches\n- **Dashboards:** Create dashboards and widgets from saved queries\n- **Export:** Download conversations in multiple formats\n- **Backups:** Create and restore backups (left sidebar)\n- **Snapshots:** Switch between active index and read-only backup snapshots\n- **Keyboard Shortcuts:** Press `?` to see all shortcuts\n- **Terminal Resume:** Resume conversations in terminal\n- **Helpful Tips:** Search tips + API integration sidebars\n\n### CLI\n\n```bash\nsearchat  # interactive mode\nsearchat web\nsearchat mcp\nsearchat setup-index [--force]\nsearchat rebuild-derived [--force]   # rebuild search indexes from already-indexed data — always safe\nsearchat reingest-sources [--force]  # re-scan source files and replace the index — guarded, see above\n\n# Download a default embedded GGUF model and update ~/.searchat/config/settings.toml\nsearchat download-model --activate\n\n# Build the initial index (first-time setup)\nsearchat-setup-index\n```\n\n### API\n\n#### Search & Discovery\n\n```bash\n# Search\ncurl \"http://localhost:8000/api/search?q=authentication&mode=hybrid&limit=10\"\n\n# Search with tool filter (claude, vibe, opencode)\ncurl \"http://localhost:8000/api/search?q=authentication&tool=claude&limit=10\"\n\n# Autocomplete suggestions\ncurl \"http://localhost:8000/api/search/suggestions?q=auth&limit=5\"\n\n# Find similar conversations\ncurl \"http://localhost:8000/api/conversation/{conversation_id}/similar?limit=5\"\n\n# Optional: request highlight terms for the UI (LLM)\ncurl \"http://localhost:8000/api/search?q=auth&mode=hybrid&highlight=true&highlight_provider=ollama\"\n\n# Search with pagination\ncurl \"http://localhost:8000/api/search?q=api&limit=20&offset=0\"\n```\n\n#### Conversations\n\n```bash\n# Get conversation\ncurl \"http://localhost:8000/api/conversation/{conversation_id}\"\n\n# List all conversations\ncurl \"http://localhost:8000/api/conversations/all?limit=50\"\n\n# Extract code snippets\ncurl \"http://localhost:8000/api/conversation/{conversation_id}/code\"\n\n# Code highlighting (Pygments)\ncurl -X POST \"http://localhost:8000/api/code/highlight\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"blocks\":[{\"code\":\"print(123)\",\"language\":\"python\",\"language_source\":\"fence\"}]}'\n\n# Conversation diff\ncurl \"http://localhost:8000/api/conversation/{conversation_id}/diff?target_id={other_conversation_id}\"\n\n# Export conversation\ncurl \"http://localhost:8000/api/conversation/{conversation_id}/export?format=markdown\"\n\n# Bulk export\ncurl -X POST \"http://localhost:8000/api/conversations/bulk-export\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"conversation_ids\": [\"id1\", \"id2\"], \"format\": \"json\"}'\n\n# Resume in terminal\ncurl -X POST \"http://localhost:8000/api/resume?conversation_id={id}\"\n```\n\n#### Bookmarks\n\n```bash\n# List bookmarks\ncurl \"http://localhost:8000/api/bookmarks\"\n\n# Add bookmark\ncurl -X POST \"http://localhost:8000/api/bookmarks\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"conversation_id\": \"abc123\", \"notes\": \"Important auth solution\"}'\n\n# Remove bookmark\ncurl -X DELETE \"http://localhost:8000/api/bookmarks/{conversation_id}\"\n```\n\n#### RAG Chat\n\n```bash\n# Ask question about conversation history\ncurl -X POST \"http://localhost:8000/api/chat\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"query\": \"How did we implement authentication?\",\n    \"model_provider\": \"openai\",\n    \"model_name\": \"gpt-4.1-mini\",\n    \"session_id\": \"optional-session-id\"\n  }'\n# Response includes X-Session-Id header for session tracking\n\n# Non-streaming RAG response with citations\ncurl -X POST \"http://localhost:8000/api/chat-rag\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\":\"Summarize how backups work\",\"model_provider\":\"ollama\",\"model_name\":\"ollama/gemma3\"}'\n\n# Streaming response (default)\ncurl -X POST \"http://localhost:8000/api/chat\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"Explain the API design\", \"model_provider\": \"ollama\", \"model_name\": \"llama3\"}' \\\n  --no-buffer\n```\n\n#### Analytics & Statistics\n\n```bash\n# Index statistics\ncurl \"http://localhost:8000/api/statistics\"\n\n# Search analytics\ncurl \"http://localhost:8000/api/stats/analytics/summary?days=7\"\n\n# Top queries / trends\ncurl \"http://localhost:8000/api/stats/analytics/top-queries?limit=10&days=30\"\ncurl \"http://localhost:8000/api/stats/analytics/trends?days=30\"\n\n# List projects\ncurl \"http://localhost:8000/api/projects\"\n\n# Project summary\ncurl \"http://localhost:8000/api/projects/summary\"\n```\n\n#### Saved Queries & Dashboards\n\n```bash\n# Saved queries\ncurl \"http://localhost:8000/api/queries\"\n\n# Dashboards\ncurl \"http://localhost:8000/api/dashboards\"\n```\n\n#### Tech Docs (Optional)\n\nRequires `export.enable_tech_docs=true`.\n\n```bash\ncurl -X POST \"http://localhost:8000/api/docs/summary\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"title\": \"Searchat Notes\",\n    \"format\": \"markdown\",\n    \"sections\": [\n      {\"name\": \"Backups\", \"query\": \"backup restore\", \"mode\": \"hybrid\", \"filters\": {\"project\": \"myapp\"}}\n    ]\n  }'\n```\n\n#### Snapshots (Read-only)\n\n```bash\n# Search within a backup snapshot\ncurl \"http://localhost:8000/api/search?q=auth&snapshot=backup_YYYYMMDD_HHMMSS\"\n```\n\n#### Indexing & Management\n\n```bash\n# Watcher status\ncurl \"http://localhost:8000/api/watcher/status\"\n\n# Index missing conversations (append-only)\ncurl -X POST \"http://localhost:8000/api/index_missing\"\n\n# Safe shutdown (checks for ongoing indexing)\ncurl -X POST \"http://localhost:8000/api/shutdown\"\n\n# Force shutdown (override safety check)\ncurl -X POST \"http://localhost:8000/api/shutdown?force=true\"\n```\n\n#### Backups\n\n```bash\n# Create full backup (plaintext)\ncurl -X POST \"http://localhost:8000/api/backup/create\"\n\n# Create full backup (encrypted; backups only)\n# Requires: SEARCHAT_BACKUP_KEY_B64 (base64-encoded 32 bytes)\ncurl -X POST \"http://localhost:8000/api/backup/create?encrypted=true\"\n\n# Create incremental backup (delta) based on a parent backup\ncurl -X POST \"http://localhost:8000/api/backup/incremental/create?parent=backup_YYYYMMDD_HHMMSS\"\n\n# Create encrypted incremental backup (parent must also be encrypted)\ncurl -X POST \"http://localhost:8000/api/backup/incremental/create?parent=backup_YYYYMMDD_HHMMSS&encrypted=true\"\n\n# List backups\ncurl \"http://localhost:8000/api/backup/list\"\n\n# Validate a backup chain and (optionally) verify hashes\ncurl \"http://localhost:8000/api/backup/validate/backup_YYYYMMDD_HHMMSS?verify_hashes=true\"\n\n# Inspect ancestry chain (base -> target)\ncurl \"http://localhost:8000/api/backup/chain/backup_YYYYMMDD_HHMMSS\"\n\n# Restore backup\ncurl -X POST \"http://localhost:8000/api/backup/restore?backup_name=backup_YYYYMMDD_HHMMSS\"\n\n# Delete backup\ncurl -X DELETE \"http://localhost:8000/api/backup/delete/backup_YYYYMMDD_HHMMSS\"\n```\n\nNotes:\n\n- Snapshot browsing (`?snapshot=...`) only supports snapshot-browsable backups (plaintext full backups). Incremental and encrypted backups are restore-only.\n- Incremental backups support chained deltas, with a max total chain length of 10.\n- Generate a new encryption key (example): `python -c \"import os,base64; print(base64.b64encode(os.urandom(32)).decode())\"`\n\n### Utilities\n\n```bash\n# Add missing conversations to index\npython scripts/index-missing\n\n# Initial setup (interactive, safe options)\npython scripts/setup-index\n\n# Convert Vibe plaintext history to searchable sessions\npython utils/vibe_converter.py\n```\n\n### As Library\n\n```python\nfrom searchat.core.search_engine import SearchEngine\nfrom searchat.config.settings import Config\n\nconfig = Config.load()\nengine = SearchEngine(config.paths.search_directory, config)\n\nresults = engine.search(\"python async\", mode=\"hybrid\")\nfor r in results.results[:5]:\n    print(f\"{r.title}: {r.score:.3f}\")\n```\n\n## Architecture\n\n**Code Organization:**\n\n- `src/searchat/api/` - FastAPI app with 14 modular routers (50+ endpoints)\n- `src/searchat/core/` - Core indexing and search logic\n- `src/searchat/services/` - Business services (chat, bookmarks, analytics, backup)\n- `src/searchat/web/` - Modular frontend (HTML + CSS modules + ES6 JS)\n- `tests/` - Comprehensive test suite (840+ tests)\n\n**Data Flow:**\n\n```plaintext\n~/.claude/projects/**/*.jsonl     (source conversations)\n~/.vibe/logs/session/*.json\n~/.local/share/opencode/.../*.json\n        │\n        ▼ index_append_only()\n        │\n~/.searchat/data/\n├── conversations/*.parquet       (DuckDB queryable)\n└── indices/\n    ├── embeddings.faiss          (semantic vectors)\n    ├── embeddings.metadata.parquet\n    └── index_metadata.json\n```\n\n**Search Flow:**\n\n1. Query → DuckDB FTS keyword search + FAISS semantic search\n2. Results merged via Reciprocal Rank Fusion\n3. Hybrid ranking returns best of both approaches\n4. Optional: Cross-encoder re-ranking of top results\n5. Query synonym expansion for keyword matching\n6. Optional: Find similar conversations via vector similarity\n\n**RAG Flow:**\n\n1. User question → Search for relevant conversations\n2. Top results used as context\n3. LLM generates answer with conversation references\n4. Streaming response to client\n\n**Live Watching:**\n\n- `watchdog` monitors conversation directories\n- New files → indexed immediately\n- Modified files → re-indexed after 5min debounce (configurable)\n- `index_append_only()` adds to existing index\n- Never deletes existing data\n\n**Documentation:**\n\n- `docs/features.md` - Complete feature list and descriptions\n- `docs/architecture.md` - System design and components\n- `docs/api-reference.md` - Complete API endpoint documentation\n- `docs/terminal-launching.md` - Platform-specific terminal launching\n\n## Configuration\n\nCreate `~/.searchat/config/settings.toml`:\n\n```toml\n[paths]\nsearch_directory = \"~/.searchat\"\nclaude_directory_windows = \"~/.claude/projects\"\nclaude_directory_wsl = \"//wsl$/Ubuntu/home/{username}/.claude/projects\"\n\n[indexing]\nbatch_size = 1000\nauto_index = true\nreindex_on_modification = true  # Re-index modified conversations\nmodification_debounce_minutes = 5  # Wait time before re-indexing\nenable_connectors = true\nenable_adaptive_indexing = true\n\n[search]\ndefault_mode = \"hybrid\"\nmax_results = 100\nsnippet_length = 200\n\n[embedding]\nmodel = \"all-MiniLM-L6-v2\"\nbatch_size = 32\ndevice = \"auto\"  # auto|cuda|mps|cpu\n\n[llm]\ndefault_provider = \"ollama\"\nopenai_model = \"gpt-4.1-mini\"\nollama_model = \"ollama/gemma3\"\n\n# Embedded (local GGUF via llama-cpp-python)\nembedded_model_path = \"\"\nembedded_n_ctx = 4096\nembedded_n_threads = 0\nembedded_auto_download = true\nembedded_default_preset = \"qwen2.5-coder-1.5b-instruct-q4_k_m\"\n\n[chat]\nenable_rag = true\nenable_citations = true\n\n[analytics]\nenabled = false\nretention_days = 30\n\n[export]\nenable_ipynb = false\nenable_pdf = false\nenable_tech_docs = false\n\n[dashboards]\nenabled = true\n\n[snapshots]\nenabled = true\n\n[performance]\nmemory_limit_mb = 3000\nquery_cache_size = 100\n\n[reranking]\nenabled = false\nmodel = \"cross-encoder/ms-marco-MiniLM-L-6-v2\"\ntop_k = 50\n\n[server]\ncors_origins = [\"http://localhost:8000\", \"http://127.0.0.1:8000\"]\n```\n\nOr use environment variables:\n\n```bash\nexport SEARCHAT_DATA_DIR=~/.searchat\nexport SEARCHAT_PORT=8000\nexport SEARCHAT_EMBEDDING_MODEL=all-MiniLM-L6-v2\nexport SEARCHAT_REINDEX_ON_MODIFICATION=true\nexport SEARCHAT_MODIFICATION_DEBOUNCE_MINUTES=5\nexport SEARCHAT_OPENCODE_DATA_DIR=~/.local/share/opencode\nexport OPENAI_API_KEY=sk-...  # For RAG chat\nexport OLLAMA_BASE_URL=http://localhost:11434\n```\n\n## Requirements\n\n- Python 3.10+\n- ~2-3GB RAM (embeddings model + FAISS index)\n- ~10MB disk per 1K conversations\n- Optional: OpenAI API key or Ollama for RAG chat\n\n### Dependencies\n\n| Package               | Purpose                       |\n| --------------------- | ----------------------------- |\n| sentence-transformers | Embeddings (all-MiniLM-L6-v2) |\n| faiss-cpu             | Vector similarity search      |\n| pyarrow               | Parquet storage               |\n| duckdb                | SQL queries on parquet        |\n| fastapi + uvicorn     | Web API                       |\n| watchdog              | File system monitoring        |\n| litellm               | Multi-provider LLM interface  |\n| rich                  | CLI formatting                |\n\n## Safety\n\n**Append-only indexing:** Never deletes existing data.\n\n```python\nindexer.index_append_only(file_paths)  # Safe: only adds new data\nindexer.index_all()                     # Blocked if index exists\nindexer.index_all(force=True)           # Explicit override required\n```\n\n**Safe shutdown:** Detects ongoing indexing operations.\n\n```bash\n# Check status, wait if indexing in progress\ncurl -X POST \"http://localhost:8000/api/shutdown\"\n\n# Override safety check (may corrupt data)\ncurl -X POST \"http://localhost:8000/api/shutdown?force=true\"\n```\n\n**Backups:** Create backups before risky operations.\n\n```bash\n# Automatic pre-restore backup before any restore operation\ncurl -X POST \"http://localhost:8000/api/backup/restore\" -d '{\"name\": \"backup_20260129_120000\"}'\n```\n\nProtects against:\n\n- Data loss from deleted/moved source files\n- Corrupted Parquet/FAISS files during indexing\n- Inconsistent metadata from interrupted operations\n\n## Performance\n\n| Metric            | Value                                              |\n| ----------------- | -------------------------------------------------- |\n| Search latency    | <100ms (hybrid), <50ms (semantic), <30ms (keyword) |\n| Filtered queries  | <20ms (DuckDB predicate pushdown)                  |\n| Index build       | ~60s per 100K conversations                        |\n| Embedding         | Batched (CPU: 0.1s/conv, GPU: 0.008s/conv)         |\n| Memory            | ~2-3GB                                             |\n| Startup           | <3s                                                |\n| RAG chat response | <2s (with OpenAI), <5s (with Ollama)               |\n| Code extraction   | <50ms per conversation                             |\n| Similarity search | <100ms (FAISS nearest neighbors)                   |\n\n## Testing\n\n```bash\npytest                          # Run the full test suite\npytest tests/api/              # Run API tests only\npytest -v                      # Verbose output\npytest -k test_search          # Run specific tests\npytest --cov=searchat          # Coverage report\npytest --cov-report=html       # HTML coverage report\nsearchat validate release      # Run the local pre-release validation matrix\n```\n\n**Test Coverage:**\n\n- 840+ tests (API, UI contract tests, unit tests, perf gates)\n- ~5,900 lines of test code\n- Comprehensive coverage of all features\n- API endpoint tests, unit tests, integration tests\n\nSee [docs/release.md](docs/release.md) for the Wave 6 release validation flow and packaging gate.\n\n## Troubleshooting\n\n**Port in use:**\n\n```bash\nSEARCHAT_PORT=8001 searchat-web\n```\n\n**No conversations found:**\n\n```bash\nls ~/.claude/projects/  # Verify conversations exist\n```\n\n**WSL not tracked:**\nConfigure `claude_directory_wsl` in `~/.searchat/config/settings.toml`:\n\n```toml\nclaude_directory_wsl = \"//wsl.localhost/Ubuntu/home/username/.claude/projects\"\n```\n\n**Missing conversations after setup:**\n\n```bash\npython scripts/index-missing  # Index files not yet in search index\n```\n\n**Slow on WSL:**\nRun from Windows Python or move repo to WSL filesystem (`~/projects/`).\n\n**Import errors:**\n\n```bash\npip install -e . --force-reinstall\n```\n\n**Empty environment variables override config:**\nRemove empty values from `~/.searchat/config/.env` or set proper values.\n\n**RAG chat not working:**\n\n```bash\n# For OpenAI\nexport OPENAI_API_KEY=sk-...\n\n# For Ollama (local)\nollama serve  # Start Ollama server\nexport OLLAMA_BASE_URL=http://localhost:11434\n```\n\n## Fork Enhancements\n\nCompared with the original upstream repository, this fork adds significant new capabilities:\n\n- **RAG Chat** - AI-powered Q&A over conversation history\n- **Session Chat** - Multi-turn RAG with session persistence\n- **Bookmarks System** - Save and organize favorite conversations\n- **Search Analytics** - Track and analyze search patterns\n- **Conversation Similarity** - Discover related conversations\n- **Code Extraction** - Extract code snippets with syntax highlighting\n- **Saved Queries** - Reusable searches with stored filters\n- **Dashboards** - Builder UI + widgets rendered from saved queries\n- **Snapshots** - Browse backups as read-only datasets\n- **Export Features** - JSON/Markdown/Text exports (optional PDF + Jupyter)\n- **Bulk Export** - Export multiple conversations at once\n- **Pagination** - Efficient navigation of large result sets\n- **Autocomplete** - Smart search suggestions\n- **Search History** - Persistent search history\n- **Keyboard Shortcuts** - Power user shortcuts\n- **Expanded Connector Coverage** - Support for OpenCode, Codex, Gemini CLI, Continue, Cursor, and Aider\n- **Tool Filtering** - Filter by specific agent across the broader connector set\n- **MCP Server** - Query local history from MCP clients\n- **Embedded LLM Support** - Run local GGUF-backed RAG without a hosted provider\n- **Expertise Store** - Extract, validate, search, and prime reusable engineering knowledge\n- **Knowledge Graph** - Track lineage, contradictions, and record relationships\n- **Pattern Mining** - Extract recurring patterns via LLM workflows\n- **Agent Config Export** - Generate agent config from patterns and expertise\n- **DuckDB FTS** - Replaced BM25 with DuckDB full-text search\n- **Cross-Encoder Re-ranking** - Optional result re-ranking\n- **Query Synonyms** - Automatic synonym expansion\n- **Operational Health Tooling** - Health endpoints, validation commands, CI checks, and release validation flows\n- **Modern Typing** - Python 3.12 type hints throughout\n- **CORS Hardening** - Configurable CORS origins (default localhost)\n\nSee `docs/features.md` for complete feature documentation.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n## License\n\nMIT\n"}