{"id":"b10148a0-606c-40ce-b172-095c4549fc07","slug":"ingestlib","kind":"mcp","name":"ingestlib","summary":"Self-hosted document intelligence for RAG — parse, classify, split, and extract from PDF/DOCX/PPTX and images: searchable, cited, retrieval-ready chunks and schema-driven extraction","owner":"LangModule","repoUrl":"https://github.com/LangModule/ingestlib","license":"MIT","licenseTier":"permissive","trust":"community","stars":5,"match":null,"fewStars":false,"format":"mcp","createdAt":"2026-06-02T23:56:51.000Z","addedAt":"2026-10-21T17:28:11.056Z","logoUrl":"https://avatars.githubusercontent.com/u/261691643?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["files-documents","search","databases"],"tags":["amazon-nova-lite","document-ingestion","docx","embeddings","hybrid-search","jinaai","ocr","paddleocr"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T17:32:32.443Z","sample":false,"url":"https://conduits.dev/tools/ingestlib","language":"Python","transports":["stdio"],"install":{"pypi":"ingestlib"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# ingestlib\n\n![ingestlib — self-hosted document intelligence for RAG](assets/cover.png)\n\nSelf-hosted document intelligence for RAG pipelines. One library that takes a\nraw document — PDF, DOCX, PPTX, or a PNG/JPEG/WebP image — and produces\nsearchable, **cited**, retrieval-ready chunks, plus schema-driven\nextraction with verified citations: the territory of LlamaParse /\nReducto / Unstructured.io, running on your own stack.\n\n```python\nfrom ingestlib.services import ingest, retrieve\n\ningest(\"finance-10k.pdf\")            # parse → classify → split → embed → vector store\nresult = retrieve(\"what were the total revenues?\")\nprint(result.context)                # ranked chunks, each citing doc · page · section\n```\n\n**Documentation: [langmodule.github.io/ingestlib](https://langmodule.github.io/ingestlib/)** —\nguides for every stage, the full configuration reference, and the API docs.\n\n## What it does\n\n| Stage | What you get |\n|---|---|\n| **Parse** | Layout-aware markdown per page: tables as HTML (merged cells intact), formulas as LaTeX, **charts converted to data tables** (estimated values marked `~`, printed callouts and growth labels captured), figures extracted as PNG crops with captions and AI descriptions — every block traceable to a bounding box on the page |\n| **Classify** | Document-type label (`invoice`, `research_paper`, …) — open-ended, or constrained to your rules (per call or preset in `rules.yaml`, with page targeting) — confidence and ranked alternatives included. Works standalone with **no OCR** |\n| **Split** | Sections (pages grouped by role: `methods`, `results`, … — LLM-discovered, or **your own categories** via rules) containing **natural chunks** — boundaries follow the content, tables never split, each chunk carries a `[category › section › heading]` breadcrumb in its `embedding_text` |\n| **Extract** | **Your Pydantic schema, filled from the document** — one instance (`mode=\"one\"`) or every instance in a batch (`mode=\"many\"`, e.g. all receipts in a scanned expense bundle). Every field carries **verified provenance**: page + region citations checked against the parse, values grounded in the cited source text, and honest confidence — uncited or ungrounded answers are capped, hallucinated citations dropped |\n| **Ingest** | The whole pipeline in one call — every stage's queryable output recorded in the internal **registry** (Postgres), its bytes (source, page images, figure crops, whole-doc markdown) in the artifact store (S3, a self-hosted MinIO, or a local folder), vectors upserted, deduplicated by content checksum |\n| **Retrieve** | Question → **hybrid search** (dense embeddings + lexical sparse, merged) → **rerank** (Jina by default; Amazon Rerank or none via `reranker:` in config.yaml) → hits with scores and citations, plus a prompt-ready context block |\n| **Query databases** | The same `retrieve()` call also answers from your **SQL databases** — natural language → read-only generated SQL behind a permission boundary (read-only role + statement allowlist + LIMIT + timeout), with **verified-query** overrides for answers that must be exact. Postgres, MySQL, SQLite, DuckDB, Snowflake — merged with document results |\n\nEngines: **PaddleOCR-VL-1.6** (0.9B VLM, runs on your GPU) for layout + recognition,\n**Amazon Nova 2 Lite** for judgment (chart reading, review, classification,\nchunk boundaries), **Nova multimodal embeddings**, **eight vector stores**\n(Pinecone, Qdrant, SQLite, Postgres/pgvector, MongoDB, Milvus, OpenSearch,\nWeaviate — all hybrid dense + sparse), **S3, a self-hosted MinIO, or a local\nfolder** for artifacts (`artifact_store: s3 | local`; `s3.endpoint_url` points\nS3 at a self-hosted MinIO — object storage with no AWS). ~$0.002/page in LLM spend. An\n**OpenAI backend** (GPT-5 vision-capable chat + text-embedding-3) ships\nalongside Bedrock — flip `llm_provider: openai` / `embedding_provider:\nopenai` to run the whole pipeline on it instead — or `ollama` to keep\nevery LLM call on your own machine. See below.\n\n## Quickstart\n\n### 1. Requirements\n\n- Python 3.12+ and [uv](https://github.com/astral-sh/uv)\n- **AWS account** with Bedrock access (`us-east-1`): Nova 2 Lite + Nova 2\n  multimodal embeddings — the default provider; the OpenAI backend or a\n  local Ollama can run the whole pipeline instead (see below)\n- **Vector database** — none at all by default: the sqlite connector\n  stores vectors in a local file. Or a Pinecone account (serverless,\n  free tier works), a Qdrant server (local docker or Qdrant Cloud), a Postgres\n  with pgvector (RDS/Supabase/Neon or self-hosted), a MongoDB with search\n  (Atlas any tier or 8.2+ self-managed), a Milvus (local docker or Zilliz\n  Cloud), an OpenSearch (Amazon domain or local docker), a Weaviate (local\n  docker or Weaviate Cloud) — each just one connection URL\n- **Internal registry (Postgres)** — ingestlib's own metadata store, required for\n  the **corpus path** (`ingest`/`retrieve`/lifecycle). Bring it up with the bundled\n  compose file: `docker compose -f infra/docker-compose.yml --profile registry up -d`,\n  then `ingestlib registry init`. Standalone operations (`parse`/`classify`/`split`/\n  `extract` on a single document) need no registry\n- **Jina AI account** for reranking (free tier: 100 RPM) — the default; or set\n  `reranker: aws` (Amazon Rerank, same AWS credentials) or `reranker: none`\n  in config.yaml and skip Jina entirely\n\n### 2. Install\n\n```bash\nuv add ingestlib               # or: pip install ingestlib\n```\n\nThe core install covers the full default stack (including sqlite vectors).\nServer-backed vector stores are extras — add the one you'll use:\n\n```bash\nuv add \"ingestlib[qdrant]\"     # pinecone · qdrant · pgvector · mongodb\n                               # · milvus · opensearch · weaviate · all\n```\n\nOr work from source:\n\n```bash\ngit clone https://github.com/LangModule/ingestlib.git\ncd ingestlib\nuv sync\n```\n\nSystem dependency — LibreOffice (DOCX/PPTX → PDF conversion):\n\n```bash\nbrew install --cask libreoffice          # macOS (binary is `soffice`)\nsudo apt install libreoffice-core libreoffice-writer libreoffice-impress   # Linux\n```\n\nOptional — PostgreSQL client tools (`pg_dump`/`pg_restore`), only for\n`ingestlib registry backup`/`restore`:\n\n```bash\nbrew install libpq                       # macOS\nsudo apt install postgresql-client       # Linux\n```\n\n### 3. Start the OCR inference server\n\nParse runs PaddleOCR-VL-1.6 behind an inference server. First launch downloads\n~1.8 GB of weights; later launches load from cache in seconds.\n\n```bash\n# Apple Silicon (Metal GPU)\nuv run python -m mlx_vlm.server --port 8111 --model PaddlePaddle/PaddleOCR-VL-1.6\n\n# NVIDIA (then set paddle_vl.backend: vllm-server in config.yaml)\nvllm serve PaddlePaddle/PaddleOCR-VL-1.6 --trust-remote-code --port 8111\n```\n\nThe layout model (PP-DocLayoutV3, ~126 MB) auto-downloads on the first parse.\n\n### 4. Configure\n\n`ingestlib init` scaffolds the setup files into the current directory:\n\n```bash\nuv run ingestlib init            # default stack: Bedrock + sqlite + Jina → config.yaml + .env\nuv run ingestlib init --local    # zero-cloud: Ollama + sqlite + local artifacts — no keys at all\n```\n\nFor the default stack, add Bedrock-enabled credentials:\n\n```bash\naws configure --profile your-aws-profile\n```\n\nEdit `config.yaml`: pick your providers, vector store, reranker, and\nartifact store — everything else has working defaults (`vector_store:\nsqlite` needs no server and no keys) — and fill `.env` with the keys your\nchoices need (Jina for the default reranker; `--local` needs none). The `aws` section is required only\nwhile a choice uses AWS (the default bedrock provider, s3 artifacts on AWS, the\naws reranker, an Amazon OpenSearch domain — a self-hosted MinIO `s3` store uses\nstatic keys, not a profile) — delete it otherwise and the\nconfig loader will tell you if something still needs it. **The S3 bucket\n(default `ingestlib-{account_id}`) and the vector indexes/collections are\ncreated automatically on first use** — no manual setup. Prefer no cloud\nstorage at all? `artifact_store: local` keeps every source file, page render,\nand figure crop in a plain folder beside your config.yaml — browsable in a file\nmanager, and moving the artifact store between backends is a plain copy.\n\nConfig is discovered at call time, never at import: `INGESTLIB_CONFIG=/path/to/config.yaml`\nwins, otherwise the working directory and its parents are searched — so\ninstalled usage works the same as running inside this repo.\n\n### 5. Verify\n\n```bash\nuv run ingestlib doctor\n```\n\nDoctor checks every configured choice with real calls — config discovery,\nLibreOffice, the OCR server, an LLM ping, an embedding (with its dimension),\nthe reranker, the artifact store, the vector store, and the registry (reachable\nand migrated). Every failed line\nprints the one-sentence fix (wrong AWS profile → your available profiles,\nmissing key → where to get one, model not pulled → the exact `ollama pull`).\n\n### 6. Run\n\n```python\nfrom ingestlib.services import ingest, retrieve\n\nr = ingest(\"report.pdf\")\nprint(r.status, r.category, r.chunks, r.durations)\n\nres = retrieve(\"what does the report conclude?\", top_k=5)\nfor hit in res.hits:\n    print(hit.rerank_score, hit.citation, hit.chunk.heading)\n```\n\n## Manage a corpus\n\nReal corpora change. Re-ingesting an edited file **replaces** the old version\n(its vectors and artifacts are deleted — no stale chunks), a rename is a cheap\n**move**, and `sync()` reconciles a whole folder in one call:\n\n```python\nfrom ingestlib.services import ingest, sync, remove, reindex\n\ningest(\"report.pdf\")                       # edited file → status=\"replaced\"\nsync(\"corpus/\", prune=True)                # add new, replace changed, drop deleted\nremove(\"old.pdf\")                          # erase one doc from all three stores\nreindex()                                  # rebuild the vector store from the registry\n```\n\nThe same verbs are on the CLI — a corpus is managed from the shell, no Python\nneeded:\n\n```bash\ningestlib ingest report.pdf docs/          # files or folders\ningestlib sync corpus/ --prune --dry-run   # preview, then drop --dry-run\ningestlib list                             # every stored document\ningestlib remove report.pdf                # erase one\ningestlib reindex                          # rebuild the index (provider switch, new store)\ningestlib recollect                        # re-sort the corpus after editing classify rules\ningestlib search \"what were the risks?\"    # cited retrieval from the shell\n```\n\nDocuments carry a logical identity (`namespace` + source path), so lifecycle\nknows a re-ingest from a brand-new file. Full guide:\n[Manage a corpus](https://langmodule.github.io/ingestlib/how-to/manage-corpus/).\n\n## Query your databases\n\n`retrieve()` can also answer from your **SQL databases**, alongside the\ndocument corpus, in the same call. Declare the databases in a `sources.yaml`\nsidecar (beside config.yaml, like rules.yaml — see\n[`sources.example.yaml`](./sources.example.yaml)) and pass their names:\n\n```python\nfrom ingestlib.services import retrieve\n\n# documents AND databases behind one question\nresult = retrieve(\"how many prescriptions are ready?\", sources=[\"prescriptions\"])\nfor r in result.results:\n    print(r.source_type, r.content, r.provenance[\"sql\"])   # the exact query that ran\n```\n\nThe model generates **read-only** SQL from your schema + `tables` hints, bounded\nby a permission boundary whose floor is a read-only database role — so a wrong\nquery is a wrong *read*, never a write. A statement allowlist, an injected\n`LIMIT`, and a timeout are defense in depth; **verified queries** let you pin\nreviewed SQL for answers that must be exact. On a **wide schema** ingestlib\nretrieves only the tables a question needs — plus their foreign-key bridges —\ninstead of dumping every table into the prompt (`schema_rag: auto`). Generate\nhints for a cryptic schema with `ingestlib describe-schema`, and measure\ngeneration accuracy on your own schema with `ingestlib eval-sql`. Each SQL\nbackend needs its pip extra (`ingestlib[postgres]` · `[mysql]` · `[duckdb]` ·\n`[snowflake]`; sqlite needs none). From the shell:\n`ingestlib search \"…\" --sources prescriptions`. Full guide:\n[Query databases (SQL)](https://langmodule.github.io/ingestlib/how-to/structured-retrieval/).\n\n## Serve it to agents (MCP)\n\n`ingestlib mcp` exposes the whole loop — search, extract, ingest, sync, remove,\nreindex, recollect, verify, and registry management — as [MCP](https://modelcontextprotocol.io) tools, so Claude Desktop,\nCursor, or any agent can drive your **self-hosted** corpus with citations,\nnothing leaving your machine.\n\n```bash\npip install \"ingestlib[mcp]\"\ningestlib mcp                                # stdio (local clients)\ningestlib mcp --transport http --port 8000   # remote; needs MCP_TOKEN (bearer auth)\n```\n\nRead tools (`search`/`extract`/`classify`/`list`/`doctor`) are always on; the\nwrite tools hide under `--read-only`. Guide:\n[Serve to agents (MCP)](https://langmodule.github.io/ingestlib/how-to/mcp-server/).\n\n**Run the whole stack in containers** — the app (MCP front-end), the registry, a\nvector store, and a MinIO artifact store come up with one `docker compose`. See\n[Deploy with Docker](https://langmodule.github.io/ingestlib/how-to/deploy-docker/)\nfor the full walkthrough.\n\n## Using the operations directly\n\nEvery operation also works standalone:\n\n```python\nfrom ingestlib.operations import parse, classify, split\n\nresult = parse(\"report.pdf\")            # ParseResult: pages, regions, figures\nprint(result.markdown)                  # whole-document markdown\nresult.save_images(\"out/\")              # extracted figures/charts as PNGs\n\nlabel = classify(\"report.pdf\")          # no OCR needed — native text + embedded images\nchunks = split(result, category=label.category)\nfor c in chunks.chunks:\n    print(c.token_estimate, c.embedding_text.splitlines()[0])\n```\n\nAnd the fourth operation pulls **structured data** out — your schema,\nfilled and cited:\n\n```python\nfrom pydantic import BaseModel\nfrom ingestlib.operations import extract\n\nclass Receipt(BaseModel):\n    merchant: str\n    total: float\n    currency: str\n\nreport = extract(parse(\"expenses.pdf\"), schema=Receipt, mode=\"many\")\nfor item in report.items:\n    print(item.value.merchant, item.value.total, item.citation)\n# BART 20.0 p.10  ·  Hilton 214.6 p.2  ·  …\nprint(report.items[0].fields[\"total\"].grounded)   # True — verified in the cited region\n```\n\n`mode=\"one\"` fills a single instance from the whole document (a 10-K's\nheadline financials); `mode=\"many\"` finds every instance across the pages.\nA `ParseResult` input gives region-level citations on scans; a raw path\nreads the native text layer with page-level citations and no OCR server.\nConfidence is honest: a field whose citation doesn't check out is capped,\nand a value not found in its cited text is flagged `grounded=False`.\n\nA stored document reads back explicitly — queryable metadata from the registry,\nbytes from the blob store:\n\n```python\nfrom ingestlib.services import get_document\n\ndoc = get_document(doc_id)              # the stored document, from the registry\nprint(doc.category, doc.chunk_count)    # queryable metadata\ndoc.markdown()                          # whole-document markdown (blob store)\n[e[\"schema_name\"] for e in doc.extractions]   # any persisted extractions\n```\n\n## Classification & split rules\n\nClassify and split are open-ended by default — the LLM decides the document\ntype and discovers the section vocabulary. Both can instead follow **your**\nrules: pass them per call, or preset them once in `rules.yaml` beside your\nconfig.yaml, and every bare call **and the whole ingest pipeline** uses\nthem automatically:\n\n```python\nclassify(\"doc.pdf\",\n         {\"invoice\": \"Itemized charges, tax info, and payment terms\",\n          \"sec_filing\": \"10-K/10-Q style regulatory filings\"},\n         target_pages=\"1,3,5-7\", max_pages=5)   # read only these pages\n```\n\n```python\nsplit(\"report.pdf\",\n      vocabulary={\"financial_statements\": \"Balance sheets, income statements\",\n                  \"notes\": \"Footnotes and disclosures\"},\n      unmatched=\"other\")   # pages fitting nothing: other (default) | require | skip\n```\n\n```yaml\n# rules.yaml — beside your config.yaml; infra stays in config.yaml,\n# what your documents MEAN lives here\nclassify:\n  max_pages: 5\n  rules:                       # up to 20 — result is one of these or \"uncategorized\"\n    invoice: \"Itemized charges, tax info, and payment terms\"\n    sec_filing: \"10-K/10-Q style regulatory filings\"\nsplit:\n  unmatched: other             # require | other | skip\n  categories:                  # up to 50 — YOUR sections; Pass 1 is skipped\n    financial_statements: \"Balance sheets, income statements, cash flows\"\n    notes: \"Footnotes and disclosures\"\n```\n\nClassify returns one of your labels or `\"uncategorized\"`, with confidence,\nreasoning, and ranked alternatives. Split labels every page against your\nsections — unmatched pages become an honest `other` section (default), get\nforced into the nearest category (`require`), or are dropped entirely\n(`skip`). Precedence everywhere: explicit arguments beat the preset, and\n`{}` forces the open-ended default even when a preset exists.\n\n## OpenAI backend\n\nThe same LLM surface Bedrock provides is also available on OpenAI — GPT-5\nchat with vision, thinking mode, schema-enforced structured output, and\ntext-embedding-3 embeddings. Add `OPENAI_API_KEY` to `.env` and pick models\nin config.yaml's `openai:` section (defaults: `gpt-5-mini`,\n`text-embedding-3-small`).\n\nTo run the whole `ingest`/`retrieve` pipeline on it, switch the providers\nin config.yaml — every LLM and embedding call routes accordingly:\n\n```yaml\nllm_provider: openai          # chart reading, review, classify, chunking\nembedding_provider: openai    # chunk + query embeddings\n```\n\nCombined with `artifact_store: local`, `vector_store: sqlite`, and\n`reranker: jina` (or `none`), the pipeline needs no AWS at all. Two rules:\nswitching `embedding_provider` changes the vector space, so re-ingest\nafterward (`skip_existing=False`) — vectors from different embedding models\nnever mix in one index. And text embeddings only: OpenAI has no\nimage-embedding model.\n\nThe backend is also importable directly, ignoring the config switch:\n\n```python\nfrom ingestlib.foundations.llm import Image\nfrom ingestlib.foundations.llm.openai import chat, chat_structured, embed_text\n\nchat(\"Read this chart\", images=[Image(png_bytes, \"png\")])   # vision works\nembed_text(\"a chunk of text\")                               # 1024-dim default\n```\n\n## Local backend (Ollama)\n\nThe third provider keeps every LLM call on your machine — the fully\nair-gapped pipeline. Point config.yaml at a local\n[Ollama](https://ollama.com) (or any OpenAI-compatible server: vLLM,\nLM Studio):\n\n```yaml\nllm_provider: ollama\nembedding_provider: ollama\n\nollama:                                    # these are the defaults\n  base_url: http://localhost:11434/v1\n  llm_model_id: qwen3.5:9b\n  embedding_model_id: qwen3-embedding:0.6b\n```\n\n```bash\nollama pull qwen3.5:9b\nollama pull qwen3-embedding:0.6b\n```\n\nNo API key. Vision, schema-enforced structured output, and 1024-dim\nembeddings all verified on the reference stack — and **local embedding\nquality is measured, not assumed**: on the retrieval eval, qwen3-embedding\nmatched or beat the cloud stack (perfect reranked hit@1/3/5 = 1.00). Two\nhonest notes: use the GGUF builds, not `-mlx` (Ollama's MLX engine\nsilently drops images and schema enforcement), and a local 9B won't match\nthe cloud models on dense charts — test with your own documents before\ncommitting.\nCombined with `artifact_store: local` and `vector_store: sqlite`,\nnothing leaves your machine but the optional Jina rerank call\n(`reranker: none` closes even that).\n\n## Architecture\n\n```\nsrc/ingestlib/\n├── services/       ingest · retrieve · lifecycle (remove · sync · reindex · recollect) · verify — the product\n├── operations/     parse · classify · split · extract — the tools (each standalone)\n├── storage/        registry (Postgres metadata hub) · artifacts (S3 | local, bytes only) · base (VectorStore contract) · 8 connectors\n│                   (pinecone · qdrant · sqlite · pgvector · mongodb · milvus\n│                    · opensearch · weaviate)\n├── sources/        structured retrieval — SQL databases & the corpus as queryable Sources\n├── foundations/    llm (Bedrock Nova · OpenAI GPT-5 · Ollama Qwen · Jina) · ocr (PaddleOCR-VL)\n├── cli/            the `ingestlib` command — init · doctor · ingest · sync · list · show · collections · remove · reindex · recollect · verify · search · describe-schema · eval-sql · registry · mcp\n├── mcp/            MCP server (ingestlib[mcp]) — expose the pipeline to agents\n├── utils/          logger · files · sync · aws\n└── config.py       config.yaml + .env + rules.yaml + sources.yaml → typed configs\n\nsrc/ingestlib_registry/   the registry's Postgres schema + Alembic migrations (a standalone package ingestlib depends on)\n```\n\nStrict downward dependencies. The `VectorStore` contract means backends drop\nin as connectors — all eight ship **hybrid search**: **Pinecone** (dense +\nhosted sparse model, merged client-side), **Qdrant** (dense + BM25 with\nserver-side IDF and RRF fusion; local docker or cloud), **SQLite**\n(sqlite-vec KNN + built-in FTS5 BM25 with porter stemming, RRF fusion — one\nlocal file, no server, no keys), **Postgres/pgvector** (HNSW cosine +\nbuilt-in full-text over a generated weighted tsvector, RRF fusion — the\nextension and table bootstrap automatically), **MongoDB** (Atlas Vector\nSearch + Atlas Search true BM25, RRF fusion — Atlas any tier or self-managed\n8.2+; both search indexes bootstrap automatically), **Milvus** (dense\nANN + server-computed BM25 sparse, fused server-side with RRF — local docker\nor Zilliz Cloud), **OpenSearch** (faiss HNSW k-NN + Lucene BM25, RRF fused\nclient-side — an Amazon OpenSearch domain SigV4-signed with your aws\nprofile, or local docker), and **Weaviate** (HNSW dense + native BM25 fused\nserver-side in one hybrid call — local docker or Weaviate Cloud). Pick one\nwith `vector_store: pinecone | qdrant | sqlite | pgvector | mongodb |\nmilvus | opensearch | weaviate` in config.yaml. Connection secrets sit in\n`.env` together (sqlite needs none) — only the selected connector ever\nbuilds a client.\n\n## Logging\n\n```bash\nINGESTLIB_LOG_LEVEL=INFO           # DEBUG | INFO | WARNING | ERROR (default INFO)\nINGESTLIB_LOG_THIRD_PARTY=1        # also show paddlex/httpx/botocore chatter\nINGESTLIB_LOG_COLOR=0              # disable colored output\n```\n\n## Testing\n\nTests hit **real APIs, never mocks**. Pure logic runs always; server-hitting\nsuites are opt-in via env gates. The sqlite connector's full suite runs\nungated in `make test` — there is no server, so in-process IS the real thing.\n\n```bash\nmake test                  # fast suite (~640 tests, ~2min; e2e groups skip)\nmake test-openai           # OpenAI backend       (skips without OPENAI_API_KEY)\nmake test-ollama           # Ollama backend       (needs a local Ollama + models)\nmake test-parse            # parse e2e            (needs VL server + LLM provider)\nmake test-classify         # classify e2e         (needs the LLM provider)\nmake test-split            # split e2e            (needs the LLM provider)\nmake test-extract          # extract e2e          (needs the LLM provider; scans need the VL server)\nmake test-s3               # artifact store e2e   (needs AWS)\nmake test-minio            # artifact store e2e   (vs a local MinIO — no AWS)\nmake test-pinecone         # vector connector e2e (needs Pinecone + embeddings)\nmake test-qdrant           # vector connector e2e (needs a Qdrant server + embeddings)\nmake test-sqlite           # vector connector suite (no gate — nothing to need)\nmake test-pgvector         # vector connector e2e (needs Postgres at PGVECTOR_URL)\nmake test-mongodb          # vector connector e2e (needs MongoDB at MONGODB_URL)\nmake test-milvus           # vector connector e2e (needs Milvus at MILVUS_URL)\nmake test-opensearch       # vector connector e2e (needs OpenSearch at OPENSEARCH_URL)\nmake test-weaviate         # vector connector e2e (needs Weaviate at WEAVIATE_URL)\nmake test-services         # full product e2e     (needs the entire stack)\nmake test-sources          # structured retrieval — SQL sources (deterministic; e2e gated)\nmake test-cli              # CLI: init/doctor + corpus commands (no gate)\nmake test-lifecycle        # remove/sync/reindex/recollect + replace-aware ingest (gates on the registry)\nmake test-mcp              # MCP server: tools, read_only, http auth (no gate)\nmake test-all              # everything\nmake eval                  # retrieval quality eval (see below)\nmake docs                  # live-preview the documentation site\n```\n\nFixtures live in `tests/data/` — 15 real PDFs (research papers, earnings\ndecks, insurance forms, 10-Ks, a 16-page receipt scan, a password-protected\none), document images incl. WebP, and real DOCX/PPTX files. The\nserver-backed connector suites need their servers:\n`docker compose -f infra/docker-compose.yml --profile all up -d` starts\nevery local store (or `--profile qdrant` etc. for just one).\n\n### Retrieval quality\n\nBeyond pass/fail tests, `evals/` measures retrieval quality: 22 ground-truth\nquestions over the fixture corpus, run through the real `retrieve()` flow\nunder dense/hybrid × rerank on/off, scored by hit@k and MRR. Measured so far\n(consistent across all eight connectors): **with reranking, every answer\nlands in the top 3 hits (hit@3 = 1.00)**; hit@1 ranges 0.86–1.00 across runs.\nThe fully-local stack holds the same bar — Ollama embeddings scored a\nperfect 1.00 across hit@1/3/5 and MRR on dense+rerank.\nEach run saves a timestamped snapshot to `evals/results/`, so quality changes\nare visible over time.\n\n## Disk footprint\n\n| Component | Size | Location |\n|---|---|---|\n| Python deps | ~1.4 GB core (+ your vector-store extra) | `.venv/` |\n| PaddleOCR-VL-1.6 weights | ~1.8 GB | `~/.cache/huggingface/hub/` |\n| PP-DocLayoutV3 | ~126 MB | `~/.paddlex/official_models/` |\n| LibreOffice | ~600 MB | system |\n\n## Scope\n\nEnglish documents; PDF / DOCX / PPTX / PNG / JPEG / WebP input. Images,\ncharts, and tables **inside** documents are fully extracted and interpreted,\nand a single image file parses as a one-page document; handwriting is out\nof scope by design.\n\n## The studio\n\n[ingestlib-studio](https://github.com/LangModule/ingestlib-studio) is the\nvisual companion: a local web UI with a setup wizard, try-before-you-commit\npipeline runs, page-by-page review with hover-to-highlight bounding boxes,\ncommitted ingestion with live progress, a content-rules editor, and a\nretrieval playground where every answer points to its source on the page.\n\n## Roadmap\n\n- XLSX input (tables-first, not a PDF conversion)\n\nRecently shipped: the **internal registry** — a Postgres metadata hub that makes\nthe whole corpus queryable, with `reindex`/`recollect`/`verify` for rebuilding\nand auditing it and event-driven backups (v1.5); **schema-RAG for wide\ndatabases** — retrieve the relevant tables (with foreign-key closure) instead of\ndumping the whole schema, plus `describe-schema` auto-documentation and the\n`eval-sql` accuracy harness (v1.4); **structured retrieval** — query your SQL\ndatabases alongside documents through one `retrieve()` call, behind a read-only\npermission boundary (v1.3); an **MCP server** to serve the corpus to agents\n(v1.2); document lifecycle — replace-aware ingestion, folder `sync()`,\n`reindex()`, and the corpus CLI (v1.1).\n\n## License\n\nSee [LICENSE](./LICENSE).\n"}