{"id":"79688d0c-fea0-40da-ab08-9e28431ae72c","slug":"dod","kind":"mcp","name":"DoD","summary":"Document Outline Discovery","owner":"Taooo-habitus","repoUrl":"https://github.com/Taooo-habitus/DoD","license":"MIT","licenseTier":"permissive","trust":"community","stars":4,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-02-12T10:15:37.000Z","addedAt":"2026-10-21T16:49:01.592Z","logoUrl":"https://avatars.githubusercontent.com/u/177202425?u=213e671fa2eba86f5ec094a09edb7861ad48f3b5&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["files-documents"],"tags":[],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-22T09:17:11.437Z","sample":false,"url":"https://conduits.dev/tools/dod","language":"Python","transports":["stdio"],"install":{"pypi":"dod-outline-discovery"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# DoD (Document Outline Discovery)\n\n> This project is intended to make local deployment and usage of PageIndex easier. It is not an official PageIndex service. Check their official PageIndex repo [here](https://github.com/VectifyAI/PageIndex).\n\nDoD is a local-first document structure extraction toolkit built around [PageIndex](https://pageindex.ai/).\n\nWhat it does:\n\n- ingests PDFs and builds normalized page artifacts\n- extracts page-level text (`pymupdf` or `pytesseract`)\n- generates hierarchical TOC/document outline trees\n- exposes both **CLI** and **server+MCP** interfaces for agent workflows\n\nUse cases:\n\n- document-grounded Q&A assistants over private document libraries\n- TOC/outline extraction for scanned or OCR-heavy PDFs\n- page-targeted retrieval pipelines for downstream RAG/agent systems\n\nAdvantages:\n\n- local deployment for sensitive documents\n- structured outputs (TOC tree + page tables + page images)\n- agent-friendly retrieval via stable server and MCP tools\n\nWhy this can make more sense than traditional RAG for long manuals:\n\n- instead of flat chunk retrieval, PageIndex builds explicit document structure (sections/subsections + page mapping)\n- this improves navigation, targeted retrieval, and answer grounding for long technical documents\n- see PageIndex's technical-manual discussion: https://pageindex.ai/blog/technical-manuals\n\nExample (Codex using DoD MCP for section-grounded Q&A):\n\n![Codex + DoD MCP example](mcp_example_pics/mcp-example-codex.png)\n\n---\n\n## Table of Contents\n\n- [Project Structure](#project-structure)\n- [0. LLM API Configuration Required](#0-llm-api-configuration-required)\n- [1. Install and Dev](#1-install-and-dev)\n- [1.1 Use as a PyPI package](#11-use-as-a-pypi-package)\n- [2. Run As A Package CLI](#2-run-as-a-package-cli)\n- [3. Run As A Server](#3-run-as-a-server)\n  - [3.1 Start server](#31-start-server)\n  - [3.2 Health check](#32-health-check)\n  - [3.3 Make requests](#33-make-requests)\n- [4. Output What And Where](#4-output-what-and-where)\n  - [4.3 Configure output paths PyPI package](#43-configure-output-paths-pypi-package)\n- [5. Request Fields Server `/v1/digest`](#5-request-fields-server-v1digest)\n- [6. Simple MCP Setup](#6-simple-mcp-setup)\n  - [6.1 Install from PyPI for MCP use](#61-install-from-pypi-for-mcp-use)\n  - [6.2 Start DoD HTTP server](#62-start-dod-http-server)\n  - [6.3 Configure MCP client recommended](#63-configure-mcp-client-recommended)\n  - [6.4 Optional manual MCP run debug only](#64-optional-manual-mcp-run-debug-only)\n  - [6.5 Available MCP tools](#65-available-mcp-tools)\n  - [6.6 Example agent skill](#66-example-agent-skill)\n- [7. Third-Party Licensing](#7-third-party-licensing)\n\nDoD turns a document into:\n\n1. `page_table.jsonl` (page text + metadata)\n2. `toc_tree.json` (hierarchical Table-of-Contents tree)\n3. `image_page_table.jsonl` (page-image index, including image paths and image payload fields)\n4. `images/` (all pages rendered as image files)\n\nCore flow:\n\n1. PDF normalization to page images\n2. Text extraction (`pymupdf` / `pytesseract`)\n3. TOC generation (PageIndex)\n4. Artifact writing (JSON/JSONL + manifest)\n\n## Project Structure\n\n- `src/DoD/cli/` - CLI entrypoint\n- `src/DoD/pipeline.py` - end-to-end pipeline orchestration\n- `src/DoD/normalize/` - PDF/image normalization to per-page images\n- `src/DoD/text_extractor/` - text extraction backends\n- `src/DoD/page_table.py` - page table data model + writer\n- `src/DoD/pageindex/` - PageIndex TOC builder\n- `src/DoD/toc/` - TOC adapters\n- `src/DoD/server/` - FastAPI server mode\n- `src/scripts/` - executable entrypoints (`dod`, `dod-server`, `dod-mcp`)\n- `.agents/skills/` - example agent skills\n- `makefile` - install/check/test developer workflow\n- `src/DoD/conf/config.yaml` - default configuration\n\n## 0. LLM API Configuration Required\n\nBefore running either the CLI package mode or the server mode, set an OpenAI-compatible endpoint and API key:\n\n```bash\nexport PAGEINDEX_API_KEY=\"<your_api_key>\"\nexport PAGEINDEX_BASE_URL=\"<your_openai_compatible_base_url>\"\n```\n\nExample (Snowflake Cortex):\n\n```bash\nexport PAGEINDEX_API_KEY=\"<snowflake_pat>\"\nexport PAGEINDEX_BASE_URL=\"https://<account-identifier>.snowflakecomputing.com/api/v2/cortex/v1\"\n```\n\nThen choose any model available on your configured endpoint via `toc.model`.\n\n## 1. Install and Dev\n\nUse these commands for setup and day-to-day development:\n\n```bash\nmake install  # bootstrap toolchain + Python deps + hooks\nmake check    # lint + format + type-check\nmake test     # run test suite\n```\n\n### 1.1 Use as a PyPI package\n\nInstall from PyPI:\n\n```bash\npip install \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\"\n```\n\nThen run:\n\n```bash\ndod --help\ndod-server --help\ndod-mcp --help\n```\n\nOne-off execution without a persistent install:\n\n```bash\nuvx --from \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\" dod --help\nuvx --from \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\" dod-server --help\nuvx --from \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\" dod-mcp --help\n```\n\n## 2. Run As A Package CLI\n\nRun one document:\n\nChoose extractor first:\n\n- use `text_extractor.backend=pymupdf` for PDFs with built-in text layers\n- use `text_extractor.backend=pytesseract` for image-based/scanned PDFs\n\n```bash\ndod \\\n  input_path=/path/to/document.pdf \\\n  text_extractor.backend=pymupdf \\\n  toc.backend=pageindex \\\n  toc.model=claude-sonnet-4-5\n```\n\nIf you are running from source repo instead of a PyPI install, use `uv run dod`.\n\nWhere output is written:\n\n- Hydra run dir: `outputs/<YYYY-MM-DD>/<HH-MM-SS>/`\n- Artifacts folder: `outputs/<YYYY-MM-DD>/<HH-MM-SS>/artifacts/`\n\nMain artifact files:\n\n- `page_table.jsonl`\n- `image_page_table.jsonl`\n- `toc_tree.json`\n- `manifest.json`\n\n## 3. Run As A Server\n\n### 3.1 Start server\n\n```bash\nexport DOD_SERVER_HOST=0.0.0.0\nexport DOD_SERVER_PORT=8000\nexport DOD_SERVER_MAX_CONCURRENT_DOCS=4\nexport DOD_SERVER_JOB_TIMEOUT_SECONDS=300\nexport DOD_SERVER_WORK_DIR=outputs/server_jobs\ndod-server\n```\n\nIf you are running from source repo instead of a PyPI install, use `uv run dod-server`.\n\n### 3.2 Health check\n\n```bash\ncurl http://localhost:8000/healthz\n```\n\n### 3.3 Make requests\n\n#### 3.3.1 Single PDF wait for final result\n\n```bash\ncurl -s -X POST \"http://localhost:8000/v1/digest\" \\\n  -F \"file=@/path/to/document.pdf\" \\\n  -F \"text_extractor_backend=pymupdf\" \\\n  -F \"toc_backend=pageindex\" \\\n  -F \"toc_model=claude-sonnet-4-5\" \\\n  -F \"toc_concurrent_requests=4\" \\\n  > result.json\n```\n\nThis call blocks until the job is done and writes full result JSON to `result.json`.\n\n#### 3.3.2 Async job submit then poll\n\nSubmit:\n\n```bash\nSUBMIT_JSON=$(curl -s -X POST \"http://localhost:8000/v1/digest?wait=false\" \\\n  -F \"file=@/path/to/document.pdf\" \\\n  -F \"text_extractor_backend=pymupdf\" \\\n  -F \"toc_backend=pageindex\" \\\n  -F \"toc_model=claude-sonnet-4-5\")\nJOB_ID=$(echo \"$SUBMIT_JSON\" | jq -r '.job_id')\nJOB_REF=$(echo \"$SUBMIT_JSON\" | jq -r '.job_ref')\n```\n\nCheck status:\n\n```bash\ncurl -s \"http://localhost:8000/v1/jobs/$JOB_REF\"\n```\n\nGet final result:\n\n```bash\ncurl -s \"http://localhost:8000/v1/jobs/$JOB_REF/result\" > result.json\n```\n\n#### 3.3.3 Process More Than One PDF At The Same Time\n\nUse async submit (`wait=false`) + parallel curl:\n\n```bash\nmkdir -p jobs\nprintf \"%s\\n\" \\\n  \"/path/to/a.pdf\" \\\n  \"/path/to/b.pdf\" \\\n  \"/path/to/c.pdf\" \\\n| xargs -I{} -P 3 sh -c '\n  name=$(basename \"{}\" .pdf)\n  curl -s -X POST \"http://localhost:8000/v1/digest?wait=false\" \\\n    -F \"file=@{}\" \\\n    -F \"text_extractor_backend=pymupdf\" \\\n    -F \"toc_backend=pageindex\" \\\n    -F \"toc_model=claude-sonnet-4-5\" \\\n    -F \"toc_concurrent_requests=4\" \\\n  > \"jobs/${name}.submit.json\"\n'\n```\n\nPoll and download all results:\n\n```bash\nfor f in jobs/*.submit.json; do\n  job_id=$(jq -r '.job_id' \"$f\")\n  name=$(basename \"$f\" .submit.json)\n  until curl -sf \"http://localhost:8000/v1/jobs/$job_id/result\" > \"jobs/${name}.result.json\"; do\n    sleep 2\n  done\ndone\n```\n\nNotes:\n\n- `-P 3` controls how many submit requests run in parallel.\n- Server-side processing concurrency is capped by `DOD_SERVER_MAX_CONCURRENT_DOCS`.\n- TOC runs in strict mode: if PageIndex fails, the job status becomes `failed` (no fallback TOC).\n\n## 4. Output What And Where\n\n### 4.1 Output in API response JSON\n\n`/v1/digest` (sync) or `/v1/jobs/{job_id}/result` returns:\n\n- `result.toc_tree` - TOC tree JSON\n- `result.page_table` - page records (JSON array parsed from JSONL)\n- `result.image_page_table` - image records (JSON array parsed from JSONL)\n- `result.artifact_paths` - filesystem paths for written artifacts\n- `result.manifest` - run metadata + config\n\nConvert arrays back to JSONL if needed:\n\n```bash\njq -c '.result.page_table[]' result.json > page_table.jsonl\njq -c '.result.image_page_table[]' result.json > image_page_table.jsonl\njq '.result.toc_tree' result.json > toc_tree.json\n```\n\n### 4.2 Output on disk Server mode\n\nFor each job:\n\n- Job folder: `${DOD_SERVER_WORK_DIR}/<job_id>/`\n- Input copy: `${DOD_SERVER_WORK_DIR}/<job_id>/input.pdf`\n- Artifact folder: `${DOD_SERVER_WORK_DIR}/<job_id>/artifacts/`\n\nInside `artifacts/`:\n\n- `page_table.jsonl`\n- `image_page_table.jsonl`\n- `toc_tree.json`\n- `manifest.json`\n\nServer-level job index:\n\n- `${DOD_SERVER_WORK_DIR}/jobs.json`\n  - persists job metadata across server restarts\n  - used by `GET /v1/jobs` and MCP `list_jobs()`\n\nTimeout:\n\n- default from `src/DoD/conf/config.yaml` → `server.job_timeout_seconds`\n- runtime override via `DOD_SERVER_JOB_TIMEOUT_SECONDS`\n\n### 4.3 Configure output paths PyPI package\n\nWhen installed from PyPI, outputs are still filesystem-based and default to the current working directory.\n\n- CLI (`dod`): defaults to Hydra output under `outputs/<YYYY-MM-DD>/<HH-MM-SS>/artifacts`\n- Server (`dod-server`): defaults to `outputs/server_jobs`\n\nUse absolute paths in production/local deployments:\n\n```bash\nexport DOD_SERVER_WORK_DIR=\"/absolute/path/to/server_jobs\"\nexport DOD_LLM_CACHE_DIR=\"/absolute/path/to/llm_cache\"\n```\n\nOptional CLI override for one run:\n\n```bash\ndod input_path=/path/to/document.pdf artifacts.output_dir=/absolute/path/to/artifacts\n```\n\n## 5. Request Fields Server `/v1/digest`\n\nMultipart form fields:\n\n- `file` (required, `.pdf`)\n- `text_extractor_backend` (optional)\n  - use `pymupdf` for PDFs with built-in text layers\n  - use `pytesseract` for image-based/scanned PDFs\n- `normalize_max_pages` (optional int)\n- `toc_backend` (optional, typically `pageindex`)\n- `toc_model` (optional model name)\n- `toc_concurrent_requests` (optional int)\n- `toc_check_page_num` (optional int)\n- `toc_api_key` (optional per-request override)\n- `toc_api_base_url` (optional per-request override)\n\nQuery parameter:\n\n- `wait` (default `true`)\n  - `true`: request returns when job finishes\n  - `false`: request returns immediately with job metadata (`job_id`, `job_ref`, status/result URLs)\n\n## 6. Simple MCP Setup\n\nIf you want agents to call DoD as tools, use the included MCP wrapper at `src/scripts/dod_mcp.py`.\nAgents do not upload PDFs. Human users submit jobs first via `/v1/digest`, then agents use `job_ref` to retrieve targeted outputs.\n\n`make install` already installs all extras including MCP/server dependencies.\n\n### 6.1 Install from PyPI for MCP use\n\n```bash\nuv tool install \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\"\n```\n\nThis installs `dod`, `dod-server`, and `dod-mcp` as system tools.\n\n### 6.2 Start DoD HTTP server\n\n```bash\ndod-server\n```\n\nIf you are running from source repo instead of PyPI install, use `uv run dod-server`.\n\n### 6.3 Configure MCP client recommended\n\nMost MCP hosts should launch `dod-mcp` themselves via command config, for example (`~/.codex/config.toml`):\n\n```toml\n[mcp_servers.dod_mcp]\ncommand = \"dod-mcp\"\n```\n\nAlternative without tool install:\n\n```toml\n[mcp_servers.dod_mcp]\ncommand = \"uvx\"\nargs = [\"--from\", \"dod-outline-discovery[pageindex,text_extractor,pdf,server,mcp]\", \"dod-mcp\"]\n```\n\nIn this mode, do not start `dod-mcp` manually. Keep only `dod-server` running.\n\n### 6.4 Optional manual MCP run debug only\n\n```bash\ndod-mcp\n```\n\nIf you are running from source repo instead of a PyPI install, use `uv run dod-mcp`.\nUse this only for debugging MCP transport behavior.\n\n### 6.5 Available MCP tools\n\n- `list_jobs()`\n  - Returns: `{ jobs: [{ file_name, job_id, job_ref, status, created_at }] }`.\n- `get_toc(job_ref)`\n  - Returns: `{ job_id, job_ref, status, toc_tree }`.\n- `get_page_texts(job_ref, pages)`\n  - Returns: `{ job_id, job_ref, status, requested_pages, returned_pages, pages }` where `pages` is a selected subset of `{ page_id, text }`.\n- `get_page_images(job_ref, pages, mode)`\n  - Returns: `{ job_id, job_ref, status, mode, requested_pages, returned_pages, pages }` where `pages` is a selected subset of `{ page_id, image_path }` (`mode=path`) or `{ page_id, image_b64 }` (`mode=base64`).\n- `pages` accepts flexible specs like `\"110,111,89-100\"`.\n- Retrieval guardrails come from `src/DoD/conf/config.yaml` under `retrieval`:\n  - `max_chars_per_page` (`null` means full page text)\n  - `max_pages_per_call`\n\n### 6.6 Example agent skill\n\nAn example Codex skill for DoD library Q&A is included at:\n\n- `.agents/skills/dod-library-qa/SKILL.md`\n\n## 7. Third-Party Licensing\n\nThis project includes third-party license attributions in:\n\n- `THIRD_PARTY_NOTICES.md`\n"}