genai-graph

GenAI Graph

Hybrid Knowledge Graph, Document Graph, and GraphRAG framework built on top of genai-tk.

GenAI Graph transforms unstructured document collections (PDF, PPTX, DOCX, Markdown), structured tables (Excel, CSV), and Neo4j exports into unified, queryable Ladybug graph databases. It provides autonomous document-walking agents, multi-dataset benchmark evaluation pipelines (cli bench), and interactive Streamlit/TUI visualization interfaces.

---

DeepWiki generated documentation is available (might be not in sync):

Architecture Overview

flowchart TD
    subgraph Sources["1. Heterogeneous Data Sources"]
        DOCS["Documents (PDF, PPTX, DOCX, Markdown)"]
        TABLES["Tabular Data (Excel, CSV)"]
        NEO["Neo4j Exports (JSONL)"]
        BENCH_DS["Benchmark Datasets (HuggingFace / Local)"]
    end

    subgraph Ingestion["2. Processing & Factory Layer"]
        MKT["Markdown Knowledge Tree Parser<br/>(Folder → Document → MarkdownSection)"]
        OCR["OCR Ladder (Mistral OCR → Docling → MarkItDown)"]
        BAML["BAML Structured LLM Extraction<br/>(MarkdownBamlFactory)"]
        TAB_F["TableBackedFactory & JsonFileBackedFactory"]
        NEO_F["Neo4jImportFactory & Schema Dedup"]
    end

    subgraph CoreEngine["3. Graph Engine & Schema Compiler"]
        SCHEMA["GraphSchema & ResolvedSchema<br/>(Auto-deduced field paths & relation endpoints)"]
        MERGE["Idempotent Batch Mergers & Fingerprint Cache"]
        LADYBUG["Ladybug Embedded Graph Database<br/>(Node/Rel Tables + CALL QUERY_VECTOR_INDEX)"]
    end

    subgraph Applications["4. Agents, Benchmarks & Interfaces"]
        DAGENT["DocGraph Deep Agent & Vectorless Tools<br/>(cli docgraph agent)"]
        CYPHER["Cypher Query Agents & Text-to-Cypher"]
        BENCH["Unified Benchmark Framework (cli bench)<br/>(6-Stage Pipeline, Mafin 2.5 Grader, Textual TUI)"]
        UI["Streamlit KG Explorer & D3 Interactive Graph (just webapp)"]
    end

    DOCS --> OCR --> MKT --> SCHEMA
    DOCS --> BAML --> SCHEMA
    TABLES --> TAB_F --> SCHEMA
    NEO --> NEO_F --> SCHEMA
    BENCH_DS --> BENCH

    SCHEMA --> MERGE --> LADYBUG
    LADYBUG --> DAGENT
    LADYBUG --> CYPHER
    LADYBUG --> BENCH
    LADYBUG --> UI

---

Core Pillars & Capabilities

📄 1. Document Graph & Markdown Knowledge Tree

Hierarchical document representation designed for deterministic, provenance-backed agent navigation:

• Folder → Document → MarkdownSection schema tracking exact line numbers, token budgets, and parent-child hierarchy.
• Multi-strategy OCR ladder: Mistral OCR (high-fidelity tables & equations) → Docling → MarkItDown fallback.
• Vectorless Agentic RAG: Tools that walk headings, search section titles, and fetch contiguous markdown ranges without vector chunk fragmentation.
• Autonomous DocGraph Agent: Pre-configured deep planning agent with specialized runtime skills (cli docgraph agent).
• See: cli docgraph, docs/document-graph.md

📊 2. Unified Benchmark Framework (cli bench)

Multi-dataset evaluation suite powering empirical AI evaluations (such as FinanceBench, OfficeQA, and MMLongBench-Doc):

• 6-Stage Pipeline: Dataset Fetch → OCR / Markdownize → Hierarchical Graph Ingestion → Autonomous Agent Execution → LLM-as-Judge Evaluation → Summary Metric Aggregation.
• Dynamic Adapter Pattern: Subclass BaseBenchmarkAdapter to encapsulate domain dataset loaders, document fetchers, and judge rubrics.
• Mafin 2.5 Evaluation Rules: Numeric equivalence normalization (currency, scale, percentages, rounding), correctness tiers (CORRECT, PARTIAL, INCORRECT), and failure taxonomy (missing_ocr, calculation, retrieval, halted).
• Interactive Textual TUI: Live trajectory replay, token breakdowns, and verdict inspection with cli bench questions -t --trajectory.
• See: cli bench, docs/benchmark_framework.md, docs/benchmarks_financebench_officeqa.md

🕸️ 3. Knowledge Graph Engine & Schemas

Declarative graph modeling with Pydantic v2 and Ladybug:

• Declarative Modeling: Define plain Pydantic models; GraphSchema automatically deduces field paths, relationship endpoints, and primary keys.
• Ladybug Backend: Fast, embedded Kùzu fork with full Cypher compatibility, vector indexes (CALL QUERY_VECTOR_INDEX), and zero external database overhead.
• Composite Factories: Merge structured tabular data, JSON files, Neo4j dumps, and LLM extractions into a single graph with deduplicated node labels.
• Parquet Caching: Fingerprint-based caching skips unchanged sources during multi-step ETL workflows.
• See: cli kg, docs/graph-definition-guide.md, docs/schema-compilation.md

🧩 4. 4-Tier Skills System

Progressive disclosure of domain knowledge for AI coding agents and runtime models:

• skills/runtime/: User capabilities (e.g. kg-query, kg-docgraph-agent, kg-document-graph, kg-explorer).
• skills/development/: Construction & evaluation recipes (e.g. benchmark-framework, kg-cli, kg-schema, kg-factories, kg-ingest, kg-workflows, kg-neo4j-import, kg-export).
• skills/governance/: Schema health, maintenance, and repository mapping (kg-repo-map, kg-schema-maintenance).
• skills/vendor/: Imported packages (atos-slidev).
• See: cli skills, skills/README.md, docs/SKILLS.md

---

📚 Architectural Design Curriculum

For a didactic, deep-dive understanding of the internal architecture and design patterns, read the numbered design specifications in docs/design/:

1. Design 01: Knowledge Graph Foundation & Schema Compilation — Embedded Ladybug engine, Pydantic v2 schema compiler, deterministic primary key hashing, batch MERGE mechanics, and buffer pool scaling.
2. Design 02: Hierarchical Document Graph & Agentic Navigation — Structure-first outline decomposition (Folder -> Document -> MarkdownSection -> SectionChunk), vectorless agentic navigation, hybrid full-text/vector index, and dynamic query relaxation.
3. Design 03: Enterprise Access Control & Security Trimming — Document-level authorization, principal propagation via harness, early-binding Cypher query filtering, and fail-closed security.
4. Design 04: Multi-Source Ingestion Routing & Route Tables — Rule-selected YAML workflows, pathspec route tables, concurrent sub-workflows (PDF, Office, Web, Git), and single-writer fan-in into Ladybug.
5. Design 05: Model Context Protocol (MCP 2.0) & Graph Navigation Tools — Document Graph MCP servers (stdio and SSE), navigation tool suite (list_documents, get_document_outline, get_section_content, search_sections, query_cypher), and IDE integration.

---

Quick Start

Recommended: Scaffold from Scratch

The preferred way to use genai-graph is to start from a clean project directory using uv and genai-tk scaffolding:

# 1. Create project directory and initialize with uv
mkdir my-graph-app && cd my-graph-app
uv init

# 2. Add dependencies with git sources
uv add "genai-tk @ git+https://github.com/tclatos/genai-tk@main"
uv add "genai_graph @ git+https://github.com/tclatos/genai-graph@main"

# 3. Bootstrap configuration, merged skills, benchmark tools, and starter files
# (--extra harnessing installs the `deepagents` dependency required by
# `cli docgraph agent` and any `type: deep` agent profile)
uv run cli init --name "My Knowledge Graph App" --with-graph --extra harnessing

# Optional: for active development with a local editable checkout of genai-graph:
# uv run cli init --name "My Benchmark Suite" --with-graph --graph-path /path/to/genai-graph

# 4. Sync dependencies and run
uv sync
just run                           # launch interactive agent chat
uv run cli bench list              # list benchmark profiles
uv run cli docgraph --help         # inspect Document Graph commands

(Note: pyproject.toml can also specify [tool.uv.sources] pointing to the git branches or local editable checkouts).

Library Development (Clone & Contribute)

If you are contributing directly to genai-graph:

git clone https://github.com/tclatos/genai-graph.git && cd genai-graph
uv sync

# Verify CLI commands
uv run cli --help

Offline / firewalled networks

• If a corporate proxy whitelists only package hosts, LLM API calls (openrouter, deepinfra, etc.) time out through the proxy. Ensure localhost and the API hosts you use are in NO_PROXY (e.g. export NO_PROXY="localhost,127.0.0.1,openrouter.ai,api.deepinfra.com"); cli info doctor (genai-tk) diagnoses and fixes this programmatically.
• The models.dev catalogue fetch (data/models_dev.json) is non-fatal offline: a warning is logged and model resolution continues from explicit entries — so declare every model your profiles reference explicitly in config/providers/llm.yaml.
• For fully offline ingestion, build a vectorless graph with wiki add --embeddings none (or the docgraph equivalent) — BM25 search still works.

What a deep agent needs

• The harnessing extra installed: uv sync --extra harnessing (deepagents is required by type: deep profiles).
• Every model referenced by agent/docgraph profiles declared explicitly in config/providers/llm.yaml — never rely on models.dev fuzzy resolution.
• Every middleware class: path verified against the real module layout (genai_tk.agents.langchain.middleware.*, genai_graph.agent.middleware.map_before_search) instead of copied from another project's YAML.

---

CLI Reference

1. Document Graph (cli docgraph)

# Build a Document Graph from raw documents (PDF, DOCX, PPTX, Markdown)
uv run cli docgraph build ./docs --db ./data/kg/tree.db

# Inspect documents and Table of Contents (TOC)
uv run cli docgraph list --db ./data/kg/tree.db
uv run cli docgraph toc <doc-name-or-hash> --db ./data/kg/tree.db

# Full-text search across section headings and content
uv run cli docgraph search "operating expenses" --db ./data/kg/tree.db

# Run the autonomous Document Graph agent interactively
uv run cli docgraph agent --chat --db ./data/kg/tree.db
uv run cli docgraph agent "What were the total revenues in Q3?" --db ./data/kg/tree.db

2. Benchmark Framework (cli bench)

# List configured benchmark profiles
uv run cli bench list

# Execute full evaluation pipeline
uv run cli bench run -p mistral_glm
uv run cli bench run -p mistral_glm -q question_001,question_002  # run specific questions
uv run cli bench run -p mistral_glm --dry-run                    # preview pipeline steps
uv run cli bench run -p mistral_glm --no-grade                   # run agent without grading

# Re-grade existing runs with LLM-as-judge
uv run cli bench grade -p mistral_glm --force

# Metric summary report
uv run cli bench report -p mistral_glm

# Inspect questions and trajectories in interactive Textual TUI
uv run cli bench questions -p mistral_glm -t --trajectory

3. Knowledge Graph Construction (cli kg)

# Create / rebuild a knowledge graph from a workflow profile
uv run cli kg create my_graph
uv run cli kg create my_graph --dry-run
uv run cli kg create my_graph --force

# Inspect graph schema and statistics
uv run cli kg schema
uv run cli kg info
uv run cli kg cypher "MATCH (n) RETURN labels(n), count(*)"
uv run cli kg query "Which companies have the most projects?"

# Launch Streamlit KG Explorer UI (Cypher console, D3 graph visualization)
just webapp

4. Neo4j Import (cli neo4j)

# Analyze schema and export structure from Neo4j JSONL export
uv run cli neo4j analyze export.jsonl -o schema.cypher

# Create a small subset for rapid testing
uv run cli neo4j subset export.jsonl subset.jsonl --max-nodes 50 --max-rels 50

# Import into Ladybug database
uv run cli neo4j import export.jsonl --db ./data/kg/imported.db --force

# Query imported database
uv run cli neo4j query "MATCH (n:Person)-[:WORKS_AT]->(c:Company) RETURN n.name, c.name" --db ./data/kg/imported.db

---

Defining a Knowledge Graph in Python

1. Declare Pydantic Domain Models

from pydantic import BaseModel, Field


class Company(BaseModel):
    name: str
    sector: str | None = None


class Person(BaseModel):
    name: str
    role: str | None = None


class Project(BaseModel):
    title: str
    client: Company  # Relationship endpoint
    team: list[Person] = Field(default_factory=list)  # Relationship collection

2. Compile Schema and Ingest

from genai_graph.kg.ingest import create_graph, restart_database
from genai_graph.kg.schema import GraphNode, GraphRelation, GraphSchema

# Define nodes with identity keys
company_node = GraphNode(node_class=Company, name_from="name", key_from="name")
person_node = GraphNode(node_class=Person, name_from="name", key_from="name")
project_node = GraphNode(node_class=Project, name_from="title", key_from="title")

# Define schema and relationships
schema = GraphSchema(
    root_model_class=Project,
    nodes=[project_node, company_node, person_node],
    relations=[
        GraphRelation(from_node=project_node, to_node=company_node, name="FOR_CLIENT"),
        GraphRelation(from_node=project_node, to_node=person_node, name="HAS_MEMBER"),
    ],
)

# Ingest data into Ladybug
backend = restart_database()
project_data = Project(
    title="Cloud Transformation",
    client=Company(name="Global Logistics Corp", sector="Supply Chain"),
    team=[Person(name="Alice", role="Tech Lead"), Person(name="Bob", role="Architect")],
)
create_graph(backend, project_data, schema)

# Query via Cypher
df = backend.execute_get_as_df("MATCH (p:Project)-[:FOR_CLIENT]->(c:Company) RETURN p.title, c.name")
print(df)

---

Authoring a Benchmark Adapter

Implement BaseBenchmarkAdapter from genai_graph.bench.adapters.base:

from pathlib import Path
from genai_graph.bench.adapters.base import (
    BaseBenchmarkAdapter,
    download_hf_file,
    load_hf_dataset_to_pandas,
)
from genai_graph.bench.models import BenchQuestion


class MyBenchmarkAdapter(BaseBenchmarkAdapter):
    """Custom benchmark adapter for evaluating contract understanding."""

    DATASET_REPO = "my_org/contract_qa"

    def load_dataset(self, split: str | None = "test", cache_dir: Path | None = None) -> list[BenchQuestion]:
        df = load_hf_dataset_to_pandas(self.DATASET_REPO, split=split or "test", cache_dir=cache_dir)
        return [
            BenchQuestion(
                id=str(row["id"]),
                doc_name=str(row["contract_name"]),
                doc_names=[str(row["contract_name"])],
                question=str(row["question"]),
                gold_answer=str(row["gold_answer"]),
                evidence=[str(row.get("clause_text", ""))],
                metadata={"category": row.get("category", "")},
            )
            for _, row in df.iterrows()
        ]

    def fetch_document(self, doc_name: str, output_dir: Path) -> Path:
        dest_path = output_dir / f"{doc_name}.pdf"
        if dest_path.exists():
            return dest_path
        return download_hf_file(
            repo_id=self.DATASET_REPO,
            filename=f"pdfs/{doc_name}.pdf",
            repo_type="dataset",
            dest_path=dest_path,
        )

    def get_judge_rubric(self) -> str:
        return (
            "You are an expert legal evaluator assessing answers against ground truth.\n"
            "Score correctness, semantic accuracy, and groundedness in contract clauses."
        )

Configure config/docgraph.yaml and config/bench.yaml:

# config/bench.yaml
default_profile: default
dataset_adapter: my_package.adapter.MyBenchmarkAdapter

paths:
  runs: ${paths.data_root}/my_bench/{profile}/runs.jsonl
  scores: ${paths.data_root}/my_bench/{profile}/scores.jsonl
  scores_summary: ${paths.data_root}/my_bench/{profile}/scores_summary.json

bench_profiles:
  default:
    description: "Default benchmark evaluation"
    docgraph_profile: default
    agent_profile: default
    files:
      pathspecs: ["*"]
    runner:
      concurrency: 10
    grader:
      enabled: true
      llm: DeepSeek-V4-Pro-0813@openrouter
      concurrency: 5

---

Documentation Index

See docs/README.md for the master topic-by-topic documentation map.

Documentation	Description
docs/README.md	Master Documentation Index: Complete categorized reading map
docs/graph-definition-guide.md	5-Minute Quick Start: Models → GraphNode → Schema → Ingestion → Query
docs/document-graph.md	Comprehensive Document Graph guide: Markdown Knowledge Tree, OCR, and DocGraph agent
docs/docgraph-agent.md	Autonomous DocGraph deep agent: tool definitions and execution flow
docs/access-control-security-trimming.md	Enterprise security trimming, document ACLs, and early-binding filtering
docs/benchmark_framework.md	Complete multi-dataset benchmark framework specification (genai_graph.bench)
docs/benchmarks_financebench_officeqa.md	Empirical evaluation and adapter implementations for FinanceBench and OfficeQA
docs/graph-authoring-patterns.md	Pattern catalog: JSON, tables, Neo4j, documents, BAML inline extraction, similarity
docs/schema-compilation.md	Schema compilation internals: field-path deduction, primary key rules, exclusions
docs/graph_construction.md	Factory architecture, canonical types, schema merging, and CLI reference
docs/workflows.md	Workflow DSL for KG pipelines; Prefect task orchestration
docs/prefect_dag_pipeline.md	Prefect DAG internals and task concurrency model
docs/baml_extraction_guide.md	Type-safe structured LLM extraction with BAML integration
docs/kg_explorer.md	Streamlit KG Explorer, Cypher console, and D3 interactive visualization
docs/SKILLS.md	Skills architecture guide, scaffolding instructions, and tier overview
docs/studies/README.md	Empirical studies, benchmark surveys, and codebase audit reports
Agents.md	Development guidelines and architectural invariants for coding agents
Agents_Skills.md	Step-by-step procedure runbooks for codebase maintenance
skills/README.md	4-Tier skills catalog and agent loading instructions

---

Interactive Notebooks

See notebooks/README.md for setup instructions and execution progression.

Notebook	Description
notebooks/01_define_graph_from_scratch.ipynb	Start Here: End-to-end tutorial defining Pydantic models, schema compilation, in-memory Ladybug ingestion, Cypher querying, and D3 visualization.
notebooks/02_document_graph_full_pipeline.ipynb	Full Document Graph pipeline: document conversion (PDF/DOCX), HTML table preservation, BAML/LLM outline enrichment, Ladybug DB indexing, and agent navigation tools.
notebooks/03_access_control_security_trimming.ipynb	Enterprise access control, hierarchical security inheritance (Folder → Document), and runtime Cypher security trimming.
notebooks/cypher_examples.ipynb	Practical Cypher idioms: traversal, aggregations, path expressions, and vector similarity search.
notebooks/document_graph_demo.ipynb	Lightweight zero-LLM demo of parsing markdown files into a Document Graph without external models.

---

Development & Testing

just install-dev   # install with development dependencies
just fmt           # format with ruff
just lint          # lint with ruff
just test          # run all tests (unit + integration)
just check         # fmt + lint + test