citekit

CiteKit

Local-first AI resource mapping SDK for precise content extraction

CiteKit enables AI agents to access specific parts of files—PDF pages, video clips, audio segments, image crops, and code blocks—without uploading entire documents to the cloud. It uses LLMs to generate structured "maps" of your content, then resolves exact references on demand. You can use the default cloud mapper or plug in local models via a custom MapperProvider.

Features

• Local-First: All file processing happens on your machine. No cloud storage persistence.
• Multi-Modal: Supports PDF, Video (MP4/MOV), Audio (MP3/WAV), Images (PNG/JPG), and Text/Code.
• Smart Ingestion:
  • Uses Gemini 1.5 by default (via File API) for handling large files (hours of video). Custom mappers are supported.
  • SHA-256 Hashing & Caching: Never re-process the same file twice.
  • Concurrency Control: Built-in queueing to manage API rate limits.
• Precise Extraction: Get exact PDF pages, video clips (stream-copied), image crops, or code slices.
• Serverless Friendly:
  • Zero Browser Dependencies: Removed pdf-parse to avoid DOMMatrix errors in Vercel/Lambda.
  • Configurable baseDir: Support for read-only filesystems (e.g., /tmp).
  • Lazy Resolution: Internal dependencies (sharp, ffmpeg) are optional and lazy-loaded.
• MCP Compatible: Works native with Claude Desktop and Cline.

Installation

Python

pip install citekit

Requirements:

• Python 3.10+
• ffmpeg (for video/audio support)

JavaScript (Node.js)

npm install citekit

Requirements:

• Node.js 18+
• ffmpeg (for video/audio support)

Quick Start

1. Set your API key (Gemini default mapper only)

export GEMINI_API_KEY=your_gemini_api_key_here

Get a free API key at Google AI Studio or use a local model by implementing a custom mapper (see Custom Mappers).

2. Ingest a file

citekit ingest research.pdf

This generates a structured "map" of the document, identifying sections, pages, and concepts.

3. List your resources

citekit list
# Output:
# - research (document): Research Paper

4. Resolve specific content

citekit resolve research.methodology
# Output: .citekit_output/research_pages_5_8.pdf

Usage

Python SDK

from citekit import CiteKitClient
import asyncio

async def main():
    client = CiteKitClient()
  # Optional: pass a custom MapperProvider if you don't use Gemini
    
    # Ingest a document
    resource_map = await client.ingest("research.pdf", "document")
    
    # Print discovered nodes
    print(f"Title: {resource_map.title}")
    for node in resource_map.nodes:
        print(f"  [{node.type}] {node.id}: {node.title}")
    
    # Resolve a specific section
    evidence = client.resolve("research", "methodology")
    print(f"Extracted to: {evidence.output_path}")

asyncio.run(main())

JavaScript SDK

import { CiteKitClient } from 'citekit';

const client = new CiteKitClient();

// Ingest a document
const map = await client.ingest('research.pdf', 'document');

// Print nodes
console.log(`Title: ${map.title}`);
map.nodes.forEach(node => {
    console.log(`  [${node.type}] ${node.id}: ${node.title}`);
});

// Resolve specific content
const evidence = await client.resolve('research', 'methodology');
console.log(`Extracted to: ${evidence.output_path}`);

MCP Integration

Connect CiteKit to your AI agents (Claude Desktop, Cline) via the Model Context Protocol.

Claude Desktop Configuration (claude_desktop_config.json):

{
  "mcpServers": {
    "citekit": {
      "command": "citekit",
      "args": ["serve"],
      "env": {
        "GEMINI_API_KEY": "your_gemini_api_key_here"
      }
    }
  }
}

If you use a custom mapper, GEMINI_API_KEY is not required.

Available MCP Tools:

• listResources - List all ingested resources
• getStructure - Get the full map of a resource
• getNode - Get details for a specific node
• resolve - Extract specific content (mini-PDF, clip, crop)

Documentation

• Full Documentation - Comprehensive guide with examples
• API Reference - Python SDK reference
• JavaScript SDK - JavaScript SDK reference
• MCP Integration - Set up with AI agents
• Examples - Real-world use cases

Project Structure

citekit/
├── python/              # Python SDK
│   ├── citekit/
│   │   ├── client.py    # Main client
│   │   ├── mapper/      # LLM-based mapping
│   │   ├── resolvers/   # Content extractors
│   │   ├── models.py    # Data models
│   │   ├── mcp_server.py # MCP server
│   │   └── cli.py       # Command-line interface
│   └── pyproject.toml
├── javascript/          # JavaScript SDK
│   ├── src/
│   │   ├── client.ts
│   │   ├── mapper/
│   │   ├── resolvers/
│   │   └── models.ts
│   └── package.json
├── docs/                # Static documentation site
└── README.md

How It Works

1. Ingestion: CiteKit sends your file to a configured mapper (Gemini by default) which analyzes it and returns a structured "map" in JSON format. This map contains:

   • Hierarchical nodes (sections, chapters, topics)
   • Physical locations (page ranges, timestamps, bounding boxes)
   • Summaries and metadata

2. Storage: Maps are stored locally in .resource_maps/ as JSON files.

3. Resolution: When you request a specific node, CiteKit:

   • Looks up the node's location in the map
   • Extracts that exact content:
     • PDFs: Creates a mini-PDF with just those pages
     • Videos: Cuts a clip using ffmpeg (stream-copied, no re-encoding)
     • Images: Crops the specified region
     • Text: Slices content by line range
   • Saves the result to .citekit_output/

Development

Setup

# Clone the repository
git clone https://github.com/abdushakurob/citekit.git
cd citekit

# Python development
cd python
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

# JavaScript development
cd javascript
npm install
npm run build

Running Tests

# Python
cd python
pytest

# JavaScript
cd javascript
npm test

Contributing

Contributions are welcome! Please follow these guidelines:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests and linters
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Code Style

• Python: Follow PEP 8, use Black for formatting
• JavaScript: Follow the existing style, use Prettier

Publishing

See PUBLISHING.md for instructions on publishing to PyPI and npm.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Roadmap

• Support custom LLM providers via MapperProvider
• Caching layer to avoid re-processing
• Web-based UI for browsing resource maps
• Support for more file formats (DOCX, PPTX, etc.)
• Collaborative map editing

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: docs/index.md

Acknowledgments

Built with:

• Gemini API for AI-powered mapping
• PyMuPDF for PDF processing
• ffmpeg for video/audio processing
• Sharp for image processing

---

Made for developers building AI agents