{"id":"b4f04768-b467-4ddc-9ad5-f6ed25bb4566","slug":"crossref-local","kind":"mcp","name":"crossref-local","summary":"Local CrossRef database with 167M+ works and full-text search","owner":"scitex-ai","repoUrl":"https://github.com/scitex-ai/crossref-local","license":"AGPL-3.0","licenseTier":"strong-copyleft","trust":"community","stars":5,"match":null,"fewStars":false,"format":"mcp","createdAt":"2025-08-15T00:29:09.000Z","addedAt":"2026-10-21T08:02:05.260Z","logoUrl":"https://avatars.githubusercontent.com/u/211223684?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["databases","search"],"tags":["abstract","citation","impact-factor","localhost","scientific-papers","claude","cli","doi"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T08:02:14.598Z","sample":false,"url":"https://conduits.dev/tools/crossref-local","language":"Python","transports":["stdio"],"install":{"pypi":"crossref-local"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"\n\n# CrossRef Local (crossref-local)\n\n

\n \n \"SciTeX\"\n \n

\n\n

Local CrossRef database with 167M+ scholarly works, full-text search, and impact factor calculation

\n\n## Demo\n\n

\n \"CrossRef\n

\n\n```bash\n# Search 167M papers locally — no API rate limits, ~22 ms full-text query\ncrossref-local search \"epilepsy seizure prediction\"\n\n# Resolve a DOI to full record (title, abstract, citations, journal IF)\ncrossref-local search-by-doi 10.1038/nature11247\n\n# Drive from MCP / Claude Code\ncrossref-local mcp serve\n```\n\nThe image is a live capture against the local DB; the `
`\nblock below has a 6m55s MCP-driven demo video.\n\n## Architecture\n\n```\n┌──────────────────────────┐ ┌──────────────────────────┐\n│ CrossRef public dump │ │ JCR / OpenAlex IF tables │\n│ (~100 GB compressed) │ │ │\n└──────────────┬───────────┘ └──────────────┬───────────┘\n │ dois2sqlite │\n ▼ ▼\n ┌─────────────────┐ ┌──────────────┐\n │ crossref.db │ ◀── joins ──▶ │ impact-factor│\n │ (SQLite + FTS5) │ │ table │\n └────────┬────────┘ └──────────────┘\n │\n ▼\n ┌──────────────────────────────────┐\n │ crossref-local — Python / CLI / MCP │\n │ search · search-by-doi · cache │\n │ stats · check-citations · relay │\n └──────────────────────────────────┘\n```\n\nThe DB lives entirely on disk; `crossref-local` is a thin facade over\nSQLite + FTS5 + a small impact-factor table. No network calls during\nqueries; rebuild scripts under `make fts-build-screen` /\n`citations-build-screen` are the only producers of state.\n\n[![PyPI version](https://badge.fury.io/py/crossref-local.svg)](https://badge.fury.io/py/crossref-local)\n[![Documentation](https://readthedocs.org/projects/crossref-local/badge/?version=latest)](https://crossref-local.readthedocs.io/en/latest/)\n[![Tests](https://github.com/ywatanabe1989/crossref-local/actions/workflows/test.yml/badge.svg)](https://github.com/ywatanabe1989/crossref-local/actions/workflows/test.yml)\n[![Coverage](https://codecov.io/gh/ywatanabe1989/crossref-local/branch/main/graph/badge.svg)](https://codecov.io/gh/ywatanabe1989/crossref-local)\n[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](LICENSE)\n\n
\nMCP Demo Video\n\n

\n \n \"Demo\n \n

\n\nLive demonstration of MCP server integration with Claude Code for `epilepsy seizure prediction` literature review:\n- Full-text search on title, abstracts, and keywords across 167M papers (22ms response)\n\n📄 [Full demo documentation](examples/demo_mcp.org) | 📊 [Generated diagrams](examples/04_mcp_demo_out/)\n\n
\n\n
\nWhy CrossRef Local?\n\n**Built for the LLM era** - features that matter for AI research assistants:\n\n| Feature | Benefit |\n|---------|---------|\n| 📝 **Abstracts** | Full text for semantic understanding |\n| 📊 **Impact Factor** | Filter by journal quality |\n| 🔗 **Citations** | Prioritize influential papers |\n| ⚡ **Speed** | 167M records in ms, no rate limits |\n\nPerfect for: RAG systems, research assistants, literature review automation.\n\n
\n\n
\nInstallation\n\n```bash\npip install crossref-local\n```\n\nFrom source:\n```bash\ngit clone https://github.com/ywatanabe1989/crossref-local\ncd crossref-local && make install\n```\n\nDatabase setup (1.5 TB, ~2 weeks to build):\n```bash\n# 1. Download CrossRef data (~100GB compressed)\naria2c \"https://academictorrents.com/details/...\"\n\n# 2. Build SQLite database (~days)\npip install dois2sqlite\ndois2sqlite build /path/to/crossref-data ./data/crossref.db\n\n# 3. Build FTS5 index (~60 hours) & citations table (~days)\nmake fts-build-screen\nmake citations-build-screen\n```\n\n
\n\n
\nPython API\n\n```python\nfrom crossref_local import search, get, count\n\n# Full-text search (22ms for 541 matches across 167M records)\nresults = search(\"hippocampal sharp wave ripples\")\nfor work in results:\n print(f\"{work.title} ({work.year})\")\n\n# Get by DOI\nwork = get(\"10.1126/science.aax0758\")\nprint(work.citation())\n\n# Count matches\nn = count(\"machine learning\") # 477,922 matches\n```\n\nAsync API:\n```python\nfrom crossref_local import aio\n\nasync def main():\n counts = await aio.count_many([\"CRISPR\", \"neural network\", \"climate\"])\n results = await aio.search(\"machine learning\")\n```\n\n
\n\n
\nCLI\n\n```bash\ncrossref-local search \"CRISPR genome editing\" -n 5\ncrossref-local search-by-doi 10.1038/nature12373\ncrossref-local status # Configuration and database stats\n```\n\nWith abstracts (`-a` flag):\n```\n$ crossref-local search \"RS-1 enhances CRISPR\" -n 1 -a\n\nFound 4 matches in 128.4ms\n\n1. RS-1 enhances CRISPR/Cas9- and TALEN-mediated knock-in efficiency (2016)\n DOI: 10.1038/ncomms10548\n Journal: Nature Communications\n Abstract: Zinc-finger nuclease, transcription activator-like effector nuclease\n and CRISPR/Cas9 are becoming major tools for genome editing...\n```\n\n
\n\n
\nHTTP API\n\nStart the FastAPI server:\n```bash\ncrossref-local relay --host 0.0.0.0 --port 31291\n```\n\nEndpoints:\n```bash\n# Search works (FTS5)\ncurl \"http://localhost:31291/works?q=CRISPR&limit=10\"\n\n# Get by DOI\ncurl \"http://localhost:31291/works/10.1038/nature12373\"\n\n# Batch DOI lookup\ncurl -X POST \"http://localhost:31291/works/batch\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"dois\": [\"10.1038/nature12373\", \"10.1126/science.aax0758\"]}'\n\n# Citation endpoints\ncurl \"http://localhost:31291/citations/10.1038/nature12373/citing\"\ncurl \"http://localhost:31291/citations/10.1038/nature12373/cited\"\ncurl \"http://localhost:31291/citations/10.1038/nature12373/count\"\n\n# Collection endpoints\ncurl \"http://localhost:31291/collections\"\ncurl -X POST \"http://localhost:31291/collections\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"name\": \"my_papers\", \"query\": \"CRISPR\", \"limit\": 100}'\ncurl \"http://localhost:31291/collections/my_papers/download?format=bibtex\"\n\n# Database info\ncurl \"http://localhost:31291/info\"\n```\n\nHTTP mode (connect to running server):\n```bash\n# On local machine (if server is remote)\nssh -L 31291:127.0.0.1:31291 your-server\n\n# Python client\nfrom crossref_local import configure_http\nconfigure_http(\"http://localhost:31291\")\n\n# Or via CLI\ncrossref-local --http search \"CRISPR\"\n```\n\n
\n\n
\nMCP Server\n\nRun as MCP (Model Context Protocol) server:\n```bash\ncrossref-local mcp start\n```\n\nLocal MCP client configuration:\n```json\n{\n \"mcpServers\": {\n \"crossref-local\": {\n \"command\": \"crossref-local\",\n \"args\": [\"mcp\", \"start\"],\n \"env\": {\n \"CROSSREF_LOCAL_DB\": \"/path/to/crossref.db\"\n }\n }\n }\n}\n```\n\nRemote MCP via HTTP (recommended):\n```bash\n# On server: start persistent MCP server\ncrossref-local mcp start -t http --host 0.0.0.0 --port 8082\n```\n```json\n{\n \"mcpServers\": {\n \"crossref-remote\": {\n \"url\": \"http://your-server:8082/mcp\"\n }\n }\n}\n```\n\nDiagnose setup:\n```bash\ncrossref-local mcp doctor # Check dependencies and database\ncrossref-local mcp list-tools # Show available MCP tools\ncrossref-local mcp installation # Show client config examples\n```\n\nSee [docs/remote-deployment.md](docs/remote-deployment.md) for systemd and Docker setup.\n\nAvailable tools:\n- `search` - Full-text search across 167M+ papers\n- `search_by_doi` - Get paper by DOI\n- `enrich_dois` - Add citation counts and references to DOIs\n- `status` - Database statistics\n- `cache_*` - Paper collection management\n\n
\n\n
\nImpact Factor\n\n```python\nfrom crossref_local.impact_factor import ImpactFactorCalculator\n\nwith ImpactFactorCalculator() as calc:\n result = calc.calculate_impact_factor(\"Nature\", target_year=2023)\n print(f\"IF: {result['impact_factor']:.3f}\") # 54.067\n```\n\n| Journal | IF 2023 |\n|---------|---------|\n| Nature | 54.07 |\n| Science | 46.17 |\n| Cell | 54.01 |\n| PLOS ONE | 3.37 |\n\n
\n\n
\nCitation Network\n\n```python\nfrom crossref_local import get_citing, get_cited, CitationNetwork\n\nciting = get_citing(\"10.1038/nature12373\") # 1539 papers\ncited = get_cited(\"10.1038/nature12373\")\n\n# Build visualization (like Connected Papers)\nnetwork = CitationNetwork(\"10.1038/nature12373\", depth=2)\nnetwork.save_html(\"citation_network.html\") # requires: pip install crossref-local[viz]\n```\n\n
\n\n
\nPerformance\n\n| Query | Matches | Time |\n|-------|---------|------|\n| `hippocampal sharp wave ripples` | 541 | 22ms |\n| `machine learning` | 477,922 | 113ms |\n| `CRISPR genome editing` | 12,170 | 257ms |\n\nSearching 167M records in milliseconds via FTS5.\n\n
\n\n
\nRelated Projects\n\n**[openalex-local](https://github.com/ywatanabe1989/openalex-local)** - Sister project with OpenAlex data:\n\n| Feature | crossref-local | openalex-local |\n|---------|----------------|----------------|\n| Works | 167M | 284M |\n| Abstracts | ~21% | ~45-60% |\n| Update frequency | Real-time | Monthly |\n| DOI authority | ✓ (source) | Uses CrossRef |\n| Citations | Raw references | Linked works |\n| Concepts/Topics | ❌ | ✓ |\n| Author IDs | ❌ | ✓ |\n| Best for | DOI lookup, raw refs | Semantic search |\n\n**When to use CrossRef**: Real-time DOI updates, raw reference parsing, authoritative metadata.\n**When to use OpenAlex**: Semantic search, citation analysis, topic discovery.\n\n
\n\n## Installation\n\n```bash\npip install crossref-local # core\npip install crossref-local[mcp] # + MCP server\n```\n\nFrom source:\n```bash\ngit clone https://github.com/ywatanabe1989/crossref-local\ncd crossref-local && make install\n```\n\n## 4 Interfaces\n\n
\nPython API\n\n```python\nfrom crossref_local import search, get, count\n\n# Full-text search (22ms for 541 matches across 167M records)\nresults = search(\"hippocampal sharp wave ripples\")\nfor work in results:\n print(f\"{work.title} ({work.year})\")\n\n# Get by DOI\nwork = get(\"10.1126/science.aax0758\")\nprint(work.citation())\n\n# Count matches\nn = count(\"machine learning\") # 477,922 matches\n```\n\n
\n\n
\nCLI\n\n```bash\ncrossref-local search \"CRISPR genome editing\" -n 5\ncrossref-local search-by-doi 10.1038/nature12373\ncrossref-local status # Configuration and database stats\n```\n\n
\n\n
\nHTTP API\n\nSee the [HTTP API section](#http-api) above for all endpoints.\n\n
\n\n
\nMCP Server\n\nSee the [MCP Server section](#mcp-server) above for configuration.\n\n
\n\n## Part of SciTeX\n\n`crossref-local` is part of [**SciTeX**](https://scitex.ai).\n\n>Four Freedoms for Research\n>\n>0. The freedom to **run** your research anywhere — your machine, your terms.\n>1. The freedom to **study** how every step works — from raw data to final manuscript.\n>2. The freedom to **redistribute** your workflows, not just your papers.\n>3. The freedom to **modify** any module and share improvements with the community.\n>\n>AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.\n\n---\n\n

\n \"SciTeX\"\n

"}