scitex-scholar

SciTeX Scholar (scitex-scholar)

---

Problem and Solution

#	Problem	Solution
1	Literature search is balkanized -- CrossRef / OpenAlex / Semantic Scholar / arXiv / PubMed each have different APIs, rate limits, auth	Unified search -- scitex scholar search "topic" federates across all, deduplicates by DOI, returns ranked results
2	BibTeX gaps -- wild BibTeX misses abstracts / DOIs / impact factors; manuscript prep wastes hours	scitex scholar bibtex enrichment -- one call resolves DOIs, fetches abstracts, adds impact factors, normalizes formatting
3	Paywalled PDFs -- each journal needs institutional login; manual login-download-rename is the bottleneck	Browser-automation + OAuth -- persistent Chrome profile with stealth; scitex scholar fetch 10.1038/... grabs the PDF end-to-end

Problem

Literature management spans many tools and APIs: searching databases, resolving DOIs, downloading PDFs through institutional access, enriching BibTeX metadata, and keeping a reproducible, deduplicated library. Each step speaks a different library, auth flow, and data format.

Solution

scitex-scholar provides a unified workflow:

• Search across CrossRef, Semantic Scholar, PubMed, arXiv, and OpenAlex
• Resolve DOIs from titles; enrich BibTeX with abstracts, citation counts, impact factors (JCR 2024), PMIDs, and arXiv IDs
• Download PDFs through institutional access (OpenAthens / SSO) with Playwright browser automation
• Organize papers in a primary-hash library with per-project symlinks at ~/.scitex/scholar/library/. One-button maintenance via library refresh (reconcile → regenerate readable names → optional rsync to remote hosts)
• Highlight each sentence of a PDF by rhetorical role — claim, method, limitation, supportive citation, contradicting citation — via Claude
• Automate the same operations from the CLI, a Python API, or the SciTeX MCP server

Demo

flowchart LR
    Q["search query<br/>or .bib / DOI"] --> S["Scholar.search() /<br/>bibtex import"]
    S --> E["metadata_engines<br/>(CrossRef · OpenAlex · arXiv ·<br/>PubMed · Semantic Scholar)"]
    E --> P["Papers<br/>(deduped, enriched)"]
    P --> U["url_finder<br/>(translators + heuristics)"]
    U --> A["auth<br/>(OpenAthens / EZProxy /<br/>Shibboleth)"]
    A --> D["pdf_download<br/>(Playwright strategies)"]
    D --> L["~/.scitex/scholar/library/<br/>MASTER/&lt;HASH&gt;/"]
    P -.-> H["pdf_highlight<br/>(claim / method / limitation)"]
    L -.-> M["MCP: scholar_* tools<br/>(scitex serve)"]

Figure 1. End-to-end demo flow: one paper fetch call runs enrich, URL resolution, authentication, download, and storage into the primary-store library.

scitex-scholar paper fetch --doi 10.1038/... exercises the full chain in one call: enrich → resolve URL → authenticate → download → store under MASTER/<HASH>/ with metadata + PDF + per-project symlinks.

Installation

uv pip install "scitex-scholar[all]"

Install	Contents
pip install scitex-scholar	core: search, enrich, library (Playwright browser automation included)
pip install "scitex-scholar[all]"	everything: GUI server (django / scitex-app / scitex-ui), MCP server (fastmcp), PDF text extraction (pdfplumber), XLSX export (openpyxl), library watcher (watchdog), provenance hashing (scitex-clew)

Architecture

flowchart TB
    CLI["scitex-scholar CLI<br/>(noun-verb groups)"] --> CORE["core/<br/>Scholar · Paper · Papers"]
    MCP["MCP tools<br/>scholar_* via scitex serve"] --> CORE
    GUI["Django GUI<br/>_django/"] --> CORE
    CORE --> SEARCH["search_engines<br/>CrossRef · OpenAlex · arXiv ·<br/>PubMed · Semantic Scholar"]
    CORE --> META["metadata_engines<br/>DOI · abstracts · citations · IF"]
    SEARCH --> META
    META --> URL["url_finder + auth<br/>translators · OpenAthens / SSO"]
    URL --> PDF["pdf_download<br/>Playwright strategies"]
    PDF --> STORE["storage/<br/>primary store + project symlinks"]
    STORE --> HL["pdf_highlight<br/>claim / method / limitation"]
    STORE --> INT["integration/<br/>Zotero · Mendeley"]

Figure 2. Module architecture: the CLI, MCP, and GUI surfaces drive core/; federated search and enrichment feed URL discovery and authenticated download into the primary-store library.

The CLI is a thin layer over the Python API: every scitex-scholar <noun> <verb> command dispatches into one of core/, pipelines/, storage/, or auth/. The MCP server (_mcp/) exposes the same handlers as scholar_* tools consumed by the unified scitex serve server.

4 Interfaces

from scitex_scholar import Scholar, Paper, Papers, apply_filters, to_bibtex

scholar = Scholar()
papers = scholar.search("deep learning EEG", year_min=2020)   # auto-enriched
papers.save("results.bib")

# Filter + export
top = apply_filters(papers, min_citations=50, min_impact_factor=5.0)
print(to_bibtex(top))

Entry point: scitex-scholar <subcommand> (Click-based).

# Discover everything
scitex-scholar --help
scitex-scholar --help-recursive          # full overview, every leaf
scitex-scholar --version                  # or -V

# Paper(s)
scitex-scholar paper fetch --doi 10.1038/nature12373 --project demo
scitex-scholar paper fetch-batch --dois 10.1038/xxx --dois 10.1126/yyy --project demo --num-workers 4

# BibTeX file
scitex-scholar bibtex import --bibtex refs.bib --project demo --output refs.enriched.bib

# PDF post-processing
scitex-scholar pdf highlight paper.pdf

# Library — daily workflow
scitex-scholar library list                                   # all projects
scitex-scholar library list neurovista                        # one project, per-paper
scitex-scholar library open-urls neurovista --watch           # browser + auto-import
scitex-scholar library refresh neurovista                     # one-button maintenance
scitex-scholar library refresh neurovista --sync spartan      # +rsync push (repeatable)

# Library — manual PDF import (when DBs haven't indexed yet, or no auto-download)
scitex-scholar paper fetch --doi 10.1002/epi.70076 \
    --pdf-main ~/Downloads/Liu_2026.pdf \
    --pdf-supple ~/Downloads/MOESM1_ESM.pdf \
    --attachment ~/Downloads/dataset.csv \
    --project neurovista

# Library — layout / share / integrity
scitex-scholar library bind neurovista ~/proj/neurovista      # one symlink, no data move
scitex-scholar library export neurovista --format bibtex
scitex-scholar library audit-files --project neurovista
scitex-scholar library db build --dry-run
scitex-scholar library db audit --json

# Auth (institutional SSO — OpenAthens / EZProxy / Shibboleth)
scitex-scholar auth status              # exit 0 if any session valid, 1 otherwise
scitex-scholar auth login               # trigger SSO flow now (debug-friendly)
scitex-scholar auth logout -y           # clear cached cookies (--yes required)
scitex-scholar auth refresh             # logout + login

# MCP server
scitex-scholar mcp start
scitex-scholar mcp list-tools --json

# Shell completion
scitex-scholar install-shell-completion --shell bash
scitex-scholar print-shell-completion --shell bash

# Skills + Python API introspection
scitex-scholar dev skills list
scitex-scholar list-python-apis -v

Debugging the SSO automator

Every browser-automation step writes a screenshot + HTML pair to ~/.scitex/scholar/cache/engine/screenshots/ and ~/.scitex/browser/cache/debug/. When a selector breaks (e.g. an Okta UI refresh), ls -lt the artifact dirs to get a frame-by-frame storyboard — the screenshot shows what was rendered, the HTML shows what the locator was reasoning over. See _skills/scitex-browser/11_debugging-visuals.md for the full pattern.

Mutating verbs accept --dry-run and -y/--yes. Read verbs support --json. Common paper/bibtex flags: --browser-mode {stealth,interactive}, --chrome-profile NAME, --force.

Migration (1.3.0): the CLI moved to noun-verb groups. Old top-level commands (single, parallel, top-level bibtex --bibtex, highlight, link-project-tree, materialize, dematerialize, db) still work but emit a DeprecationWarning and will be removed in 1.4.0. See CHANGELOG.md for the full migration table.

The package ships MCP tool handlers consumed by the unified scitex serve server (tools prefixed scholar_*). A standalone server at scitex_scholar.mcp_server is still shipped but deprecated. See the Skills documentation for the full tool list.

Agent skill pages are published at scitex-scholar.readthedocs.io/en/latest/skills.html. The semantic-highlight skill documents the PDF-highlighting workflow.

Core API

Symbol	Purpose
Scholar	Main search / enrich / download / save interface
Paper, Papers	Single paper / collection with export methods
ScholarConfig	Paths, API keys, auto-enrich toggle, browser settings
apply_filters	Filter a Papers collection
to_bibtex, to_ris, to_endnote, to_text_citation	Export formats
generate_cite_key, make_citation_key	Deterministic BibTeX keys
CitationGraphBuilder, plot_citation_graph	Optional citation graph
pdf_highlight.highlight_pdf	Overlay semantic highlights on a PDF

Table 1. Public Python API symbols and their purposes.

Sources: core/, search_engines/, metadata_engines/, pdf_download/, pipelines/, browser/, auth/, storage/, pdf_highlight/, _mcp/.

Semantic PDF Highlighting

Overlay colour-coded highlights on a PDF that separate what the paper claims from its methods, self-admitted limitations, and stance toward related work. Highlights are standard PDF annotation objects placed on a copy of the source — the original bytes are unchanged and any viewer can show or strip them.

colour	category	meaning
green	focal_claim	what the paper clarifies, suggests, demonstrates
purple	focal_method	novel method, model, cohort, or analysis
red	focal_limitation	self-admitted caveat or threat to validity
blue	related_supportive	prior work whose finding supports the paper
orange	related_contradictive	prior work whose finding contradicts the paper

Table 2. Highlight colours and their rhetorical categories.

A compact colour legend + signature (model name, timestamp) is stamped in the lower-right corner of the last page. See docs for full details.

export ANTHROPIC_API_KEY=sk-ant-...
scitex-scholar pdf highlight paper.pdf        # sentence-level, Haiku, writes paper.highlighted.pdf
scitex-scholar pdf highlight paper.pdf --stub # offline keyword heuristic (no API calls)

from scitex_scholar.pdf_highlight import highlight_pdf
result = highlight_pdf("paper.pdf", output_path="paper.highlighted.pdf")
print(result.counts(), result.annotations_added)

Also exposed as the scholar_highlight_pdf MCP tool (unified scitex serve server) and as the semantic-highlight agent skill (see skills documentation).

Storage layout

~/.scitex/scholar/library/
├── MASTER/<HASH>/               # Canonical per-paper storage (metadata.json + PDF)
└── <project>/<human-label> -> ../MASTER/<HASH>

Cache and auth state live under ~/.scitex/scholar/cache/ (URL resolver, Chrome profiles, OpenAthens cookies). Override with SCITEX_DIR.

License

AGPL-3.0-only.

Part of SciTeX

scitex-scholar is part of SciTeX. Install via the umbrella with pip install scitex[scholar] to use as scitex.scholar (Python) or scitex scholar ... (CLI).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

---