scitex-dataset

SciTeX Dataset (scitex-dataset)

---

Problem and Solution

#	Problem	Solution
1	Public dataset repositories balkanized -- OpenNeuro (BIDS) + DANDI (NWB) + PhysioNet (WFDB) + Zenodo (generic) + GEO / ChEMBL / ClinicalTrials — different APIs, auth, download tools	Unified fetcher -- stx.dataset.neuroscience.openneuro.fetch_all_datasets() same call shape across all; local FTS5 search across metadata
2	DataLad setup required — fetching one BIDS dataset means reading DataLad docs first; the barrier is tooling, not knowledge	One-line fetch -- no DataLad setup; the module handles auth, resumption, checksums transparently

Supported repositories

Domain	Repository	Description	Data Types
neuroscience	OpenNeuro	Open BIDS neuroimaging platform	MRI, EEG, MEG, iEEG, PET
neuroscience	DANDI	BRAIN Initiative archive (NWB)	Electrophysiology, Ophys
neuroscience	PhysioNet	Physiological signal databases	ECG, EEG, clinical data
neuroscience	GIN	G-Node Infrastructure (Gogs + git-annex)	iEEG, ephys, behaviour
general	Zenodo	General scientific data (CERN)	Any research data
general	Figshare	Research data sharing platform	Any research data
general	OpenML	Machine-learning datasets	Tabular ML benchmarks
general	HuggingFace Hub	ML datasets / models (on-demand)	Any
biology	GEO	Gene Expression Omnibus (NCBI)	Transcriptomics, microarray
pharmacology	MoleculeNet	Molecular ML benchmark suite	SMILES, properties
pharmacology	ChEMBL	Bioactivity database (EBI)	IC50/Ki/EC50 assays
medical	ClinicalTrials.gov	NIH study registry	Trial metadata

Demo

flowchart LR
    Q["query<br/>'eeg motor imagery'"] --> A["scitex_dataset"]
    A --> N["neuroscience/<br/>openneuro · dandi · physionet"]
    A --> B["biology/<br/>geo · figshare · zenodo"]
    A --> M["medical/<br/>clinicaltrials.gov"]
    A --> P["pharmacology/<br/>chembl · moleculenet"]
    A --> G["general/<br/>huggingface · openml"]
    N & B & M & P & G --> F["filter_results()"]
    F --> D["shared SciTeX store<br/>(database.py)"]
    D --> O["DataFrame · JSON · download"]

Installation

uv pip install "scitex-dataset[all]"

Requires Python >= 3.10.

MCP support: pip install scitex-dataset[mcp]

Extra	Provides
mcp	MCP server (scitex-dataset mcp ..., 20+ FastMCP tools)
huggingface	HuggingFace Hub fetch/search/download (huggingface-hub)
all	Everything above: uv pip install "scitex-dataset[all]"

When the host has multiple Python interpreters — e.g. a pre-built agent venv at /opt/venv-agent, a Spartan module load Python/3.11.3, or an existing ~/.venv you want to leave alone — point uv at the exact interpreter you want with --python:

# install into a specific venv WITHOUT touching project-local .venv
uv pip install --python /opt/venv-agent/bin/python "scitex-dataset[all]"

# one-shot CLI run under that interpreter
uv run --python /opt/venv-agent/bin/python scitex-dataset list-python-apis

# install the CLI as a tool, pinned to the same interpreter
uv tool install --python /opt/venv-agent/bin/python "scitex-dataset[mcp]"

# in a downstream script that bootstraps scitex-dataset, honour
# SCITEX_PYTHON if the caller sets it:
SCITEX_PYTHON="${SCITEX_PYTHON:-/opt/venv-agent/bin/python}"
uv pip install --python "$SCITEX_PYTHON" "scitex-dataset[all]"

The Makefile honours PYTHON=:

PYTHON=/opt/venv-agent/bin/python make install

See issue #38.

Architecture

flowchart LR
    U["user code / agent"] --> P["Python API<br/>_api.py"]
    U --> C["CLI<br/>scitex-dataset"]
    C --> D["dispatch<br/>_sources.py"]
    P --> D
    M["MCP server<br/>_mcp/"] --> D
    D --> N["domain fetchers<br/>neuroscience · biology · medical · pharmacology · general"]
    N --> F["filter_results()"]
    F --> S["shared SciTeX store<br/>database.py"]
    S --> O["DataFrame · JSON · download"]

Sub-packages group fetchers by scientific domain; each leaf module exposes a <source>_fetch(query, ...) entry point registered in _sources.py and re-exported from __init__.py.

Four Interfaces (Python · CLI · MCP · Skills)

from scitex_dataset import (
    openneuro_fetch, dandi_fetch, huggingface_search,
    filter_results, list_sources,
    db_build, db_search,
)

# 1) Fetch from any catalog source — every <src>_fetch alias is 1:1
#    with the dataset_<src>_fetch MCP tool.
records = openneuro_fetch(max_datasets=100)

# 2) Filter + rank in memory.
top = filter_results(
    records, modality="eeg", min_subjects=20,
    sort_by="downloads", limit=10,
)

# 3) Search HuggingFace Hub directly (on-demand, no catalog).
hf_hits = huggingface_search("biology", limit=20)

# 4) Build the full-text dataset index in the shared SciTeX store.
db_build()
db_search("Alzheimer EEG")

# 5) Inspect the supported sources.
list_sources()["count"]   # 11

Full API reference

scitex-dataset --help-recursive             # Show all commands

# Grammar: scitex-dataset <domain> <dataset> <action>
scitex-dataset neuroscience openneuro fetch -n 100 -o datasets.json -v
scitex-dataset neuroscience dandi fetch -n 50 -o dandi.json -v
scitex-dataset neuroscience physionet fetch -n 50 -v
scitex-dataset general zenodo fetch -q "neuroscience" -n 20

# HuggingFace (general/huggingface noun-group has 4 verbs)
scitex-dataset general huggingface fetch Anthropic/BioMysteryBench-full
scitex-dataset general huggingface search "biology" -n 20 --json

# Local database
scitex-dataset db build                     # index all catalog sources
scitex-dataset db search "epilepsy EEG"     # full-text search
scitex-dataset db show-stats                # show statistics

# Introspection
scitex-dataset list-python-apis -v          # list Python API tree
scitex-dataset mcp list-tools -v            # list MCP tools

Full CLI reference

AI agents can discover and query neuroscience datasets autonomously.

Tool	Description
dataset_list_sources	Enumerate the 11 supported sources
dataset_filter_results	Filter / rank fetched datasets in memory
dataset_<src>_fetch	One per catalog source (10 total)
dataset_huggingface_fetch / _search / _info / _download_file	HuggingFace family
dataset_db_build / _search / _show_stats	Full-text dataset index
dataset_skills_list / _get	Bundled skill pages

Table 2. 21 MCP tools across catalog fetchers, HuggingFace, the dataset index, and skill introspection. Every MCP tool has a matching public Python alias (e.g. scitex_dataset.openneuro_fetch).

scitex-dataset mcp start

Full MCP specification

Skills provide workflow-oriented guides that AI agents query to discover capabilities and usage patterns.

scitex-dataset skills list              # List available skill pages
scitex-dataset skills get SKILL         # Show main skill page
scitex-dev skills export --package scitex-dataset  # Export to Claude Code

Skill	Content
installation	pip install + extras + verify
quick-start	Search, fetch, sort across sources
python-api	Top-level exports, domain submodules, examples
cli-reference	CLI grammar, domains, flags, config precedence
mcp-tools	MCP tools for AI agents
env-vars	SCITEX_* environment variables
data-sources	All 11 supported repositories

Part of SciTeX

scitex-dataset is part of SciTeX. Install via the umbrella with pip install scitex[dataset] to use as scitex.dataset (Python) or scitex dataset ... (CLI).

import scitex
from scitex_dataset import fetch_all_datasets, format_dataset

@scitex.session
def main(logger=scitex.INJECTED):
    datasets = fetch_all_datasets(max_datasets=100, logger=logger)
    formatted = [format_dataset(ds) for ds in datasets]
    scitex.io.save(formatted, "openneuro_datasets.json")
    return 0

The SciTeX ecosystem follows the Four Freedoms for Research, inspired by the Free Software Definition:

Four Freedoms for Research

0. The freedom to run your research anywhere -- your machine, your terms.
1. The freedom to study how every step works -- from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 -- because we believe research infrastructure deserves the same freedoms as the software it runs on.

---