{"id":"a38744bb-0238-4ada-8bd0-f3e5251d55c3","slug":"github-activity-agent","kind":"mcp","name":"github-activity-agent","summary":"Verifiable NL2SQL agent: self-hosted MCP server, 3-config A/B/C experiment, deterministic offline grading, re-run noise audit (DeepSeek + DuckDB)","owner":"meiming2019-web","repoUrl":"https://github.com/meiming2019-web/github-activity-agent","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-10-23T12:15:05.000Z","addedAt":"2026-10-28T09:29:45.910Z","logoUrl":"https://avatars.githubusercontent.com/u/245770851?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["databases","cloud-infra","developer-tools"],"tags":["agent-evaluation","deepseek","deepseek-api","duckdb","llm-agents","nl2sql","text-to-sql"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-28T09:40:03.110Z","sample":false,"url":"https://conduits.dev/tools/github-activity-agent","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# GitHub Activity Query Agent\n\n[![tests](https://github.com/meiming2019-web/github-activity-agent/actions/workflows/tests.yml/badge.svg)](https://github.com/meiming2019-web/github-activity-agent/actions/workflows/tests.yml)\n\nAsk plain-English questions about GitHub activity — *\"which Rust projects gained\nthe most stars last week?\"* — and get answers computed from real GH Archive data\nby an agent that writes and executes its own SQL against DuckDB.\n\nThe point of this project is not the agent. It's **how the agent was measured**:\nthe configurations vary the scaffolding (loop, tools, context, error handling),\nwhile using a floating model alias. Preregistered endpoints, exploratory\nobservations, and post-registration adjudications are reported separately;\ninsufficient evidence is reported as such.\n\n## Quickstart\n\n```bash\npython3.12 -m venv .venv\n.venv/bin/pip install -r requirements.txt\n\n# frozen data snapshot (distributed via GitHub Release)\nmkdir -p data\n# download github.duckdb.gz from the v1.0-freeze Release into data/, then:\ngunzip data/github.duckdb.gz\nprintf '2a063cab4176b97cfd00ea8e92932c64346127ba1da4b120452fb48236c25812  data/github.duckdb\\n' | shasum -a 256 -c -\n\n# your key goes in .env (gitignored), never in code\necho 'DEEPSEEK_API_KEY=...' > .env\n\n# run the local UI (contacts DeepSeek; costs API usage)\n.venv/bin/streamlit run ui/app.py --server.address 127.0.0.1\n```\n\nRun the offline test suite (no credentials, no database needed — snapshot-bound\ntests skip themselves with an explicit reason):\n\n```bash\n.venv/bin/python -B -m pytest -q -p no:cacheprovider\n```\n\n## How it works\n\n```text\nQuestion\n  -> A: one SQL-generation request\n     B/C: bounded tool-calling loop (agent/loop.py)\n        -> C only: keyword glossary lookup (agent/context.py)\n        -> stdio MCP server (agent/mcp_server.py) -> read-only DuckDB\n        -> client-side submit_answer -> MCP run_query\n  -> submitted SQL + rows -> offline grader (eval/grader.py)\n```\n\n- Four read-only MCP tools: `list_tables`, `describe_table`, `sample_values`,\n  `run_query`. `submit_answer` and `lookup_glossary` are client-side.\n- SQL is checked against a sqlglot allowlist; DuckDB opens read-only; queries\n  are timed out and results bounded. These controls constrain the query\n  interface — they are not a complete sandbox.\n- Data: a frozen 14-day GH Archive snapshot (2026-10-08 → 2026-10-22 UTC),\n  28,203 events across 100 repositories. The metrics glossary\n  ([eval/glossary.md](eval/glossary.md)) defines every metric the agent may use.\n\n## Live mode\n\nAn opt-in live mode (`AGENT_MODE=live`) serves recent complete UTC days from an\nisolated database, with hourly incremental ingestion and dynamic date context.\nEval mode stays frozen and unchanged. See [docs/live-mode.md](docs/live-mode.md).\n\n## Results at a glance\n\nFive harness configurations over a frozen 300-question set. A/B were collected\nseparately from the C-family runs; the requested model alias was the same, but\nits provider version was not pinned. This table is descriptive:\n\n| Config | What it adds | Accuracy | Tokens per correct answer (derived) |\n|---|---|---:|---:|\n| A | one-shot, full glossary | 78.67% | 17.1K |\n| B | + bounded retry loop | 83.67% | 60.4K |\n| C | + on-demand glossary lookup | 89.00% | 70.3K |\n| C2.1 | + per-question date injection | 89.67% | 64.8K |\n| C3.1 | + actionable error feedback | 91.00% | 64.9K |\n\nThree things we learned:\n\n1. **Observed loop outcomes differ by stratum.** B recorded 63/90 vs 37/90\n   correct for A on advanced questions, but a net −11 in window-question\n   row-set correctness. These paired final verdicts do not identify a causal\n   mechanism or count date-boundary errors.\n2. **Per-question date injection improved the operational window-boundary\n   endpoint on this targeted distribution.** The preregistered comparison had\n   9:0 discordant pairs in favor of C2.1 (exact McNemar p = 0.00390625),\n   including an unparseable classification rather than an established semantic\n   error. C2.1 had 180/180 submitted window queries classified boundary-correct\n   in this run; post-registration adjudications and parser limits remain disclosed.\n3. **Evidence for error feedback remains insufficient.** C3.1 had 2:0\n   discordant recovery pairs in its favor over C2.1 (p = 0.5). The paired\n   subset is underpowered; this does not establish equivalence or no effect.\n\nOne more lesson, from the early rounds: across two 30-question batches the\nconfigurations ranked one way in the first, and the second batch reversed it. Reporting only\nthe first batch would have told a different story — which is why questions,\ndata, and results are frozen and every grading correction is disclosed in an\nerrata trail instead of quietly edited.\n\n## Why trust these numbers\n\n- **Preregistered** endpoints and analysis plan before any v4 run\n  ([eval/v4_preregister.md](eval/v4_preregister.md)).\n- **Blinded collection**: 1,080 v4 raw run records committed before grading.\n- **Explicit configuration differences**, with A/B collected separately from\n  the C-family runs; the five-configuration ladder is descriptive.\n- **Repeat-run flip substudy** to size run-to-run stochasticity.\n- **Hash ledger** over the snapshot, questions, results, and reports; raw run\n  records are committed (the exploratory log is stored gzipped).\n- Post-registration adjudications, parser limits, and negative results are\n  disclosed in the reports, not buried. Full account:\n  [eval/v4_report.md](eval/v4_report.md) ·\n  [eval/v4_ab_exploratory.md](eval/v4_ab_exploratory.md).\n\n## Limitations\n\n- One frozen snapshot of 100 top repositories; a templated question\n  distribution. This does not establish free-form NL robustness.\n- One model, one provider; `deepseek-chat`/`deepseek-flash` are floating\n  aliases, so provider-side drift between run days cannot be ruled out.\n- Accuracy comparisons other than the two preregistered endpoints are\n  descriptive only, including the ladder table above. Operational\n  post-registration adjudications and parser limitations remain disclosed\n  for the endpoints; secondary confidence intervals are descriptive.\n\n## Documentation map\n\n| Want to know | Read |\n|---|---|\n| What was preregistered, and the claim language allowed by it | [eval/v4_preregister.md](eval/v4_preregister.md) |\n| Full v4 results, guardrails, adjudications, sensitivity analyses | [eval/v4_report.md](eval/v4_report.md) |\n| The exploratory A/B runs (kept out of v4's claims) | [eval/v4_ab_exploratory.md](eval/v4_ab_exploratory.md) |\n| Earlier rounds: 60-question ablation, mechanism probes, re-run audit | [eval/results/summary.md](eval/results/summary.md), [eval/c2_on_v2_regression.md](eval/c2_on_v2_regression.md), [eval/c3_on_v2.md](eval/c3_on_v2.md) |\n| Metric and date-semantics definitions | [eval/glossary.md](eval/glossary.md) |\n| Live mode: windows, ingestion, scheduling | [docs/live-mode.md](docs/live-mode.md) |\n| Rebuilding the snapshot, CI behavior, full command reference | [docs/REPRODUCING.md](docs/REPRODUCING.md) |\n\n## References\n\n- Yao et al., [τ-bench](https://arxiv.org/abs/2406.12045) — its `pass^k`\n  framing motivates distinguishing success on one trial from reliability\n  across trials.\n- Li et al., [BIRD](https://arxiv.org/abs/2305.03111) — motivates attention to\n  database values and external knowledge; this workload is not a BIRD\n  evaluation."}