{"id":"a48cffa2-02c3-4d67-a7a8-9ac31c642e45","slug":"discovery-agents","kind":"mcp","name":"discovery-agents","summary":"Production-grade multi-agent RAG workflow + an ML-infrastructure platform (PyTorch/DDP, RAG, evals, guardrails, MCP). Keyless-by-default; ships a real, reproducible Banking77 retrieval benchmark.","owner":"deepanshumody","repoUrl":"https://github.com/deepanshumody/discovery-agents","license":"MIT","licenseTier":"permissive","trust":"community","stars":4,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-05-20T18:18:23.000Z","addedAt":"2026-10-21T08:53:08.031Z","logoUrl":"https://avatars.githubusercontent.com/u/39831386?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["search","memory-knowledge"],"tags":["agentic-workflows","guardrails","langgraph","llm","llm-evaluation","multi-agent","rag","react"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T14:02:01.078Z","sample":false,"url":"https://conduits.dev/tools/discovery-agents","language":"Python","transports":["stdio"],"install":{"pypi":"discovery-agents"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# discovery-agents\n\n[![CI](https://github.com/deepanshumody/discovery-agents/actions/workflows/ci.yml/badge.svg)](https://github.com/deepanshumody/discovery-agents/actions/workflows/ci.yml)\n[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![Checked with mypy](https://img.shields.io/badge/types-mypy%20strict-blue.svg)](https://mypy-lang.org/)\n[![Lint: ruff](https://img.shields.io/badge/lint-ruff-orange.svg)](https://docs.astral.sh/ruff/)\n\n> A production-grade, **LLM-powered multi-agent workflow** for enterprise product\n> discovery — with **RAG**, a **ReAct tool-use loop**, a **rigorous evaluation\n> harness**, **guardrails**, full **observability**, and an **MCP server**.\n> **Keyless by default, real on demand.**\n\nGiven a product brief and a corpus of customer evidence, a graph of nine specialized\nagents clusters the evidence into insights, generates several evidence-grounded\nproduct directions, critiques and scores them, selects a winner, and emits a\ncoding-agent-ready handoff packet — every step traced, evaluated, and guarded.\n\n**Why it's built this way.** You can run the whole thing in 30 seconds with **no\nAPI key** (a deterministic mock provider + in-memory retrieval). Set\n`ANTHROPIC_API_KEY` (or `--provider cohere`/`openai`) and the *same graph* runs on a\nfrontier model. CI, tests, and the eval regression gate all run keyless and\ndeterministically.\n\n## 30-second quickstart (no API key)\n\n```bash\npip install -e \".[dev]\"\ndiscovery-agents --output outputs/demo        # runs on the deterministic mock provider\ndiscovery-agents --eval                        # run the evaluation harness + regression gate\npytest -q                                      # deterministic, keyless test suite\n```\n\n`outputs/demo/` gets: `run_summary.md`, `coding_agent_handoff.md`, `canvas.html` (visual\ncanvas + eval dashboard + trace timeline), `agent_run.json`, and `agent_trace.md`.\n\n## Run it on a real frontier model\n\n```bash\npip install -e \".[anthropic]\"      # or .[cohere] / .[openai] / .[gemini]\nexport ANTHROPIC_API_KEY=...\ndiscovery-agents --provider anthropic --model claude-sonnet-4-6 --output outputs/live\n```\n\nProviders are pluggable behind one `LLMClient` protocol; if a key or SDK is missing the\nfactory logs a warning and falls back to the mock, so the workflow always runs. The\nreasoning agents (EvidenceInsight, Strategy, Ideation, Critique, Handoff, Memory) call the\nmodel and parse structured JSON; Canvas layout and Selection stay deterministic.\n\n## Live demo (FastAPI)\n\nA deployable web app runs the whole pipeline with the reasoning agents on a real LLM\n(default **Gemini**). Keyless by default; bring your own key per request, or set a host key\nfor a rate-limited number of free runs.\n\n```bash\npip install -e \".[web,gemini]\"\nuvicorn discovery_agents.webapp:app --reload          # open http://localhost:8000\n# optional: a host key enables free server-side runs (visitors can also paste their own)\nexport GEMINI_API_KEY=...                              # never commit this\n```\n\n- `GET /` — form: pick a dataset (enterprise sample / Banking77-style support), a provider,\n  an optional API key, and a goal override; renders the selected direction, all directions\n  with scores, the eval scorecard, decision memory, and trace totals.\n- `POST /api/run` — JSON in `{provider?, api_key?, dataset?, brief?, evidence?}`, JSON out\n  `{selected, directions, evals, decision_log, trace, provider_used}`.\n- `GET /healthz` — liveness.\n\nKeys are read from the request or the host env only — never logged, stored, or echoed.\nContainer: `docker build -f deploy/Dockerfile.web -t discovery-agents-web .` then\n`docker run --rm -p 8000:8000 -e GEMINI_API_KEY=$GEMINI_API_KEY discovery-agents-web`.\nSee [`docs/webapp.md`](docs/webapp.md) for deploy notes (Render/Fly/Cloud Run).\n\n## Architecture\n\n```mermaid\nflowchart LR\n  subgraph Interfaces\n    CLI[CLI] ; MCP[MCP server - stdio]\n  end\n  subgraph Runtime[\"Runtime: typed state machine + ReAct loop\"]\n    SM[StateMachine] --- LG[optional LangGraph adapter]\n  end\n  subgraph Agents[\"9 agents - LLM-backed\"]\n    EI[EvidenceInsight] --> ST[Strategy] --> ID[Ideation] --> CR[Critique] --> CA[Canvas] --> SE[Selection] --> HO[Handoff] --> ME[Memory]\n  end\n  subgraph Capabilities\n    LLM[LLM adapter: Claude/Cohere/GPT/Gemini/Mock]\n    RAG[Retrieval: embeddings + vector store]\n    TOOLS[Tools: evidence_search, web_search, calculator]\n    GR[Guardrails: input/output]\n    OBS[Observability: latency/tokens/cost]\n  end\n  EVAL[Eval harness: LLM-judge + metrics + regression gate]\n  CLI --> Runtime ; MCP --> Runtime\n  Runtime --> Agents\n  Agents -. uses .-> LLM ; Agents -. uses .-> RAG ; ID -. ReAct .-> TOOLS\n  Agents -. wrapped by .-> GR ; Agents -. emit .-> OBS\n  Runtime --> EVAL\n```\n\nSee [`docs/architecture.md`](docs/architecture.md) for the module-by-module breakdown.\n\n## What's inside\n\n- **Provider-agnostic LLM layer** (`llm/`) — one `LLMClient` protocol; deterministic\n  keyless `MockLLMClient` (default) + lazy Anthropic / Cohere / OpenAI adapters with\n  tool-calling and structured-output support.\n- **RAG** (`retrieval/`) — `Embedder` + deterministic `HashingEmbedder`, a `VectorStore`\n  (in-memory cosine + lazy Pinecone adapter), and an `EvidenceIndex` that returns cited\n  passages.\n- **ReAct runtime** (`runtime/`) — a typed `StateMachine` (topological ordering, cycle\n  detection, a trace span per node) and an `LLMAgent` plan-execute loop with tool use,\n  step budgets, and full step tracing; plus an optional **LangGraph** adapter that runs\n  the same graph.\n- **Tools** (`tools/`) — a `Tool` protocol + registry, a RAG `evidence_search`, a safe\n  AST `calculator`, and a mockable `web_search`. See [`docs/adding-a-tool.md`](docs/adding-a-tool.md).\n- **Guardrails** (`guardrails/`) — input (PII redaction, prompt-injection block) and\n  output (citation-required, groundedness, schema) checks, each recorded to the trace.\n- **Observability** (`observability/`) — a `Trace` of spans carrying latency, token\n  usage, and cost, rendered to Markdown and an HTML timeline.\n- **Evaluation** (`eval/`) — deterministic quality metrics + an **LLM-as-judge**\n  (faithfulness / relevance / helpfulness), aggregated into a scorecard and gated against\n  a committed `baseline.json` in CI. See [`docs/evals.md`](docs/evals.md).\n- **MCP server** (`mcp_server.py`) — exposes `discovery_run`, `evidence_search`, and\n  `eval_run` over stdio for Claude Desktop / Claude Code. See [`docs/mcp.md`](docs/mcp.md).\n\n## ML-infrastructure platform (`mlinfra/`)\n\nThe retrieval embedder isn't a toy — it can be **trained** by a production ML-infra\nplatform and served back through the same `Embedder` protocol:\n\n- **Distributed PyTorch training** — a custom SimCSE/InfoNCE training loop with **DDP**\n  (`gloo` on CPU, `nccl` on GPU) and **fault-tolerant, resumable checkpointing**\n  (atomic, SIGTERM-safe; resume reproduces the trajectory exactly — a tested invariant).\n- **GPU-native data I/O** — a multi-backend **tensor archive** (`Numpy` / **Zarr** /\n  **HDF5**) with a GPU-native `DataLoader`, plus an **I/O benchmark** (samples/s, MB/s,\n  p50/p95) and a resource profiler.\n- **Distributed data curation** — a pluggable `Executor` (Local / **Dask**) that tokenizes\n  and shards a corpus into the archive.\n- **MLOps** — experiment/artifact tracking (**MLflow** or a keyless JSON tracker), a\n  **Dockerfile**, `docker-compose`, and a **Kubernetes** Indexed-Job training manifest.\n\n### Benchmark result — it actually works\n\nOn a **real** task (Banking77 intent retrieval, 9,993 train / 3,076 test, 77 intents; relevant\n= same intent), the trained embedder **beats the lexical baseline** — a measured, reproducible\nresult, not a demo ([`benchmark/RESULTS.md`](benchmark/RESULTS.md)):\n\n| embedder | hit@1 | hit@5 | MRR | mAP |\n|---|---|---|---|---|\n| hashing (lexical baseline) | 0.769 | 0.922 | 0.835 | 0.503 |\n| **torch (supervised contrastive)** | **0.830** | 0.911 | **0.865** | **0.775** |\n| sentence-transformers (reference) | 0.921 | 0.970 | 0.942 | 0.842 |\n\n`hit@k` = fraction of queries with ≥1 same-intent neighbor in top-k (success@k). The trained\nmodel wins hit@1 (+6pts), MRR, and **mAP (+27pts)** over lexical (which edges it out on\nhit@5/@10); a pretrained `sentence-transformers` model is the reference upper bound. Reproduce\nin ~80s on CPU:\n\n```bash\npip install -e \".[ml,dask,benchmark,st]\"\npython -m discovery_agents.mlinfra.cli benchmark --full --with-st   # writes benchmark/RESULTS.md\npython -m discovery_agents.mlinfra.cli train --smoke                # curate -> train -> export, CPU\nDISCOVERY_EMBEDDER=torch discovery-agents                           # the agent RAG uses the trained embedder\ndiscovery-agents --dataset banking77                                # run the agents on real support messages\n```\n\nTrains **CPU-first**, but is written for multi-GPU / petabyte scale. Full details in\n[`docs/ml-platform.md`](docs/ml-platform.md).\n\n## Capabilities at a glance\n\n| Capability | Where |\n|---|---|\n| Production engineering (typed, tested, observable, CI) | strict `mypy`, `ruff`, `pytest`, GitHub Actions, `observability/` |\n| Agentic architectures (ReAct / plan-execute, tools/APIs) | `runtime/agent.py`, `tools/` |\n| Provider-agnostic LLM layer + RAG + vector store + LangGraph | `llm/`, `retrieval/`, `runtime/langgraph_adapter.py` |\n| Rigorous evaluation (accuracy / safety / latency) | `eval/` harness + LLM-judge + CI regression gate |\n| Reliable, observable, safe, auditable | `guardrails/`, `observability/`, decision memory |\n\n## Project layout\n\n```\nsrc/discovery_agents/\n  llm/            # LLMClient protocol, mock + Anthropic/Cohere/OpenAI/Gemini adapters, factory\n  retrieval/      # embeddings, vector store, evidence index (RAG)\n  tools/          # Tool protocol + registry; evidence_search, calculator, web_search\n  runtime/        # state machine, ReAct agent loop, discovery graph, LangGraph adapter\n  agents/         # the 9 discovery agents (LLM-backed, with deterministic baselines)\n  guardrails/     # input/output safety checks + pipeline\n  observability/  # trace spans (latency/tokens/cost), cost table, HTML report\n  eval/           # metrics, LLM-as-judge, harness, committed baseline.json\n  config.py       # RunConfig (provider/model/flags), env-driven\n  pipeline.py     # builds the graph + capabilities and runs it\n  cli.py          # discovery-agents entry point\n  webapp.py       # FastAPI live demo (UI + /api/run + /healthz); webdata.py = demo datasets\n  mcp_server.py   # MCP server (discovery-agents-mcp)\ntests/            # deterministic, keyless tests (+ an ml-infra suite under tests/mlinfra)\ndocs/             # architecture, evals, adding-a-tool, mcp + design specs\n```\n\n## Development\n\n```bash\nruff check src tests && ruff format src tests   # lint + format\nmypy                                            # strict type check\npytest -q                                       # tests (keyless, deterministic)\n```\n\nCI runs all of the above plus the eval regression gate on Python 3.9–3.12. See\n[`CONTRIBUTING.md`](CONTRIBUTING.md).\n\n## License\n\nMIT — see [`LICENSE`](LICENSE).\n"}