{"id":"d02963ca-19da-40b8-aeae-a85d11097322","slug":"polydev-swe-bench","kind":"mcp","name":"polydev-swe-bench","summary":"SWE-bench evaluation framework demonstrating multi-model consultation improves accuracy","owner":"backspacevenkat","repoUrl":"https://github.com/backspacevenkat/polydev-swe-bench","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2025-12-16T21:22:42.000Z","addedAt":"2026-10-21T20:13:12.337Z","logoUrl":"https://avatars.githubusercontent.com/u/174402739?u=4ed4da9cc612ee73ad315a00afc0978d66bec5c9&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["utilities"],"tags":[],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-23T07:01:11.921Z","sample":false,"url":"https://conduits.dev/tools/polydev-swe-bench","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Matching Frontier Code Agents with Lightweight Models via Multi-Model Consultation\n\n[![SWE-bench Verified](https://img.shields.io/badge/SWE--bench%20Verified-66.6%25%20(single)%20%7C%2074.6%25%20(Resolve@2)-brightgreen)](https://www.swebench.com/)\n[![arXiv](https://img.shields.io/badge/arXiv-2501.XXXXX-b31b1b.svg)](https://arxiv.org/abs/2501.XXXXX)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n\n**Can inference-time compute substitute for model scale?** We demonstrate that Claude Haiku 4.5 (a lightweight model) achieves **66.6% on SWE-bench Verified** as a single policy, and **74.6% Resolve@2 (oracle)** when taking the best of two independent policies—matching Claude 4.5 Opus (74.4%)—when augmented with extended agent turns, large thinking budgets, and multi-model consultation.\n\n## Key Results\n\n| Approach | Resolution Rate | Cost/Resolved |\n|----------|-----------------|---------------|\n| Baseline (Claude Haiku 4.5) | 64.6% | $0.18 |\n| Polydev (+ Multi-Model) | 66.6% | $0.24 |\n| **Resolve@2 (oracle)†** | **74.6%** | **$0.37** |\n| Claude 4.5 Opus (reference) | 74.4% | $0.97 |\n\n†Resolve@2 (oracle): Best result from two independent Haiku 4.5 policies (baseline + Polydev). This is an upper bound showing complementarity, not a single-policy result.\n\n**Key Finding:** Single-agent and multi-model approaches have only **76% overlap** in solved instances—24% of Resolve@2 successes come from one approach succeeding where the other failed.\n\n## Quick Start\n\n```bash\n# Clone and install\ngit clone https://github.com/backspacevenkat/polydev-swe-bench.git\ncd polydev-swe-bench\npip install -r requirements.txt\n\n# Set API key\nexport ANTHROPIC_API_KEY=\"your-key\"\n\n# Run evaluation (validation set)\npython scripts/swe_bench_baseline.py --mode validation --workers 4\n```\n\n## Repository Structure\n\n```\npolydev-swe-bench/\n├── paper/                          # Research paper\n│   ├── arxiv_paper.tex             # Main paper (LaTeX, for arXiv submission)\n│   └── ARXIV_PAPER.md              # Paper in Markdown format (readable)\n├── scripts/                        # Evaluation scripts\n│   ├── swe_bench_baseline.py       # Baseline agent\n│   └── swe_bench_polydev.py        # Polydev-enhanced agent\n├── agent_v3/                       # Agent implementation\n│   ├── agent.py                    # Main agent logic\n│   └── consultation.py             # Multi-model consultation\n├── results/                        # Evaluation results\n│   ├── baseline/                   # Baseline predictions & trajectories\n│   └── polydev/                    # Polydev predictions & trajectories\n├── submission/                     # SWE-bench leaderboard submission\n│   └── 20251227_hybrid-ensemble-haiku/\n│       ├── all_preds.jsonl         # Final predictions\n│       └── trajs/                  # Reasoning trajectories\n└── FINAL_RESULTS.json              # Complete evaluation results\n```\n\n## Methodology\n\n### Inference-Time Compute Dimensions\n\nWe identify three dimensions of inference-time investment:\n\n1. **Agent Turns** (up to 250): More iterations for exploration and refinement\n2. **Extended Thinking** (128K tokens): Large reasoning budget per turn\n3. **Model Consultation**: Querying GPT 5.2 Codex and Gemini 3 Flash Preview\n\n### Dual-Policy Evaluation Strategy\n\n```\nProblem Statement ─┬─► [Baseline Path] ─► Patch A ─┐\n                   │   (Haiku alone)               ├─► Test Validation ─► Best Patch\n                   └─► [Polydev Path] ──► Patch B ─┘\n                       (Haiku + MCP)\n```\n\n## When Does Multi-Model Consultation Help?\n\n| Problem Characteristic | Consultation Helpful |\n|------------------------|---------------------|\n| Multi-file changes | 78.2% |\n| Ambiguous requirements | 84.7% |\n| Single-file change | 61.4% |\n| Clear problem statement | 65.2% |\n\n**Takeaway:** Consultation is most valuable for complex, multi-file changes and ambiguous problem statements.\n\n## Complementarity Analysis\n\n| Category | Count | Description |\n|----------|-------|-------------|\n| Both solved | 283 | Core overlap (76%) |\n| **Baseline only** | 40 | Simple fixes where consultation added noise |\n| **Polydev only** | 50 | Complex issues where consultation helped |\n| Neither solved | 127 | Remaining failures |\n\n## Cost Analysis\n\n| Approach | Total Cost | Cost/Instance | Cost/Resolved |\n|----------|------------|---------------|---------------|\n| Baseline only | $57.76 | $0.12 | $0.18 |\n| Polydev only | $78.58 | $0.16 | $0.24 |\n| **Resolve@2 (both)** | **$136.34** | **$0.27** | **$0.37** |\n\n**Note:** Resolve@2 runs two full pipelines. Cost-effective vs. frontier models ($0.97 for Opus) but requires double compute. Effective input rates are lower than list prices due to prompt caching (~90% cache hit rate).\n\n## Reproducibility\n\n### Environment\n- **Benchmark:** SWE-bench Verified (500 instances)\n- **Base Model:** Claude Haiku 4.5 (`claude-haiku-4-5-20251001`)\n- **Thinking Budget:** 128,000 tokens\n- **Max Turns:** 250\n- **Evaluation Period:** December 25-27, 2025\n\n### Data Availability\n- **Predictions:** `submission/20251227_hybrid-ensemble-haiku/all_preds.jsonl`\n- **Trajectories:** `submission/20251227_hybrid-ensemble-haiku/trajs/`\n- **Metrics:** `results/baseline/metrics.jsonl`, `results/polydev/metrics.jsonl`\n\n## Citation\n\n```bibtex\n@article{ghanta2026matching,\n  title={Matching Frontier Code Agents with Lightweight Models\n         via Multi-Model Consultation},\n  author={Ghanta, Venkata Subrhmanyam and Paladugu, Pujitha Sri Lakshmi},\n  journal={arXiv preprint arXiv:2501.XXXXX},\n  year={2026},\n  url={https://github.com/backspacevenkat/polydev-swe-bench}\n}\n```\n\n## Authors\n\n- **Venkata Subrhmanyam Ghanta** - Arizona State University & Polydev AI (vsghanta@asu.edu)\n- **Pujitha Sri Lakshmi Paladugu** - Microsoft (pupaladu@microsoft.com)\n\n## Related Work\n\n- [SWE-bench](https://www.swebench.com/) - The benchmark\n- [Polydev](https://polydev.ai) - Multi-model consultation platform\n- [Model Context Protocol](https://modelcontextprotocol.io/) - MCP specification\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Acknowledgments\n\n- Anthropic for Claude API access\n- Princeton NLP for SWE-bench benchmark\n- OpenAI and Google for consultation model APIs\n"}