multi-agent-orchestration-mcp

Multi-Agent Orchestration System

A distributed multi-agent system demonstrating enterprise-grade orchestration, resilience patterns, and protocol-driven communication using the Model Context Protocol (MCP).

Architecture | Features | Quick Start | Docs

---

Overview

Seven autonomous AI agents communicate over JSON-RPC 2.0 to orchestrate, referee, and compete in a strategic league — demonstrating production-grade distributed computing patterns at scale.

The system tackles real-world challenges in multi-agent deployment: protocol compliance across service boundaries, concurrent task handling, fault tolerance with circuit breakers, and distributed state management — the same problems faced when building reliable LLM-based agent systems in production.

Highlights

588 tests across 5 categories (unit, integration, E2E, protocol, edge)	85%+ code coverage enforced by CI
7 autonomous agents communicating via 18 message types	12 Architecture Decision Records documenting design choices
Circuit breaker + exponential backoff for fault tolerance	Structured JSONL observability with distributed tracing
Jupyter research notebook with statistical analysis & visualizations	Full CI/CD pipeline with GitHub Actions

---

Architecture

                    ┌──────────────────────────────┐
                    │     League Manager (LM01)     │
                    │         Port 8000             │
                    │  ┌────────────────────────┐   │
                    │  │  Registration Service  │   │
                    │  │  Round-Robin Scheduler  │   │
                    │  │  Standings Engine       │   │
                    │  └────────────────────────┘   │
                    └───────────┬──────┬────────────┘
                                │      │
                  JSON-RPC 2.0  │      │  JSON-RPC 2.0
                                │      │
                 ┌──────────────┘      └──────────────┐
                 │                                    │
      ┌──────────▼───────────┐          ┌─────────────▼──────────┐
      │   Referee REF01      │          │   Referee REF02        │
      │   Port 8001          │          │   Port 8002            │
      │  ┌─────────────────┐ │          │  ┌─────────────────┐   │
      │  │ Match Conductor │ │          │  │ Match Conductor │   │
      │  │ Game Logic      │ │          │  │ Game Logic      │   │
      │  │ Timeout Enforce │ │          │  │ Timeout Enforce │   │
      │  └─────────────────┘ │          │  └─────────────────┘   │
      └──┬───┬─────┬───┬────┘          └──┬───┬─────┬───┬──────┘
         │   │     │   │                  │   │     │   │
         ▼   ▼     ▼   ▼                  ▼   ▼     ▼   ▼
      ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
      │ P01  │ │ P02  │ │ P03  │ │ P04  │
      │ 8101 │ │ 8102 │ │ 8103 │ │ 8104 │
      └──────┘ └──────┘ └──────┘ └──────┘
            Player Agents (Autonomous)

Agent	Role	Key Responsibilities
League Manager	Central Orchestrator	Agent registration, round-robin scheduling, standings, league lifecycle
Referees (x2)	Game Conductors	Match orchestration, rule enforcement, timeout handling, result reporting
Players (x4)	Autonomous Competitors	Strategy selection, match participation, history tracking

All communication uses league.v2 — a custom protocol built on JSON-RPC 2.0 with 18 message types, 18 error codes, token-based auth, and correlation IDs for distributed tracing.

---

Key Features

Resilience Engineering

• Exponential backoff retry (2s → 4s → 8s, max 3)
• Circuit breaker (5-failure threshold, 60s reset)
• Multi-level timeouts (5s / 10s / 30s per op)
• 18 error codes with retryable vs. terminal classification

Async Architecture

• Non-blocking I/O via httpx + FastAPI async
• 50+ concurrent matches without blocking
• Atomic writes (temp file + rename pattern)
• Thread-safe state management

Observability

• JSON Lines (JSONL) structured logging
• Correlation IDs across agent boundaries
• Per-agent, per-league, system log streams
• 100MB rotation with 5-gen backup

Data Management

• Repository pattern for persistence
• Pydantic v2 schema validation
• Automated retention policies
• Gzip archival (80% compression)

---

Tech Stack

Layer	Technology
Framework	FastAPI, Uvicorn, Pydantic v2
Communication	JSON-RPC 2.0, httpx (async), Model Context Protocol
Testing	pytest (588 tests), pytest-cov, pytest-asyncio
Code Quality	black, isort, flake8, mypy, pylint, pre-commit hooks
CI/CD	GitHub Actions (Python 3.10 / 3.11 matrix)
Research	Jupyter, NumPy, Pandas, Matplotlib, SciPy
Packaging	PEP 621 pyproject.toml, installable SDK wheel

---

Quick Start

# Clone and install
git clone https://github.com/igornazarenko434/multi-agent-orchestration-mcp.git
cd multi-agent-orchestration-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install -e SHARED/league_sdk

# Verify
python3 -c "from league_sdk import protocol, logger, retry; print('SDK ready')"

# Run the league
./scripts/start_league.sh        # Start all 7 agents
./scripts/check_health.sh        # Health check
./scripts/trigger_league_start.sh # Execute league
./scripts/query_standings.sh     # View results
./scripts/stop_league.sh         # Graceful shutdown

# Run tests (588 tests, 85% coverage gate)
PYTHONPATH=SHARED:$PYTHONPATH pytest tests/ --cov=SHARED/league_sdk --cov=agents --cov-fail-under=85 -v

---

Project Structure

.
├── SHARED/                        # Shared SDK & configuration
│   ├── league_sdk/                #   Core SDK: protocol, retry, logging, repositories, cleanup
│   └── config/                    #   JSON configs: system, agents, leagues, games
├── agents/                        # 7 autonomous agent implementations
│   ├── league_manager/            #   Central orchestrator (LM01)
│   ├── referee_REF01/             #   Game conductor with match logic, timeouts
│   ├── referee_REF02/             #   Second referee (DRY architecture)
│   ├── player_P01/                #   Reference player implementation
│   └── player_P02-P04/            #   Additional players (reuse PlayerAgent)
├── tests/                         # 588 tests across 50 files
│   ├── unit/                      #   29 files — component isolation
│   ├── integration/               #   11 files — multi-component flows
│   ├── e2e/                       #   4 files — full system validation
│   ├── protocol_compliance/       #   6 files — protocol correctness
│   └── edge_cases/                #   1 file — boundary conditions
├── doc/                           # 20,000+ lines of documentation
│   ├── architecture/              #   C4 model, 12 ADRs, thread safety analysis
│   ├── reference/                 #   API reference, error codes, data retention
│   ├── research_notes/            #   Jupyter notebook, plots, statistical analysis
│   └── guides/                    #   Developer, configuration, testing, deployment
├── scripts/                       # 13 operational bash scripts
└── .github/workflows/test.yml     # CI/CD: lint + type-check + test + coverage

---

Documentation

Document	Description
Architecture Guide	System design, protocol specs, agent roles, data flow
Configuration Guide	System, agent, league, and game configuration
Developer Guide	Setup, development workflow, adding agents, debugging
Testing Guide	Test strategy, running tests, coverage analysis
API Reference	18 MCP message types and JSON-RPC formats
Error Codes	E001-E018 with retry/terminal classification
Architecture Decisions	12 ADRs documenting every major design choice
Deployment Guide	Production deployment and release packaging

---

Research & Experiments

The project includes a Jupyter research notebook with data-driven analysis:

• Strategy comparison across agent decision models
• Timeout impact on match outcomes and system throughput
• Retry success rates under various failure scenarios
• Game theory modeling with LaTeX-formatted equations
• 7 visualization plots (Matplotlib + Seaborn)

Pre-rendered version: doc/research_notes/experiments.html

---

Contributing

See CONTRIBUTING.md for code style, testing requirements, and PR process.

pre-commit run --all-files                                          # Quality checks
PYTHONPATH=SHARED:$PYTHONPATH pytest tests/ --cov-fail-under=85     # Test suite

---

License

MIT License. See LICENSE for details.