{"id":"3d49e86a-b7d0-44b6-9f77-b78b43f94cb2","slug":"multi-agent-workflow-lab","kind":"mcp","name":"multi-agent-workflow-lab","summary":"Testing and observability for multi-agent delegation, MCP tools, permissions, sandboxed actions, prompts, and workflow replay.","owner":"christiangrey922","repoUrl":"https://github.com/christiangrey922/multi-agent-workflow-lab","license":"MIT","licenseTier":"permissive","trust":"community","stars":86,"match":null,"fewStars":false,"format":"mcp","createdAt":null,"addedAt":"2026-10-14T05:15:15.458Z","logoUrl":"https://avatars.githubusercontent.com/u/201748870?u=ae071717d2bbf0b935be30e321c82590fcfbad6d&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["developer-tools","cloud-infra","memory-knowledge"],"tags":["agent-orchestration","agent-security","agent-testing","ai-agent","ai-agents","ai-security","multi-agent"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-14T05:24:26.229Z","sample":false,"url":"https://conduits.dev/tools/multi-agent-workflow-lab","language":"TypeScript","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Multi-Agent Workflow Lab\n\nAn open-source testing and observability framework for multi-agent delegation, tool execution, MCP workflows, permissions, sandboxed actions, prompts, and runtime behavior.\n\n> **Status:** Experimental — v0.1.0 release candidate. Suitable for local development, evaluation, and policy testing; not production-hardened infrastructure.\n\n## Why this exists\n\nMost model evaluation stops at `input → model → output`. Multi-agent systems add behavior that a final answer cannot explain:\n\n- Which agent delegated a task, to whom, and why?\n- What context, permissions, tools, MCP servers, and budget did the child receive?\n- Did an agent attempt privilege escalation, repeat work, or enter a loop?\n- Was the delegation efficient, and was the child result actually integrated?\n- Can the run be inspected, compared, or replayed without repeating side effects?\n\nMAWL makes those decisions explicit, policy-controlled, and traceable. Prompts are versioned assets; delegation is a first-class runtime event; deterministic rules evaluate behavior independently from optional model judges.\n\n## Key capabilities\n\n| Area                     | What is implemented                                                                                                                        |\n| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |\n| Agent runtime            | Typed model actions, task limits, cancellation, retries, and deterministic mock execution                                                  |\n| Delegation               | Parent/child task graph, target and capability checks, depth/fan-out limits, loop detection, and result integration events                 |\n| Tool execution           | Registry-based invocation with schema validation, allowlists, permissions, policy, optional approval, redaction, timeout, and audit        |\n| MCP                      | Official stdio and Streamable HTTP client adapters plus an in-memory mock connector; MCP data remains untrusted                            |\n| Sandbox                  | Restricted local-process adapter and optional Docker provider interface with auditable lifecycle events                                    |\n| Identity and permissions | Runtime-issued agent identity, local development auth, deny-by-default permission and contextual policy engines                            |\n| Prompt system            | Semantic versions, SHA-256 hashes, strict variables, trusted/untrusted provenance, and deterministic prompt-injection signals              |\n| Input parsing            | JSON, YAML, Markdown, text, and structured-task normalization                                                                              |\n| Observability            | Append-only events, SQLite/JSONL storage, structured logs, task traces, Mermaid delegation graphs, metrics, budgets, and monitors          |\n| Evaluation               | Nine deterministic rules, eight diagnostic dimensions, assertions, YAML specs, run comparison, prompt regression, and optional model judge |\n| Replay                   | Exact reconstruction, dry run, model rerun, and guarded tool rerun                                                                         |\n| Security testing         | Prompt injection, escalation, malicious MCP/tool output, secret leakage, traversal, sandbox, recursion, and resource-exhaustion cases      |\n\n## Architecture\n\n```mermaid\nflowchart TD\n  Input[\"User or test input\"] --> Parser[\"Input parsers\"]\n  Parser --> Runtime[\"Agent runtime + task graph\"]\n  Runtime --> Delegation[\"Delegation engine\"]\n  Delegation --> Guards[\"Identity + permission + policy\"]\n  Guards --> Agents[\"Child agents\"]\n  Guards --> Tools[\"Tool executor\"]\n  Tools --> MCP[\"MCP connectors\"]\n  Tools --> Sandbox[\"Sandbox providers\"]\n  Agents --> Events[\"Runtime events\"]\n  MCP --> Events\n  Sandbox --> Events\n  Events --> Observe[\"Trace + metrics + monitors\"]\n  Events --> Evaluate[\"Rules + optional judge + replay\"]\n```\n\n# Quick Start\n\nRequirements: Node.js 22 or newer and pnpm 11.\n\nFrom a clone or an extracted release archive:\n\n```bash\ncd multi-agent-workflow-lab\npnpm install --frozen-lockfile\npnpm demo\n```\n\nThe demo requires no API key or paid service. A public clone URL can be added after the maintainer chooses and publishes the repository location.\n\n## What the demo shows\n\n`pnpm demo` runs a deterministic review chain through the CLI:\n\n```text\nstatus: completed\n\norchestrator\n  → researcher\n    → analyst\n      → reviewer\n        → evaluator\n\ndelegation score: 88.75/100\nworkflow completed\n```\n\nThe actual CLI output is JSON and includes the run ID, every task/agent/status, all eight evaluation dimensions, derived metrics, and the JSONL trace path.\n\n## Delegation observability\n\nA trace records behavior rather than only the final answer:\n\n```text\norchestrator\n  delegation.requested → researcher\n  delegation.created   context + permissions + budget + depth\n\nresearcher\n  tool.requested       → mcp.mock.search\n  policy.allowed       explicit decision evidence\n  tool.completed       size + redaction metadata\n\norchestrator\n  delegation.result.accepted → child task integrated\n```\n\nUse these commands after a run:\n\n```bash\npnpm mawl trace show <run-id>\npnpm mawl evaluate <run-id>\npnpm mawl graph <run-id>\npnpm mawl replay <run-id> dry-run\n```\n\nEvaluation scores are normalized diagnostics, not mathematical truth or a security guarantee.\n\n## Example workflows\n\n| Workflow                           | Purpose                                                    | Expected result                                |\n| ---------------------------------- | ---------------------------------------------------------- | ---------------------------------------------- |\n| `01-basic-delegation.yaml`         | Orchestrator delegates one bounded research task           | Completes                                      |\n| `02-parallel-research.yaml`        | Two research tasks fan out and an analyst joins them       | Completes                                      |\n| `03-review-chain.yaml`             | Researcher → analyst → reviewer → evaluator                | Completes                                      |\n| `04-permission-denied.yaml`        | Direct delegation to a non-allowlisted target              | Fails safely                                   |\n| `05-mcp-untrusted-output.yaml`     | Hostile MCP content is treated as data in the test harness | Completes safely                               |\n| `06-delegation-loop.yaml`          | Agent A → B → A recursion                                  | Fails safely                                   |\n| `07-human-approval.yaml`           | Approval-provider checkpoint behavior                      | Completes; approval logic is tested separately |\n| `08-budget-exceeded.yaml`          | Workflow-wide token budget exhaustion                      | Fails safely                                   |\n| `09-bad-vs-good-orchestrator.yaml` | Good baseline for comparative evaluation                   | Completes                                      |\n\nRun any positive example with:\n\n```bash\npnpm mawl run workflows/02-parallel-research.yaml\n```\n\nThe negative examples intentionally return a non-zero exit code when the requested action is blocked.\n\n### Good vs. bad orchestrator\n\nThe lab compares a bounded, least-privilege orchestration trace with a synthetic bad trace containing duplicate work, leaked context, escalated permission, and an unauthorized tool request. `RunComparator` and `DelegationEvaluator` report changes in:\n\n- agent selection and delegation edges;\n- task decomposition and depth;\n- context and permission minimization;\n- tool use, retries, tokens, runtime, and failures;\n- result integration and overall diagnostic score.\n\nThe executable coverage is in [`tests/lab.test.ts`](tests/lab.test.ts), with scenario inputs under [`scenarios/`](scenarios/).\n\n## CLI\n\nIn this workspace, prefix CLI arguments with `pnpm mawl`:\n\n```text\npnpm mawl run <workflow.yaml> [json-input]\npnpm mawl agents list\npnpm mawl workflow inspect <run-id>\npnpm mawl task inspect <task-id>\npnpm mawl inspect agent permissions <agent-id>\npnpm mawl inspect mcp server [server-id]\npnpm mawl trace show <run-id>\npnpm mawl evaluate <run-id>\npnpm mawl graph <run-id> [output.mmd]\npnpm mawl replay <run-id> [exact|model-rerun|tool-rerun|dry-run]\npnpm mawl compare <run-a> <run-b>\npnpm mawl test <workflow-test.yaml>\npnpm mawl prompts list\npnpm mawl prompts inspect <prompt-id> [version]\npnpm mawl permissions inspect <agent-id>\npnpm mawl doctor\n```\n\n`tool-rerun` blocks tool names or manifests classified as external side effects unless the embedding application supplies explicit permission for that exact tool. The CLI supplies no side-effect permissions.\n\n## Prompts are first-class assets\n\nThis is not a directory of free-form prompt snippets. YAML assets under [`prompts/`](prompts/) include:\n\n- stable ID and semantic version;\n- owner, type, purpose, and known risks;\n- declared input schema and expected output;\n- allowed actions and recommended capabilities;\n- content hash verified by `PromptRegistry`;\n- provenance and trust classification when assembled at runtime.\n\nRuntime and security policy layers remain separate from untrusted user, tool, MCP, and child-agent content. Prompt quality tests detect duplicate IDs, invalid versions, missing metadata, variable/placeholder errors, unexpected hashes, and undeclared references.\n\n## Security model\n\n- Permissions and contextual policies deny unmatched requests by default.\n- Agent execution identities are issued by the runtime and linked to workflow/task/session IDs.\n- Child context, authority, and budget are reduced from the parent.\n- MCP responses and model/tool/child output are untrusted data, not authority.\n- Secret references and output redaction limit accidental exposure.\n- Tool requests pass manifest resolution, validation, permission, policy, optional approval, output checks, and audit.\n- Delegation depth, ancestry, task count, retries, calls, tokens, output, and runtime are bounded.\n- Replay refuses external side effects without an explicit per-tool permission.\n\nThe restricted local sandbox is intended for development and policy testing. It is **not** equivalent to hardened container, VM, microVM, or kernel isolation, and network denial is best-effort. MAWL does not claim to be prompt-injection-proof, sandbox-escape-proof, or fully zero-trust. Read the [threat model](docs/threat-model.md) and [security policy](SECURITY.md).\n\n## Using a real model provider\n\nThe default path always uses `MockModelProvider`. An optional OpenAI-compatible chat-completions adapter is available in `@mawl/providers`:\n\n```bash\ncp .env.example .env\n```\n\nPopulate `MODEL_PROVIDER`, `MODEL_ENDPOINT`, and `MODEL_API_KEY` from a secret manager, then inject `OpenAICompatibleProvider` into the runtime. The CLI intentionally does not auto-load credentials. See [provider integration](docs/provider-integration.md).\n\n## Repository structure\n\n```text\nmulti-agent-workflow-lab/\n├── apps/                    CLI and runnable examples\n├── packages/\n│   ├── core/                Schemas and provider contracts\n│   ├── runtime/             Agent loop, tasks, delegation, scheduler, replay\n│   ├── evaluation/          Rules, assertions, scores, specs, comparison\n│   ├── observability/       Events, traces, graphs, metrics, budgets, monitors\n│   ├── tools/ mcp/ sandbox/ Controlled execution surfaces\n│   ├── auth/ permissions/ policy/ security/ secrets/\n│   └── prompts/ parsers/ providers/ storage/ testing/\n├── agents/                  Agent definitions\n├── prompts/                 Versioned prompt assets\n├── workflows/               Executable workflow examples\n├── examples/                Compatibility examples and guide\n├── scenarios/ redteam/      Deterministic failure and attack inputs\n├── tests/                   Unit, security, lab, and end-to-end tests\n├── config/                  Example pricing configuration\n└── docs/                    Architecture, operations, security, and release docs\n```\n\n## Testing and development\n\nAll default tests use local mock providers and require no external API access:\n\n```bash\npnpm install --frozen-lockfile\npnpm typecheck\npnpm lint\npnpm build\npnpm test\npnpm test:e2e\npnpm test:security\npnpm demo\npnpm mawl doctor\n```\n\nTo extend the framework, add YAML agents under `agents/`, versioned prompts under `prompts/`, workflows under `workflows/`, tool manifests through `ToolRegistry`, MCP adapters through `McpConnection`/`McpConnector`, rules through `RuleEvaluator`, and scenario coverage under `tests/` or `scenarios/`. See the [documentation index](docs/README.md) and [contributing guide](CONTRIBUTING.md).\n\n## Roadmap\n\n- distributed execution and atomic shared budgets;\n- hardened remote container or microVM sandbox providers;\n- OpenTelemetry exporters and remote observability storage;\n- additional model, identity, and MCP adapters;\n- workflow visualization UI and benchmark datasets.\n\n## Contributing, security, and license\n\nContributions to agents, prompts, tools, MCP connectors, workflow scenarios, evaluators, and security tests are welcome. Read [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md).\n\nReport vulnerabilities privately as described in [`SECURITY.md`](SECURITY.md). Do not place sensitive vulnerability details in public issues.\n\nLicensed under the [`MIT License`](LICENSE).\n"}