{"id":"5cddbcc7-e535-43f3-a591-de0bd698876a","slug":"flakeguard","kind":"mcp","name":"FlakeGuard","summary":"Flaky test detection for CI, exposed to AI coding assistants over MCP — tells you whether a red test is a real bug or known-flaky","owner":"dhananjaya-hbc","repoUrl":"https://github.com/dhananjaya-hbc/FlakeGuard","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-09-20T20:18:21.000Z","addedAt":"2026-10-21T09:44:55.732Z","logoUrl":"https://avatars.githubusercontent.com/u/171498451?u=22bd7c4c6c26bb57e9e6910deed95f4550078b03&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["developer-tools","databases"],"tags":["ci","developer-tools","flaky-tests","golang","postgresql"],"runtime":"go","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-21T10:44:09.029Z","sample":false,"url":"https://conduits.dev/tools/flakeguard","language":"Go","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# FlakeGuard\n\nFlaky test detection for CI pipelines, exposed to AI coding assistants over MCP — so when a developer's test fails, their assistant can tell them whether it's a real bug or a known-flaky test, right inside the editor.\n\n## Contents\n\n- [The problem](#the-problem)\n- [How it works](#how-it-works)\n- [What you get: the three MCP tools](#what-you-get-the-three-mcp-tools)\n- [Prerequisites](#prerequisites)\n- [Quick start (Docker Compose)](#quick-start-docker-compose)\n- [Local development (without Docker)](#local-development-without-docker)\n- [Running the tests](#running-the-tests)\n- [Connecting to Claude Code](#connecting-to-claude-code)\n- [Worked example](#worked-example)\n- [Ingestion API reference](#ingestion-api-reference)\n- [GitHub Actions integration](#github-actions-integration)\n- [Environment variables](#environment-variables)\n- [Project structure](#project-structure)\n- [Language support](#language-support)\n- [License](#license)\n\n## The problem\n\nTests that fail intermittently — not because of a real bug, but timing issues, race conditions, or flaky external dependencies — waste huge amounts of developer time. A red CI check could mean \"you broke something\" or \"this test is just unreliable,\" and there's usually no easy way to tell which apart. FlakeGuard tracks test history over time and answers that question automatically, using the AI coding assistant the developer already has open.\n\n## How it works\n\n<picture>\n  <source media=\"(prefers-color-scheme: dark)\" srcset=\"docs/architecture-dark.svg\">\n  <img alt=\"FlakeGuard architecture: a GitHub Actions runner converts go test JSON and POSTs it to the flakeguard http process, which writes through the repository layer into Postgres. A background ticker recomputes flakiness scores. Separately, an AI assistant spawns the flakeguard mcp process over stdio, which reads through the same repository layer.\" src=\"docs/architecture-light.svg\">\n</picture>\n\nData enters on the left from CI and leaves on the right into an AI assistant. The two halves run as **separate processes that never talk to each other** — they share only the `repository` package and the tables underneath it.\n\n1. A developer pushes code, and GitHub Actions runs the test suite.\n2. A CI step converts `go test -json` output into a small JSON report and POSTs it to FlakeGuard's `/ingest` endpoint.\n3. FlakeGuard writes every individual test result — repo, branch, commit, test name, status, duration, timestamp — into Postgres.\n4. A background job periodically recomputes a **flakiness score** per test: a test \"flips\" whenever consecutive runs on the same branch have different results (pass then fail, or vice versa).\n   ```\n   flakiness_score = flip_count / (total_runs - 1)\n   ```\n   A score near `0` means stable; a score near `1` means the result is essentially noise.\n5. An MCP server exposes that data to AI coding assistants as three tools (below).\n6. A developer asks \"why did this test fail?\" — their assistant calls `check_test_flakiness` and gives a grounded answer instead of guessing.\n\n## What you get: the three MCP tools\n\n| Tool | When it's called | Returns |\n|---|---|---|\n| `check_test_flakiness(repo, test_name)` | A specific test failed and you need a verdict | Flakiness score, plain-language verdict, run/failure/flip counts |\n| `get_test_history(repo, test_name, limit=20, max=100)` | You want the raw pass/fail/skip timeline for one test | Recent runs, most recent first |\n| `list_flakiest_tests(repo, limit=10, max=50)` | Broad triage — \"which tests are unreliable here?\" | Ranked list of tests by flakiness score |\n\n## Prerequisites\n\n- **Go 1.25 or newer** — matches the `go` directive in [`go.mod`](go.mod), which is set by pgx/v5's own minimum. Check with `go version`.\n- **PostgreSQL** — either via Docker Compose (nothing to install), or a local instance for native development.\n- **Docker + Docker Compose** — only needed for the [Quick start](#quick-start-docker-compose) path.\n- **Python 3** — only needed to run `scripts/convert_go_test_json.py` (stdlib only, no `pip install` required).\n\n## Quick start (Docker Compose)\n\nThis brings up Postgres (with the schema auto-applied) and the ingestion API together — no manual migration step needed.\n\n```bash\ncp .env.example .env   # fill in INGEST_API_KEY with a secret of your choosing\ndocker compose up -d --build\n```\n\nThe API listens on `http://localhost:8080`. Verify it's up:\n\n```bash\ncurl -X POST http://localhost:8080/ingest \\\n  -H \"Content-Type: application/json\" \\\n  -H \"X-API-Key: <your INGEST_API_KEY>\" \\\n  -d '{\n    \"repo\": \"myorg/myrepo\",\n    \"branch\": \"main\",\n    \"commit_sha\": \"abc123\",\n    \"ci_run_id\": \"1\",\n    \"tests\": [\n      {\"test_name\": \"TestExample\", \"status\": \"passed\", \"duration_ms\": 42, \"ran_at\": \"2026-01-01T00:00:00Z\"}\n    ]\n  }'\n```\n\nYou should get back `{\"recorded\":1}` with a `201` status.\n\n## Local development (without Docker)\n\nYou'll need Go and a local Postgres (see [Prerequisites](#prerequisites)).\n\n**1. Create the database and apply the schema:**\n```bash\ncreatedb flakeguard_dev\npsql -d flakeguard_dev -f migrations/001_init.sql\n```\n\n**2. Create a `.env` file:**\n```bash\nDATABASE_URL=postgres://<user>@localhost:5432/flakeguard_dev?sslmode=disable\nINGEST_API_KEY=local-dev-secret\n```\n\n**3. Build and run:**\n```bash\ngo build -o flakeguard ./cmd/server\n\n# terminal 1 — the ingestion API\nset -a; source .env; set +a\n./flakeguard http\n\n# terminal 2 — the MCP server, over stdio\nset -a; source .env; set +a\n./flakeguard mcp\n```\n\nIn `mcp` mode, the binary speaks newline-delimited JSON-RPC 2.0 over stdin/stdout — it's meant to be launched as a subprocess by an MCP client (see below), not used interactively, though you can feed it raw JSON-RPC lines for manual testing.\n\n## Running the tests\n\nThe flakiness scoring is the one part of the system whose failure mode is silent — a wrong query returns a plausible number rather than an error — so it is covered by tests that run against a real PostgreSQL instance rather than a mock.\n\nCreate a scratch database, apply the schema, and point `TEST_DATABASE_URL` at it:\n\n```bash\ncreatedb flakeguard_test\npsql -d flakeguard_test -f migrations/001_init.sql\n\nTEST_DATABASE_URL=postgres://<user>@localhost:5432/flakeguard_test?sslmode=disable \\\n  go test ./...\n```\n\nThe tests **skip** when `TEST_DATABASE_URL` is unset, and deliberately never fall back to `DATABASE_URL` — they insert and delete rows, and must not run against a database holding real data. Each case writes under its own unique `repo` identifier and removes only its own rows, so they are safe to run against a shared development database and do not truncate shared tables.\n\nCoverage focuses on the `LAG()` flip detection: alternating results, always-passing, always-failing (which must score `0.0` — broken is not flaky), single-run divide-by-zero, cross-branch partitioning, skipped-status transitions, repo isolation, and upsert-on-repeat.\n\n## Connecting to Claude Code\n\nAdd a `.mcp.json` file at your project root (or wherever you want FlakeGuard's data available from):\n\n```json\n{\n  \"mcpServers\": {\n    \"flakeguard\": {\n      \"type\": \"stdio\",\n      \"command\": \"/absolute/path/to/flakeguard\",\n      \"args\": [\"mcp\"],\n      \"env\": {\n        \"DATABASE_URL\": \"postgres://user@localhost:5432/flakeguard_dev?sslmode=disable\"\n      }\n    }\n  }\n}\n```\n\nOr register it via the CLI:\n\n```bash\nclaude mcp add --transport stdio flakeguard --scope project \\\n  --env DATABASE_URL=\"postgres://user@localhost:5432/flakeguard_dev?sslmode=disable\" \\\n  -- /absolute/path/to/flakeguard mcp\n```\n\nRestart Claude Code (or run `/mcp` to check connection status) and the three tools become available to the assistant automatically — no further setup needed on the assistant side.\n\n> **Note:** `.mcp.json` with an absolute path and inline credentials is machine-specific, so it's `.gitignore`d in this repo rather than committed. If your whole team should share one config, use a relative path (`./flakeguard`) and reference a host env var instead of hardcoding the connection string — Claude Code expands `${DATABASE_URL}` inside `.mcp.json`.\n\n## Worked example\n\nSay `TestFoo` in `myorg/myrepo` just failed in CI. A developer asks Claude Code *\"did I break something, or is this test just flaky?\"* — Claude Code calls `check_test_flakiness`:\n\n**Request** (what the assistant sends, over stdio):\n```json\n{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"tools/call\",\"params\":{\"name\":\"check_test_flakiness\",\"arguments\":{\"repo\":\"myorg/myrepo\",\"test_name\":\"TestFoo\"}}}\n```\n\n**Response** (what FlakeGuard sends back):\n```json\n{\n  \"jsonrpc\": \"2.0\",\n  \"id\": 3,\n  \"result\": {\n    \"content\": [\n      {\n        \"type\": \"text\",\n        \"text\": \"Test: TestFoo\\nRepo: myorg/myrepo\\nFlakiness score: 0.75\\nVerdict: highly flaky — failures here are likely noise, not a real bug\\nTotal runs: 5\\nFailed runs: 2\\nFlip count: 3\\nLast updated: 2026-01-01T12:00:00Z\"\n      }\n    ]\n  }\n}\n```\n\nClaude Code then tells the developer:\n\n> *\"This test has a flakiness score of 0.75 — it's flipped between pass and fail 3 times over its last 5 runs. This failure is very likely noise, not something you broke. Want me to check `get_test_history` for the exact timeline, or look at other flaky tests in this repo?\"*\n\n— instead of assuming the developer's change caused the failure.\n\n## Ingestion API reference\n\n**`POST /ingest`**, header `X-API-Key: <INGEST_API_KEY>`, JSON body:\n\n```json\n{\n  \"repo\": \"myorg/myrepo\",\n  \"branch\": \"main\",\n  \"commit_sha\": \"<sha>\",\n  \"ci_run_id\": \"<ci run identifier>\",\n  \"tests\": [\n    {\"test_name\": \"TestFoo\", \"status\": \"passed\", \"duration_ms\": 120, \"ran_at\": \"2026-01-01T00:00:00Z\"}\n  ]\n}\n```\n\nAll fields are required. `status` must be one of `passed`, `failed`, `skipped`.\n\n| Response | Meaning |\n|---|---|\n| `201 {\"recorded\": N}` | Success — `N` test results written |\n| `400` | Validation error (missing field, invalid status, malformed JSON) |\n| `401` | Missing or incorrect `X-API-Key` |\n\n`scripts/convert_go_test_json.py` builds the request body above from `go test -json` output:\n\n```bash\ngo test -json ./... | python3 scripts/convert_go_test_json.py \\\n  --repo myorg/myrepo --branch main --commit-sha \"$SHA\" --ci-run-id \"$RUN_ID\" \\\n  > report.json\n```\n\n## GitHub Actions integration\n\nSee [`.github/workflows/example-usage.yml`](.github/workflows/example-usage.yml) for the full loop: run tests, convert output, report to FlakeGuard — even when the tests themselves fail, so flaky *and* broken tests both get recorded.\n\nIt needs two repository secrets:\n\n| Secret | Value |\n|---|---|\n| `FLAKEGUARD_INGEST_URL` | Your deployed FlakeGuard API's base URL |\n| `FLAKEGUARD_API_KEY` | Matches your deployment's `INGEST_API_KEY` |\n\nInside GitHub Actions, `--repo`/`--branch`/`--commit-sha`/`--ci-run-id` are auto-detected from `GITHUB_REPOSITORY`/`GITHUB_REF_NAME`/`GITHUB_SHA`/`GITHUB_RUN_ID` if you don't pass them explicitly — the workflow relies on this instead of hardcoding them.\n\n## Environment variables\n\n| Variable | Required in | Purpose |\n|---|---|---|\n| `DATABASE_URL` | both modes | Postgres connection string |\n| `INGEST_API_KEY` | `http` mode | Shared secret required on all `/ingest` requests (`X-API-Key` header) — the server refuses to start without it |\n| `HTTP_ADDR` | `http` mode | Listen address, defaults to `:8080` |\n| `FLAKINESS_REFRESH_INTERVAL` | `http` mode | How often to recompute flakiness scores, defaults to `2m` (Go duration syntax) |\n\n## Project structure\n\n```\nFlakeGuard/\n├── cmd/\n│   └── server/\n│       └── main.go                  # entrypoint — \"http\" or \"mcp\" mode, chosen by arg\n├── internal/\n│   ├── mcp/                         # protocol types, tool definitions, stdio transport loop\n│   │   ├── protocol.go              # JSON-RPC 2.0 + MCP message types\n│   │   ├── server.go                # stdio transport loop (initialize/tools/list/tools/call)\n│   │   ├── register.go              # wires the three tools into a Server\n│   │   ├── tool_check_flakiness.go\n│   │   ├── tool_history.go\n│   │   └── tool_list_flakiest.go\n│   ├── ingest/                      # HTTP handler + API key auth for /ingest\n│   │   ├── handler.go\n│   │   ├── auth.go\n│   │   └── models.go\n│   └── repository/                  # all Postgres queries — nothing else writes SQL\n│       ├── repository.go            # Repository struct + pgxpool constructor\n│       ├── models.go\n│       ├── record.go                # RecordRun, RecordRuns\n│       ├── history.go               # History\n│       └── flakiness.go             # Flakiness, FlakiestTests, RefreshFlakinessScores\n├── migrations/\n│   └── 001_init.sql                 # schema: test_runs (raw history), test_flakiness (computed scores)\n├── scripts/\n│   └── convert_go_test_json.py\n├── .github/\n│   └── workflows/\n│       └── example-usage.yml\n├── Dockerfile                       # multi-stage build\n├── docker-compose.yml                # Postgres + API for local dev\n├── go.mod\n└── go.sum\n```\n\nTwo files intentionally aren't committed, since they're machine-specific: `.env` (your local secrets — copy `.env.example` to start) and `.mcp.json` (your local Claude Code MCP registration).\n\n## Language support\n\nOnly one piece of this project is Go-specific: `scripts/convert_go_test_json.py`, which parses the particular JSON shape `go test -json` emits. Everything downstream — the `/ingest` API, the Postgres schema, the flakiness scoring, the MCP tools — works on plain `{test_name, status, ran_at}` data and has no idea what language produced it.\n\nTo use FlakeGuard with another ecosystem, write a different converter that emits the same [ingestion report shape](#ingestion-api-reference) — for example, from `pytest --json-report`, Jest's `--json` output, or generic JUnit XML (which many test runners across languages can produce). Nothing else in the system needs to change.\n\n## License\n\n[MIT](LICENSE) © Dhananjaya Chamod\n"}