{"id":"d40ae18f-5ddf-4e30-ab92-1db5d3521261","slug":"jvm-incident-agent","kind":"mcp","name":"jvm-incident-agent","summary":"JVM incident analysis from thread dumps, GC logs and heap histograms - deterministic rules, evidence chain on every finding, runs offline.","owner":"JingYu-create520","repoUrl":"https://github.com/JingYu-create520/jvm-incident-agent","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-09-20T16:51:38.000Z","addedAt":"2026-10-21T06:48:54.567Z","logoUrl":"https://avatars.githubusercontent.com/u/285314266?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["cloud-infra","memory-knowledge"],"tags":["ai-agents","cli-tool","deadlock","devops","gc","heap-dump","java","jvm"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-27T00:45:41.644Z","sample":false,"url":"https://conduits.dev/tools/jvm-incident-agent","language":"Java","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# jvm-incident-agent\n\n**Drop in a thread dump, a GC log, a heap histogram and an application log. Get a root-cause report where every finding quotes the exact line it came from.**\n\nDeterministic parsers produce the facts. A language model — optional, off by default — only turns\nthose facts into prose. It runs offline: your dumps never leave the machine, and the same inputs\nalways produce the same findings.\n\n![Reproduce an incident in a container, analyze it with one command, gate on the exit code](docs/assets/demo.gif)\n\n```\n$ jia analyze ./incident-2026-09-20/\n\n## Verdict\n\n**The live set is growing — after each major collection more heap survives than the last\ntime. Full GC pressure is downstream of this, not the cause. Corroborated by GCA003.**\n\nConfidence 90% · severity CRITICAL · corroborated by GCA003\n\n$ echo $?\n1\n```\n\nThat is a real run against a real JVM in [`corpus/incident-heap-leak`](corpus) — a Spring Boot\napp with an unbounded `byte[]` cache planted in it. The rules found the leak, not just the\nsymptom:\n\n```\nGCA001 CRITICAL 13 Full GC collections inside 1.0 minute(s) (13.0/min), stopping the world for 551 ms\nGCA003 CRITICAL  Across 13 major collections the old gen low-water mark is 217M, sits at 11 of 13\n                  collections with 88% of the heap still live after a Full GC — nothing is being\n                  reclaimed any more (capacity 256M)\nHIS001 HIGH      [B holds 165.2 MB of 229.2 MB (72.1% of all bytes counted, 2,144,188 instances,\n                  81 bytes each)\nGCA004 HIGH      11 collection(s) ran out of copy space; 41 triggered by humongous allocation\n```\n\nAnd on the same tool's capture of a **healthy** service in the same shape: `0 findings`, exit\ncode `0`. That is the gate this project is built around — see [False positives](#false-positives-are-the-real-problem).\n\n---\n\n## Why not just …\n\nEclipse MAT is the right tool for a heap dump, and this project sends you to MAT on purpose. But\nMAT is an IDE-shaped GUI with a real learning curve; you cannot script it at 3 a.m., and it does\nnot read your GC log and your thread dump as one incident.\n\ngceasy.io does good GC analysis for a price: your production GC log ends up on someone's website.\n\nAsking a chatbot gets you \"increase `-Xmx`\" from a stack trace, with no evidence chain, no\nreproducibility, and no idea that 200 idle Tomcat workers are just a healthy server at night.\n\n`jstack` plus grep is what most of us actually do. This is that, except the wait-for graph, the\nsliding GC windows and the histogram arithmetic are already done, and it reads all four artifacts\ntogether instead of one at a time.\n\nThe position I am claiming is narrow on purpose: command line, scriptable, callable by an agent,\ndata never leaves the machine.\n\n## Install\n\nNeeds a Java 17+ runtime. No installation step beyond getting one jar.\n\n```bash\n# build the shaded jar (target/jia.jar)\n./mvnw -q -DskipTests package\n\n# or with a released jar\ncurl -LO https://github.com/JingYu-create520/jvm-incident-agent/releases/latest/download/jia.jar\njava -jar jia.jar --version\n```\n\nConvenience launcher: `bin/jia analyze ./incident/` builds the jar on first use.\n\nDocker, if you prefer:\n\n```bash\ndocker build -f Dockerfile.cli -t jia .\ndocker run --rm -v \"$PWD/incident:/input:ro\" jia          # no path needed: WORKDIR is /input\n```\n\nThe image analyses whatever is mounted at `/input` and prints the report to stdout.\n**Windows + Git Bash note:** mount with a Windows-style host path —\n`-v \"D:/projects/incident:/input:ro\"` — because Git Bash rewrites `$PWD` into an MSYS path\n(`/d/…`) that Docker silently mounts as empty. PowerShell does not have this problem.\n\n## Quickstart\n\nPoint it at a **directory**. File kinds are detected from content, not filenames, and mixed\ninputs are fine.\n\n```bash\nmkdir -p incident-2026-09-20 && cd incident-2026-09-20\njstack -l 12345 > threads.dump\nsleep 5 && jstack -l 12345 > threads-2.dump     # a second dump unlocks the growth rules\njmap -histo  12345 > heap.histo\ncp /var/log/app/app.log .                        # your GC log too, if -Xlog wrote one\ncd .. && jia analyze ./incident-2026-09-20 -o ./incident-2026-09-20/ --format both\n```\n\nYou get `report.md` for a human and `report.json` for a program, each finding carrying\n`file:line` evidence. Useful switches:\n\n```\n--sla-ms 100            your pause budget, not the default 200\n--full-gc-per-min 2     where a storm starts\n--thread-leak-threshold 60\n--lock-waiters 5        how many waiters make a monitor \"contended\"\n--exception-threshold 10\n--mat-share 0.09        the boundary in the corpus docs: CachedRow holds 9.32%, so 9% names it\n--stall-min 1           how many ZGC allocation stalls make a pattern\n--format json           structured findings\n--llm                   narrate with an OpenAI-compatible endpoint (findings unchanged)\n--no-narrative          rule output only\n```\n\nEvery threshold that can decide a finding is a flag (`jia analyze --help` lists them all), and\n`report.json` echoes the whole set under `thresholds`. That is deliberate: a boundary you cannot\nmove is not an engineering decision, it is a default somebody picked — so if you disagree with\n\"9.32 % is not enough to call a leak\", you can prove the other answer in one command rather than\nopen an issue about it.\n\n## What it reads\n\n| Artifact | Format | How |\n|---|---|---|\n| Thread dump | `jstack -l <pid>`, `jcmd Thread.print`; several captures in one file | state machine, not one regex — JDK 8 → 21 drift (the `cpu=`/`elapsed=` columns, module-qualified frames, the deadlock trailer that repeats stanzas) is handled explicitly |\n| GC log | JDK 7/8 traditional `-XX:+PrintGCDetails` (Parallel, Serial, CMS — `ParNew`, `[CMS: …]` old gen, and a permanent-generation block that is excluded from heap accounting), and JDK 9+ unified `-Xlog:gc*` | per-line parse into one normalised event timeline; `GC(n)` records merged back into one collection; each event keeps its own stop-the-world total, not its first phase |\n| Heap histogram | `jmap -histo[:live]`, `jcmd GC.class_histogram` | table parse, module suffix stripped, totals recovered when the `Total` row is missing |\n| Application log | any text with stack traces, including a log a terminal coloured (`logback`/`log4j` shapes) | throwable blocks lifted out with their `Caused by:` chains, fingerprinted by class + first 5 root frames |\n\nCharsets are detected (UTF-8, else GBK, else Latin-1) because these files come from other\npeople's machines. Terminal escape sequences — the `ESC[…m` colour codes a Spring Boot process\nwrites when its output is redirected into a file, or a CI runner captures — are stripped at read\ntime, so evidence quotes show text a human can read and line numbers still point at the same line.\n\nA run reads **one GC log and one histogram**, because that is what one incident has: a window and an\ninstant. When you hand it a rotated set (`gc.log` plus `gc.log.0`) or two `jmap` files, it keeps the\nfirst one it sees — walking a directory means the suffix-less, live rotation wins — and names what it\ndropped, in the Markdown's \"Coverage and limits\" section and in `report.json` under `ignoredInputs`.\nA report over half the window has to say so, because it otherwise looks exactly like a report over all\nof it. The same rule applies to fields, not just files: if your application log timestamps with\n`HH:mm:ss.SSS` only, the burst rule cannot place any stack in a minute, and the report says how many of\nthe parsed stacks had no absolute timestamp (`exceptionClock` in JSON) rather than leaving \"no burst\" to\nread as an all-clear.\n\nAnd the third case is a rule that could not measure at all. A Full GC rate needs two major collections, a\nlive-set fingerprint needs a few, a throughput percentage needs ten events across ten seconds, thread\n*growth* needs a second dump — and a rule that had the artifact but not enough of it used to return\nnothing and be recorded as `clean`, which let a truncated log print *\"Every rule ran clean against this\nsnapshot.\"* Now it says what it did: `ruleStatus` reports `declined: <what it needed>`, the same lines\nappear under \"Coverage and limits\", and the verdict only claims everything ran clean when nothing\nabstained. One rule got stricter in the same pass — `GCA002` ignored any log with fewer than three pauses,\nso a lone 305 ms stop against a 200 ms SLA went unreported; it reports that now, capped at `MEDIUM` and\nlabelled as the single-data-point measurement it is.\n\n## The 19 rules\n\n`jia rules` prints this list; `jia explain <ID>` prints how it works **and where it can be\nwrong** — every rule documents its own false-positive limits.\n\n**Thread dumps**\n\n| ID | Detects |\n|---|---|\n| TDA001 | Deadlock: wait-for graph + Tarjan SCC, monitors *and* `ReentrantLock` (the JVM's own detector misses the latter) |\n| TDA002 | Contended monitor: one lock, many waiters, and who holds it |\n| TDA003 | Thread leak: oversized name family, or a family that only grows across dumps |\n| TDA004 | Stack hotspot: a crowd in one place, plus RUNNABLE threads actually stuck on sockets |\n| TDA005 | Pool starvation: no idle worker left, all busy in the same frame |\n| TDA006 | CPU-burning thread: cores from `cpu`/`elapsed`, as a lifetime average or a two-dump delta |\n\n**GC log**\n\n| ID | Detects |\n|---|---|\n| GCA001 | Full GC storm in the densest window, not averaged over the log |\n| GCA002 | Pauses over your SLA, reported as p50/p95/p99/max |\n| GCA003 | Rising post-GC live set — and the plateau of the same leak once the heap is full |\n| GCA004 | Premature promotion: to-space exhausted, humongous allocation, low-yield young GC |\n| GCA005 | Config smells: metaspace pressure, `System.gc()`, and the JVM's own hint strings |\n| GCA006 | GC throughput below target |\n| GCA007 | Allocation stalls — ZGC stopping the thread that needed memory, and the name of that thread |\n\n**Heap histogram**\n\n| ID | Detects |\n|---|---|\n| HIS001 | One class dominating counted bytes; and the case where only `byte[]`/`char[]`/`String` can be named |\n| HIS002 | An application class big enough to be worth opening in MAT |\n| HIS003 | Implausible collection/node counts |\n\n**Application log**\n\n| ID | Detects |\n|---|---|\n| EXC001 | Repeated exception clusters by root-cause stack |\n| EXC002 | Causal-chain attribution: past the framework wrapper, into your code |\n| EXC003 | Time clustering — the minute your incident started |\n\nFindings are then **correlated** into ranked hypotheses, because \"Full GC storm\" and \"the live\nset is growing, which is causing the storm\" come out of the same log and only one of them is\nactionable.\n\n## Evidence, not vibes\n\nEvery finding says where it came from, and you can check it:\n\n```\n### TDA001 · Deadlock (cycle in the wait-for graph)\nCRITICAL · confidence 99% · from thread dump\n\nvictim-statement-writer waits for 0x00000000ff6309a0 (java.lang.Object), held by\nvictim-ledger-poster; victim-ledger-poster waits for 0x00000000ff6309b0, held by\nvictim-statement-writer.\n\n**Evidence**\nthreads.dump:495  \"victim-ledger-poster\" #51 daemon prio=5 … ← BLOCKED, waiting on 0x…9b0\nthreads.dump:498  - waiting to lock <0x00000000ff6309b0> (a java.lang.Object)\nthreads.dump:511  - locked <0x00000000ff6309b0>   ← monitor held here by the other thread\n```\n\nThe exit code is part of the interface: `0` nothing at or above the gate, `1` something is, `2` the\ninput could not be understood. The gate is `--fail-on`: default `high`, `--fail-on critical` in CI\nso an advisory MEDIUM cannot break a build, `--fail-on never` when you only want the report.\n\n## LLM: narration only\n\nThe rules run first and finish before any model is contacted. `--llm` adds a paragraph; it\ncannot add, drop, reweight or reword a finding, and there is a test that fails if it ever does:\n\n```java\n// EngineTest.narrativeCannotMoveFindings\nProvider blabber = result -> \"THERE IS NO DEADLOCK, EVERYTHING IS FINE, RESTART THE POD\";\nassertEquals(before, after);                     // findings byte-identical\nassertTrue(report.contains(\"TDA001\"));           // the lie did not survive into the findings\n```\n\nConfigure it with environment variables — any OpenAI-compatible endpoint (OpenAI, DeepSeek,\nQwen, Ollama, vLLM, your gateway):\n\n```bash\nexport JIA_LLM_BASE_URL=https://api.openai.com/v1\nexport JIA_LLM_API_KEY=sk-…\nexport JIA_LLM_MODEL=gpt-4o-mini\njia analyze ./incident/ --llm\n```\n\nOnly finding summaries are sent — never a raw dump, log or histogram. Without the env vars, or\nif the endpoint fails, you get the built-in offline narrator, and the run still succeeds.\n\n## Use it from an agent (MCP)\n\n```bash\njia mcp        # stdio, newline-framed JSON-RPC 2.0\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"jvm-incident-agent\": { \"command\": \"java\", \"args\": [\"-jar\", \"/path/to/jia.jar\", \"mcp\"] }\n  }\n}\n```\n\nThree tools: `analyze_snapshot` (a path, or pasted `content` + `fileName`), `explain_finding`\n(rule id → documentation), `list_rules`. A pasted `jstack` in a chat window becomes a real\nanalysis with line numbers that refer back to what the user pasted.\n\nThe transport is hand-written rather than using the official SDK — the reasoning is in\n[`docs/adr/0001-hand-written-mcp-transport.md`](docs/adr/0001-hand-written-mcp-transport.md).\n\n## CI gate\n\n`jia` is designed to be an assertion:\n\n```yaml\n- name: JVM incident triage\n  run: |\n    java -jar jia.jar analyze ./dist/tomcat/logs/incident --format json -o /tmp/jia/ || rc=$?\n    # 0 = clean, 1 = something high-severity, 2 = could not parse\n    test \"${rc:-0}\" -lt 2\n```\n\nOr use it as a summariser on a failing build: the JSON schema is\n`jvm-incident-agent/1`, versioned in the payload.\n\n## Reproduce an incident on purpose\n\nAnalysis needs something to analyze, and dumping production data into a test suite is not\nreasonable. So the repo ships a **target application** and the real output of breaking it.\n\n[`demo-victim/`](demo-victim) is a Spring Boot app with planted incidents:\n\n| endpoint | planted fault |\n|---|---|\n| `GET /victim/deadlock` | two threads, two monitors, opposite order (plus a hot lock with 8 waiters) |\n| `GET /victim/leak?mb=32` | `byte[]` chunks appended to an unbounded cache |\n| `GET /victim/gcstorm?rounds=40` | thousands of short-lived large arrays |\n| `GET /victim/leak-threads?count=80` | threads spawned per call and never closed |\n| `GET /victim/errors?count=40` | wrapped exception chains with real framework frames |\n| `GET /victim/healthy` | the clean path the zero-false-positive gate is measured against |\n\n```bash\n# one command: builds the victim image, starts it, plants the deadlock,\n# captures the four artifacts into corpus/incident-deadlock/\ndocker compose -f server/docker-compose.yml --profile repro run --build --rm incident deadlock\n\n# then analyze from the host (or point a container at it, see Install)\njia analyze corpus/incident-deadlock\n```\n\nThe victim binds **no host port at all** — the trigger reaches it over the compose network at\n`http://victim:8080`. That is deliberate: it used to publish `8081` by default, and on a machine\nthat already had something listening there the one-command repro died with\n`Bind for 0.0.0.0:8081 failed: port is already allocated`, which is a confusing way to fail a flow\nthat never needed the port. Want a browser URL for `/victim/*`, run it on a port you choose:\n`docker compose -f server/docker-compose.yml run --publish 8081:8080 victim`.\nIncidents: `deadlock`, `heap-leak`, `gc-storm`, `thread-leak`, `exceptions`,\n`healthy`. On the host without Docker:\n\n```bash\ncd demo-victim && ../mvnw -q -DskipTests package && cd ..\nmkdir -p live\njava -Xmx256m -Xms256m -XX:+UseG1GC \\\n     -Xlog:gc*:file=live/gc.log:time,uptime,level,tags \\\n     -jar demo-victim/target/demo-victim.jar > live/app.log 2>&1 &\nscripts/capture.sh -o /tmp/incident -d 6 --gc-log live/gc.log --app-log live/app.log\n```\n\n`capture.sh` copies the JVM's own `-Xlog` output; it cannot invent a GC log, so that flag is what\nmakes the fourth artifact exist. Then plant something while it is still running — the trigger\nendpoints are listed above — and analyze.\n\nEverything in [`corpus/`](corpus) is byte-for-byte real tool output from a live JVM — no\nhand-written dumps. Seven scenarios, each with a `TRUTH.md` stating what was planted and which\nrules should fire. `CorpusTest` asserts exactly that, and `TruthDocTest` asserts the markdown\nagainst the engine — every one of the 19 rules accounted for in every folder. So both a rule\nchange that breaks the heap-leak-vs-allocation-storm distinction and a ground-truth file that\ndrifts away from what the tool reports will fail the build.\n\n## False positives are the real problem\n\nA triage tool that cries wolf gets uninstalled. Three rules of construction, all enforced by tests:\n\n1. **Healthy input must produce zero findings.** `corpus/healthy` is a real Spring Boot app\n   under load with 10 servlet threads and a `[B` top row in its histogram, and the suite asserts\n   it comes back silent. Startup `Metadata GC Threshold` collections — which every Spring Boot\n   log contains — are explicitly not a finding, and that sentence is now enforced where it matters:\n   the Full-GC **rate** rule and the leak-fingerprint rule drop them (and anything an outside actor\n   forced, like the `Heap Inspection Initiated GC` a `jmap -histo:live` causes) before they count,\n   inside `--gc-settle-sec` of uptime. Until 0.3.1 only the config-smell rule knew about the\n   distinction, and a 78-line slice of a healthy service's own boot log came back as\n   \"Full GC storm, HIGH\".\n2. **Idle is not stuck.** Two hundred workers parked in `ThreadPoolExecutor.getTask` is a quiet\n   night. The exclusion list is explicit (`ThreadNoise`) rather than inferred, and it is the\n   single biggest difference between a usable thread-dump rule and a noisy one.\n3. **Absolute floors next to ratios.** \"`byte[]` is 24% of the heap\" means nothing on a 6 MB\n   histogram; the rule needs 16 MB and 35% before it speaks.\n\nWhere a rule can still mislead, it says so in its own documentation — that section is required\nby `RulesTest.catalogueIsComplete`.\n\n## Limits\n\n- **No `.hprof`.** Heap-dump parsing is a different project; the report tells you what to look\n  for in MAT instead.\n- **Tested on JDK 8 → 21 shapes**, but captured on JDK 17. Older formats are covered by\n  hand-written fixtures; a genuinely exotic format degrades to an `INFO` finding rather than an\n  error.\n- **The GC rules read three vocabularies.** G1/Parallel/Serial/CMS, where a `Pause Full` is the\n  collection that reveals the live set; ZGC, where the whole-heap concurrent cycle is that\n  collection and `Allocation Stall` is its pressure signal (GCA007, and `corpus/incident-zgc-leak`\n  is there to keep both honest); and Shenandoah, which stops the world in `Pause Init Mark` /\n  `Pause Final Mark` pairs and still runs real Full GCs — a slice of a real 1 GB Shenandoah capture\n  is in `src/test/resources/fixtures/gc-jdk17-shenandoah.log`, and it is what caught two of the\n  bugs below.\n  The ZGC support is **verified on non-generational ZGC (JDK 17)**. Generational ZGC (21+) labels\n  its cycles `(Minor)`/`(Major)`, and this tool does not distinguish them yet — a minor cycle is\n  counted as major, which can only over-report, never under-report.\n- **Virtual threads are invisible.** A JDK 21 dump's `-- virtual thread … mounted on carrier`\n  trailer carries real incident information (pinning being the interesting one) and this tool\n  neither parses nor reasons about it. A Loom-heavy service will look like a JVM with a suspiciously\n  small thread count.\n- **One JVM per snapshot.** No cross-service or distributed correlation.\n- GC throughput counts the pauses `-Xlog:gc*` prints; safepoint work that is not a GC pause is\n  invisible, so the real figure can be worse than reported, never better. A ZGC allocation stall\n  stops one thread rather than the world, so it is excluded here and reported by GCA007 instead.\n- A wall-clock ↔ uptime axis needs a log carrying both decorators; without it, the timeline\n  says so in the report.\n\n## Development\n\n```bash\n./mvnw test                       # the whole suite, ~4 s\n./mvnw -q -DskipTests package     # target/jia.jar\njia rules --format json           # the machine-readable catalogue\n```\n\nLayout: `parse/` (four parsers + sniffing), `analyze/` (graph, Tarjan, 19 rules, hypothesis\nranking), `llm/`, `report/`, `mcp/`, `Cli.java`. `docs/PLAN.md` is the original design document\nthe build follows.\n\nContributions that add a rule must add its negative case. That is the actual requirement.\n\n## How this was actually built\n\nNot the polished version. The rules all look reasonable now because the corpus kept catching them\nbeing wrong in specific ways:\n\n- The exception parser found **zero** stacks in a real Spring Boot log, while every hand-written\n  fixture passed. Cause: my\n  frame regex demanded the line end with `)`, and logback appends `~[spring-web-6.2.8.jar:6.2.8]`\n  after it. Every fixture I had written by hand passed.\n- `[Metaspace: 3072K->3072K(1056768K)]` has exactly the shape of a heap transition and sits at the\n  end of the line, so the \"last transition wins\" rule read a JDK 8 heap as 1032 MB.\n- `Found one Java-level deadlock:` ends with a colon. I matched it with `matches()`, which demands\n  a whole-line match, so the check never fired and the trailer's `- locked` lines were attributed\n  to whichever thread stanza happened to be open — the last real thread appeared to own monitors\n  it never touched, and the cycle disappeared.\n- The same `matches()` mistake meant all four content signatures were silently dead and file\n  identification was working off filenames alone. Nothing failed; the tool was just quietly wrong.\n- jstack prints the capture date on the line *before* `Full thread dump`, and that branch rebuilt\n  the parser state unconditionally. Result: every timeline entry said `—`.\n- The healthy capture kept tripping GCA005, because our Spring Boot startup logs produce a\n  `Metadata GC Threshold` collection or two during startup. The threshold is now \"three or more,\n  at least one after the JVM has been up for a minute\".\n- The Docker path shipped unverified in 0.1.0 because no daemon was running. Running it found\n  `capture.sh` dying under `sh` on `set -o pipefail`, port 8080 already taken, and the one-click\n  thread-leak capture firing both bursts *before* the first dump — so it produced 140/140 and\n  could not demonstrate the growth rule at all.\n\n`docs/PLAN.md` is the design document I worked from, including the list of places where the build\ndiverged from it. `CHANGELOG.md` records what changed and why. Both are honest about the mistakes\nabove; that is the part I would want to read if I were evaluating someone else's repo.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n"}