\n\n
![\"AiSOC\"](\"apps/web/public/logo-mark.svg\")
\n\n# AiSOC\n\n**An open-source, self-hostable AI Security Operations Center.** It ingests your security telemetry, detects and correlates threats, investigates them with AI agents whose reasoning is fully auditable, and proposes responses a human approves.\n\n[](https://opensource.org/licenses/MIT) [](CHANGELOG.md) [](https://github.com/beenuar/AiSOC/actions/workflows/ci.yml)\n[](https://github.com/beenuar/AiSOC/actions/workflows/codeql.yml) [](https://securityscorecards.dev/viewer/?uri=github.com/beenuar/AiSOC) [](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)\n\n**[Technical Guide (PDF)](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf)** · [Docs](https://beenuar.github.io/AiSOC/) · [Architecture](docs/architecture/README.md) · [What actually works](docs/audit/REPOSITORY_REALITY.md) · [Discussions](https://github.com/beenuar/AiSOC/discussions)\n\n
![\"AiSOC](\"apps/web/public/demo/hero.gif\")
![\"Alerts](\"apps/web/public/screenshots/alerts-queue.png\")
| ![\"AI](\"apps/web/public/screenshots/ai-triage-verdict.png\")
|\n| **Alerts** — each attributed to the connector that fed it. | **Automated triage** — the bundled local model's verdict, confidence and rationale, verbatim. |\n| ![\"Threat](\"apps/web/public/screenshots/threat-intel-kev.png\")
| ![\"SOC](\"apps/web/public/screenshots/soc-operations.png\")
|\n| **Threat intelligence** — the real CISA KEV catalog, minutes after boot, with no API key. | **SOC operations** — with nothing connected yet, and it says so rather than showing a placeholder. |\n\n## Quick start\n\n```bash\ngit clone https://github.com/beenuar/AiSOC && cd AiSOC\nmake up\n```\n\nThe [Technical Guide](https://raw.githubusercontent.com/beenuar/AiSOC/main/apps/web/public/papers/aisoc-technical-guide.pdf) covers this in depth: server sizing, where the\nmodel runs, every failure mode with its cause and fix, the REST API, MCP, and every console surface.\n\nNeeds Docker Compose v2 with **8 GB memory and 20 GB free disk in the Docker VM**, plus `python3`\n(3.9+) and `bash`; `make doctor` checks all of it and\n[Installation](https://beenuar.github.io/AiSOC/docs/installation#requirements) says what each number\nwas measured against. The first run downloads a ~2 GB model into a volume only `make clean` clears.\n\n`make up` also creates `.env` and generates the **fifteen** secrets in it — the credential vault, the\nsession signing key, five service-to-service credentials and four datastore passwords — then creates an\nadministrator and prints its password, generated on your machine, shown once and stored nowhere. Lost\nit? `make bootstrap ARGS=--reset-password` mints another.\n\n**A port already in use does not stop the install.** AiSOC publishes on a free one, names what held the\nold one, and moves the console address with it: 64 seconds to a signed-in console on a bare clone with\n5432 and 11434 both taken.\n\nThen **prove it works**. `make smoke` posts one real event to the ingest API and follows it through\nKafka, detection, correlation and Postgres, then reads the alert back out of the public API. Every\nstage reports PASS or FAIL:\n\n```\n$ make smoke\n[PASS] raw telemetry accepted by ingest\n[PASS] event traversed the spine and became an alert\nPASS: 10/10 stages\n```\n\nSign in at the address `make up` printed. A tenant with nothing connected lands on a **setup\nwizard** rather than an all-zero dashboard, and its state is read from your own data so it stays\nright if you connect a source through the API. The spec is\n[`docs/openapi.yaml`](docs/openapi.yaml) — interactive docs are off in this production-class stack.\nStuck? `make doctor`, which on a host where you have not run `make up` yet says exactly that.\n\nPress **Load sample data** in the wizard. Five scenarios take the same ingest path a real connector uses — not inserted rows — so watching them become alerts means watching the pipeline work. They span low to critical on purpose, because a first run where everything is a crisis teaches you nothing about how triage separates signal from routine. They are attributed to `AiSOC` in the source column, refuse to load into a tenant that already has real alerts, and do **not** mark setup complete. ([what each step proves](https://beenuar.github.io/AiSOC/docs/console/getting-started)) For the larger fixed corpus used by demos and evals, `make demo` loads a **synthetic** dataset — the pipeline shape, never a benchmark, a customer or an incident. Every row is `is_synthetic = true` and labelled in the console.\n\n## Connect real data\n\nPush, with a credential from `make ingest-token` (the tenant comes from it, not a header):\n\n```bash\ncurl -X POST http://localhost:8081/v1/ingest/batch \\\n -H 'Content-Type: application/json' -H \"Authorization: Bearer $AISOC_INGEST_TOKEN\" \\\n -d '{\"connector_id\":\"edr-1\",\"connector_type\":\"crowdstrike\",\"events\":[{\"severity\":\"high\",\n \"title\":\"Encoded PowerShell from Office\",\"host\":\"WIN-FIN-01\"}]}'\n```\n\nOr pull, by configuring one of **87 click-and-connect data connectors** in **Settings → Connectors**\n— Splunk, Sentinel, Elastic, CrowdStrike, Okta, AWS and Kubernetes audit among those with\nvendor-specific normalization and setup docs\n([coverage](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage)). Without a vendor profile\na connector still ingests through a generic mapping that resolves host, user and source IP.\n\n**A real cloud estate does not hand you its logs over a REST call.** Four collectors read the\nsurfaces one actually emits to: a CloudTrail organisation trail on **S3 notified over SQS**\n(management events, data events and VPC flow logs), a **GCP Pub/Sub** subscription on a Cloud Logging\nsink, an **Azure Event Hubs** capture, and a **syslog/CEF** listener. Each keeps a resumable cursor\nand a bounded per-poll budget, so a backlog applies backpressure instead of flooding ingest.\n\nBringing existing detections? `packages/aisoc-migrate` translates Splunk SPL, Sentinel KQL and\nElastic EQL and **refuses rather than approximating** what it cannot carry, because an almost-right\nrule is harder to find than a missing one: of the 2,005 Splunk rules bundled here 1,734 translate and\n1,711 of those are partial ([what that means](apps/docs/docs/migration/from-splunk.md)).\n\n## How it works\n\nIngest normalizes to a common shape and stamps an **activity projection** on every event — actor, action, resource, location, outcome, with `actor.kind` read from each vendor's own documented field rather than inferred — then Kafka carries it.\nThen fusion runs 2535 executable detection rules, of 6991 on disk, plus 76 sliding-window rules (73 count across events; 3 match an ordered sequence), applies **your tenant's own tuning** on top — the disables, floors and suppressions the console writes, so a rule you turned off actually stops firing — and decides what becomes an alert.\nCorrelation groups related alerts, an agent investigates and writes its reasoning to the Investigation Ledger, and a playbook may start from the result. Separately, new threat intelligence sweeps the lake for sightings you already collected, and a hypothesis becomes a hunt without anyone writing a query — the model fills a closed schema and every value is bound as a parameter, so it cannot express a query at all.\n\n**A playbook triggered by an alert previews before it acts.** Three switches must agree — the\ndeployment, the tenant, the playbook — and every default is off; anything less runs in preview with\nits plan attached to the alert. An approval step is a durable pause: the run suspends to Postgres,\nsurvives a restart, resumes after the approval, and expires with a recorded outcome rather than\nhanging.\n\n**Executable is earned, not declared.** A rule joins the compiled ruleset only after a vendor-shaped\nevent is replayed through the real connector and engine and that rule is *watched to fire* — never\ninferred from a directory or an `enabled:` flag. The proof can fail: `--prove-gate` reverts the\nWindows connector and requires all 1,687 Windows rules to go silent. It means reachable, not that it\ndetects an attack. **Two** rules still cannot fire — down from 119, because a rule that could not fire\nstopped being counted as executable — and both are waiting on a behavioural baseline. The ratchet\nonly moves down. ([why 1,362 were refused](docs/detections/sigma-compilation.md))\n\n**Every answer carries its receipts.** The copilot cites each checkable claim to the ledger entry behind\nit and labels the rest *uncited* rather than dropping them, and any investigation exports as a **signed\nevidence bundle** — byte-identical, prompts as digests, OCSF 1.9.0. ([how](docs/architecture/evidence-bundles.md))\n\n**[docs/architecture/README.md](docs/architecture/README.md)** walks that path one step at a time —\neleven steps, five diagrams, every box linking to the code — and [mirrors to the docs portal](https://beenuar.github.io/AiSOC/docs/architecture).\n\n## Deployment profiles\n\n| Profile | Command | Services | RAM | What you get |\n|---|---|---|---|---|\n| **core** | `make up` | 16 | ~8 GB | The full alerting pipeline: ingest → detect → correlate → alert → triage → console, plus the LLM gateway, a local model, the CISA KEV threat feed, and the connector and response services the agent's vendor tools reach |\n| **full** | `make up-full` | 22 | ~12 GB | Core plus event lake, entity graph, full-text search, enrichment |\n| **demo** | `make up && make demo` | 16 | ~8 GB | Core plus labelled synthetic data |\n\nCORE is the smallest deployment that takes a real event and produces a real alert, and **it needs no\ncredentials to do either.**\n\n**The model ships with the gateway.** Ollama runs a pinned ~2 GB `llama3.2:3b-instruct-q4_K_M`\nsized for CPU-only inference, so `make up` produces real triage verdicts with real token counts in\nthe Investigation Ledger — not a stub. It is not a frontier model: over 50 alerts it gave triage\nusable output 44 times before the reply was constrained to JSON and 50 after\n([method](scripts/measure_triage_reliability.py)), and the rail labels which path answered. Run it\nfaster with `make up-gpu`, `make up-host-llm`, or your own provider from the console\n([all four](apps/docs/docs/operations/where-the-model-runs.md)). **No hosted provider has ever been\nexercised here** — there is no funded key, so per-model rows read *not measured* rather than zero.\n([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))\n\n**One real external feed ships too.** `services/threatintel` polls the CISA Known Exploited\nVulnerabilities catalog — public, no API key — into the console's Threat Intelligence page: the one\nthing in a fresh install that is neither synthetic nor yours.\n\n## Real vs synthetic data\n\n| Kind | Where | How you can tell |\n|---|---|---|\n| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |\n| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified |\n| **Sample** | The wizard's **Load sample data** | Source column reads `AiSOC`; RFC 5737 / RFC 2606 reserved addresses only |\n| **Demo** | `make demo` | `is_synthetic = true`, labelled in the console |\n| **Benchmark / fixtures** | `services/agents/tests/eval_data/`, `**/tests/` | Published rows carry `substrate: true`; fixtures never ship in an image |\n\n**Production never silently falls back to synthetic data.** An unreachable backend makes the console name the failure rather than invent an investigation, and an unmeasured figure reads *not measured*, never `0`. It was not always so: [the reality audit](docs/audit/REPOSITORY_REALITY.md) has each case.\n\n## AI agents\n\nAgents triage alerts and investigate incidents. What they can and cannot do:\n\n- **They read** the alert, its correlated siblings, entity context, and prior verdicts for the same signature.\n- **They call typed tools** — lake queries, graph traversals, enrichment lookups. The model picks the tool and passes arguments; it never writes SQL.\n- **Everything is logged** to the Investigation Ledger — prompts, tool calls, citations, verdict, token cost — and exports as a signed bundle.\n- **Grounding is checked.** A verdict citing an indicator the evidence never contained is demoted to human review rather than auto-closed.\n- **A prompt is validated before it is sent.** Raw logs, OCSF payloads and secret-shaped values are refused, not redacted after the fact.\n- **No vendor is touched without a human**, unless a tenant has explicitly granted autonomy for that verb. Every response step is graded against its own capability contract at dispatch, so approving a playbook never authorises whatever its steps happen to contain, and an approver must hold the required permission tier and must not be the person who requested the action.\n\n## Project maturity\n\n**Stable is defined, and a gate enforces it.** It was ungated prose until three rows were found describing coverage that did not exist. A row is Stable only with a check that runs on every pull request with no path filter, drives the real production path against real infrastructure, and has a **negative control proven by breaking the thing and watching it go red**. ([the bar](docs/audit/MATURITY_DEFINITION.md))\n\n| Capability | Status | Tested | Production ready |\n|---|---|---|---|\n| Ingest → detect → correlate → alert | Stable | E2E + unit | Yes |\n| Detection engine (2535 executable rules) of 6991 | Stable | Replay proof | Yes |\n| Alert correlation into incidents | Stable | Unit | Yes |\n| REST API + web console | Stable | Unit + integration | Yes |\n| AI triage + Investigation Ledger | Stable | Live Postgres ledger + a PR-gated local-model agent run. No hosted provider has been exercised | Yes, copilot mode |\n| Event lake + hunting (ClickHouse) | Stable | Live ClickHouse on the shipped DDL, with a negative control | Yes, `full` profile |\n| Retro-hunts when new intel arrives | Stable | Live ClickHouse + Kafka with the flag on, with a negative control | Opt-in, `full` profile |\n| 68-hunt YAML library, compiled against tenant events in the lake | Stable | Live ClickHouse: scheduled hunts read tenant data and refuse to fall back to the fixture; all 114 field names the corpus filters on compile, via a lake column or the stored payload | Yes, `full` profile |\n| SCIM 2.0 provisioning | Stable | Live Postgres through the real app: PATCH in both providers' dialects, group membership, deprovisioning proved by its consequences, with a negative control | Yes |\n| White-label branding | Stable | Live Postgres: the scheduled report and its covering email carry the organisation's brand, with two negative controls | Yes |\n| Usage metering | Stable | Live ClickHouse on the shipped DDL, with a negative control that makes an unscoped meter fail | Yes |\n| Entity graph (Neo4j) | Stable | Live Neo4j against the production reader, with a negative control | Yes, `full` profile |\n| Governed response actions | Stable | Live socket: permits, refuses, never leaks a refusal, with a negative control | Human-approved only |\n| Alert-triggered playbooks, with a durable approval pause | Stable | Live Postgres: suspend, restart, resume, expiry, with a negative control | Yes — three opt-ins deep, preview by default |\n| Per-tenant detection tuning in the live engine | Stable | Live Postgres: tuning written changes what the engine fires, with a negative control | Yes |\n| Scheduled connectors | Stable | Live scheduler polls a stub vendor into ingest, with a negative control | Yes |\n| UEBA | Stable | Live Postgres: migrations, scoring, persistence, isolation, with a negative control | Yes, `full` profile |\n| Package distribution (npm/PyPI) | Beta | The five PyPI packages resolve at the tree's versions, checked against the live registries in both directions with a self-test; the npm half has no registry state to verify | `pip install aisoc-sandbox` (and the other four) works today; the three npm packages still need the first-upload `NPM_TOKEN`, an account action |\n\n## What AiSOC is not\n\n- **Not a drop-in SIEM replacement.** It correlates and investigates; it does not replace long-term log retention and compliance search.\n- **Not able to see telemetry you have not connected.** There is no discovery.\n- **Not autonomous by default.** Response requires explicit policy authorization and a human approver.\n- **Demo and sample incidents are not real incidents**, and benchmark numbers are substrate self-consistency measures, not live agent accuracy — labelled as such wherever published.\n\n## Troubleshooting\n\n`make doctor` checks host tools, memory, disk, every port and each datastore by *querying* it rather than asking whether its container is up, then prints the command to run next. A container killed by a full Docker VM is named as that, not as the service that happened to die.\n([the six most common failures](https://beenuar.github.io/AiSOC/docs/installation#the-six-most-common-failures))\n\n## Security\n\nSecrets are generated per deployment and never committed; connector credentials are encrypted at rest. Services connect to Postgres as a DML-only role, so row-level security actually applies, and tenant isolation is enforced at the query layer in every store. RBAC gates every mutating route, ingest is authenticated, and the default install sends no prompt anywhere — the model runs beside it.\n\nSAML and OIDC sign-in with per-connection tenant and group mapping, SCIM provisioning, and console\n**MFA** — TOTP with recovery codes, enforceable per tenant. Attribute conditions, time-boxed\nelevation and workload identities run inside the one permission path, where a condition can only\nnarrow a permission and never grant one. ([SSO](apps/docs/docs/operations/enterprise-sso.md) ·\n[access governance](docs/security/access-governance.md))\n\n**A service with no credential refuses to serve rather than serving unauthenticated.** The [changelog](CHANGELOG.md) records each fix; report via [SECURITY.md](SECURITY.md).\n\n## Developing\n\n```bash\nmake test # unit tests for every service\nmake smoke # the golden pipeline, against a running stack\nmake stats # recount every figure this README publishes\n```\n\nGuides: [add a connector](https://beenuar.github.io/AiSOC/docs/plugins/hello-plugin) · [add a detection](https://beenuar.github.io/AiSOC/docs/detections/hello-hunt) · [plugin lifecycle](https://beenuar.github.io/AiSOC/docs/plugins/lifecycle). Every count above is recounted from the tree, and CI fails if this README disagrees.\n\n## Funding · Roadmap · Contributing · License\n\nDevelopment is funded and supported by **[Cyble](https://cyble.com)**, who pay for the engineering time behind AiSOC and release it under the MIT licence rather than keeping it. That buys no special treatment here — no Cyble-only features, no gated modules, no telemetry. ([full credits](.github/CREDITS.md))\n\n[ROADMAP.md](ROADMAP.md) · [CONTRIBUTING.md](CONTRIBUTING.md) · [SECURITY.md](SECURITY.md) · MIT\n"}