{"id":"de42c182-965c-4824-9988-3f2bec6b0291","slug":"truegrain","kind":"mcp","name":"truegrain","summary":"A semantic layer that refuses to answer wrongly. Compiles Apache Ossie models into governed, dialect-correct SQL, and declines the joins that would silently inflate a total. Self-hosted, MCP for agent","owner":"rk-chavali","repoUrl":"https://github.com/rk-chavali/truegrain","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-09-16T06:15:25.000Z","addedAt":"2026-10-21T09:41:05.003Z","logoUrl":"https://avatars.githubusercontent.com/u/113842175?u=13c69bd57ed75802b36c5afe4ac59839a6fa5906&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["databases","data-analytics","utilities"],"tags":["apache-ossie","bigquery","data-engineering","data-governance","gitops","golang","metrics-layer","postgresql"],"runtime":"go","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-27T19:03:21.578Z","sample":false,"url":"https://conduits.dev/tools/truegrain","language":"Go","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# truegrain\n\nAn Apache Ossie semantic model compiles to governed, dialect-correct SQL for\nBigQuery, Postgres and Snowflake, served over MCP for agents and JSON over HTTP\nfor everything else, with column access resolved before a query is ever sent.\n\nBigQuery and Postgres execute, and both are checked against the same fixture as\nDuckDB. Snowflake compiles and has no executor yet.\n\nDuckDB is in here too, as the local one: it makes the quickstart below need no\ncloud account, and it is what the correctness suite executes against, so the\nrefusals are tested rather than asserted. Nobody should serve a company from it.\n\nStatus: working engine, v1 scope in progress. See [what is not built](#not-built-yet).\n\n## What it does\n\n```\n$ truegrain compile -models ./models \\\n    -metric order_revenue,order_count \\\n    -dim customers.region,orders.order_date -grain month -limit 20\n\nSELECT\n  \"customers\".\"region\" AS \"region\",\n  date_trunc('month', \"orders\".\"order_date\") AS \"order_date\",\n  SUM(\"orders\".\"order_total\") AS \"order_revenue\",\n  COUNT(DISTINCT \"orders\".\"order_id\") AS \"order_count\"\nFROM \"main\".\"orders\" AS \"orders\"\n  LEFT JOIN \"main\".\"customers\" AS \"customers\" ON \"orders\".\"customer_id\" = \"customers\".\"customer_id\"\nGROUP BY 1, 2\nLIMIT 20;\n```\n\nAnd what it will not do:\n\n```\n$ truegrain compile -models ./models -metric order_revenue -dim order_lines.item_id\n\nrefused (fan_out_would_inflate): metric \"order_revenue\" aggregates SUM(...) over\norders, but this query repeats orders rows: order_lines is not unique on\norder_id, so joining it repeats every orders row once per matching order_lines row\n  hint: the total would be inflated, so the engine will not run it; drop the\n  dimensions or metrics that pull in relationship \"lines_to_orders\", or use an\n  aggregate that survives duplication such as COUNT(DISTINCT ...), MIN or MAX\n```\n\nOn the fixture data that query would have answered **2361.00** instead of\n**885.50**, an overstatement of 167% that looks entirely plausible. Silently\nreturning it is the failure mode that kills trust in a semantic layer, so the\nengine refuses instead.\n\n## Install\n\n```bash\n# A single static binary, no runtime and no toolchain needed\ncurl -fsSL https://github.com/rk-chavali/truegrain/releases/latest/download/truegrain_linux_amd64.tar.gz | tar xz\n\n# Or a container, against BigQuery\ndocker run --rm -v \"$PWD:/models\" ghcr.io/rk-chavali/truegrain:latest validate -models /models\n```\n\nEvery release carries `checksums.txt` and a build attestation, so a download\ncan be verified back to the commit and the workflow that produced it:\n\n```bash\ngh attestation verify truegrain_linux_amd64.tar.gz --repo rk-chavali/truegrain\ntruegrain version    # says which commit it came from\n```\n\nThe container deliberately does not bundle the DuckDB CLI. DuckDB is the local\nquickstart below; a container is what runs against a real warehouse.\n\n## Quickstart\n\nNo cloud account, no credentials, no C compiler.\n\n```bash\nmake build\nmake validate\nmake compile\n```\n\nTo run real queries, put the [DuckDB CLI](https://duckdb.org/docs/installation/)\non your PATH (a single executable) and:\n\n```bash\nmake demo\n```\n\n## Start from your warehouse\n\nThe quickstart runs on a model that ships with the repository. To get one for\nyour own data, `truegrain init` reads what the warehouse already knows:\n\n```bash\ntruegrain init\n```\n\n```\nWarehouse [bigquery]:\nBigQuery project [acme-analytics]:\nDataset: main\n\nReading acme-analytics.main\n\n  campaigns             4 columns   key: campaign_id\n  customers             4 columns   key: customer_id\n  order_lines           5 columns   key: order_id, line_number\n  orders                5 columns   key: order_id\n\n  3 relationship(s) from declared foreign keys\n    campaigns.order_id -> orders.order_id\n    order_lines.order_id -> orders.order_id\n    orders.customer_id -> customers.customer_id\n\n  wrote  truegrain.yaml\n  wrote  models/main/namespace.yaml\n  wrote  models/main/datasets.generated.yaml\n  wrote  models/main/metrics.yaml\n```\n\nEvery flag can be given instead of answered, which is how to run it\nunattended:\n\n```bash\ntruegrain init -dialect bigquery -project acme-analytics -dataset main -location US\n```\n\n**It reads no rows.** Tables, columns, types, primary keys and foreign keys,\nthrough the metadata API. So it needs `roles/bigquery.metadataViewer` and\nnothing else: no ability to run a query, no access to a single value. That is a\ncredential somebody will give a tool they are still deciding about.\n\n**The keys are the point.** Columns alone produce a model that compiles and\ncannot tell that joining `order_lines` multiplies `orders` rows. Grain comes\nfrom the primary key and joins come from the foreign keys, so a table that\ndeclares neither is named in the output and again in the generated file:\n\n```\n! events declares no primary key, so its grain cannot be derived and a\n  fan-out through it cannot be detected\n```\n\nNothing is guessed. A guessed key is worse than a missing one, because the\nengine would believe it knows the grain, pass the fan-out check, and inflate a\nsum with complete confidence.\n\n**Metrics are not generated.** A schema can say an order has a total. Only you\ncan say which of those totals the business calls revenue, and a tool that\nguesses produces a catalogue of plausible metrics nobody trusts. `metrics.yaml`\narrives with one commented example built from a real column in your model, and\n`init` never writes that file twice.\n\nRe-run it after a schema change with `-force`. That rewrites\n`datasets.generated.yaml` and leaves your metrics, your namespace manifest and\nyour `truegrain.yaml` alone.\n\nThen the loop the rest of this README describes:\n\n```bash\ntruegrain validate -models .    # does the model hold together\ntruegrain test     -models .    # does it answer, and refuse, what it should\ntruegrain doctor   -models .    # does the warehouse still agree with it\ntruegrain query    -models . -metric main.order_revenue -dim main.customers.region\n```\n\n## Already have a semantic layer\n\n```bash\ndbt parse                                    # writes target/manifest.json\ntruegrain import -from dbt    -manifest target/manifest.json -dialect postgres\ntruegrain import -from lookml -project ./my-looker-project   -dialect bigquery\ntruegrain import -from cube   -project ./model               -dialect postgres\n```\n\nNo warehouse credentials for any of them: each reads files.\n\nEach source already declares what fan-out detection needs. dbt's MetricFlow\nentities are the grain and the join graph; Looker and Cube state join\ncardinality directly, so `one_to_many` arrives as a fan-out nobody had to\ninfer. An imported project therefore starts refusing questions the source\nanswered. That is the point, and it is worth being ready for:\n**importing can reveal that a number somebody has been reading for a year is\ninflated.** The engine refuses it rather than reproducing it, and names the\nmetric that answers correctly.\n\nWhat does not come across is printed in full, not counted: cumulative and\nconversion metrics are window functions this engine refuses by name, a\nderived metric can compose metrics at different grains, and a semi-additive\nmeasure is the thing the whole project exists to not guess at. Each is\nreported with the reason, because a catalogue quietly missing a third of its\nmetrics is worse than one that refused to import.\n\n## Assert what the model answers\n\n`validate` says the model holds together and `diff` says a number changed.\nNeither says a number was ever right. `truegrain test` is where you say so.\n\n```yaml\n# tests/retail.yaml\nversion: 1\ntests:\n  - name: total revenue is 885.50\n    metrics: [order_revenue]\n    expect_rows:\n      - [885.50]\n\n  - name: revenue by item is refused, because it would inflate\n    metrics: [order_revenue]\n    dimensions: [order_lines.item_id]\n    expect_refusal: fan_out_would_inflate\n```\n\n```bash\ntruegrain test -models . -tests ./tests\n```\n\nThe second case is the one worth copying, and it is the one no other semantic\nlayer can express. **A refusal assertion needs no warehouse**: the planner\ndeclines a fan-out before any SQL exists, so it runs in a pull request with no\ncredentials configured, alongside `validate` and `diff`. What a model refuses\nis the most valuable thing to test about it and the cheapest thing to check.\n\nRow assertions need a warehouse. A run without one reports how many cases it\ncould not check rather than printing a clean pass over work it did not do, and\n`-fail-on-skip` turns that into a failure for a pipeline that believes it is\nchecking rows. `-json` emits a report for a CI step to read.\n\nNumbers compare numerically, so `750`, `\"750.00\"` and `750.0` all match: which\nof those arrives depends on the warehouse driver, not on the model. `tolerance`\nexists for division, which genuinely differs between warehouses, and nothing\nelse should need it.\n\nToday `init` reads BigQuery and PostgreSQL. For Postgres, name the variable\nholding the connection string rather than passing it:\n\n```bash\nexport DATABASE_URL=postgres://reader@localhost:5432/warehouse?sslmode=require\ntruegrain init -dialect postgres -dsn-env DATABASE_URL -dataset public\n```\n\nIt reads `pg_catalog` only, never a row, so the credential it needs is CONNECT\nplus USAGE on the schema. DuckDB and Snowflake have no reader: those models are\nwritten by hand and then queried like any other. See\n[what is not built](#not-built-yet).\n\n## Try it in a browser\n\nThe console is a separate React application at\n[truegrain-console](https://github.com/rk-chavali/truegrain-console). It is a\nstatic build you host yourself: explore the model, read what is and is not\nenforced, and copy the client code for a query you just ran.\n\n```bash\ntruegrain serve rest -models ./models -db ./demo.duckdb   -identity you@example.com -token-env SEMANTIC_TOKEN   -cors-origin http://localhost:5180\n```\n\n`-cors-origin` names the origins a browser client may read responses from. It\nis off by default and never accepts a wildcard: on an engine running without\nauthentication, which is the normal local configuration, a wildcard would let\nany website you happen to visit read your entire model.\n\n## Non-negotiables\n\nThese shape every other decision in the codebase.\n\n1. **No `run_sql` escape hatch on any interface.** The only expressible request\n   is a semantic one. `TestNoRawSQLTool` and `TestNoEndpointAcceptsSQL` guard it.\n2. **Governance resolves at compile time.** An unauthorized plan never becomes\n   SQL. `TestDeniedProducesNoSQL` asserts the absence of a compiled statement,\n   not just the presence of an error.\n3. **Never a shared admin service account.** Identity is passed to the executor\n   on every call so queries run as the caller.\n4. **The model format is Ossie.** No proprietary YAML dialect. The engine reads\n   the published `tpcds_semantic_model.yaml` from the Apache Ossie repository\n   unmodified; `TestUpstreamExamples` proves it on every run.\n5. **Correctness is proven in CI**, not claimed here.\n\n## More than one team\n\nA directory of Ossie files is either one model, in which case every team shares\none flat namespace and collides, or several models, which most loaders refuse.\nNeither works once two teams own definitions. So each team's directory carries\nits own manifest:\n\n```\ndomains/\n  sales/\n    namespace.yaml        # name, owners, exports\n    sales.yaml            # stock Ossie, unmodified, portable\n  marketing/\n    namespace.yaml        # name, owners, imports\n    campaigns.yaml\n```\n\nNothing is shared, so there is no root file for twelve teams to contend on.\nAdding a team is a new directory.\n\n```yaml\n# domains/sales/namespace.yaml\nversion: 1\nname: sales\nowners: [\"@acme/sales-ops\"]\nexports:\n  datasets: [orders]      # private by default; this is the whole public surface\n```\n\n```yaml\n# domains/marketing/namespace.yaml\nversion: 1\nname: marketing\nowners: [\"@acme/growth\"]\nimports:\n  - namespace: sales\n    datasets: [orders]    # both sides opt in\n```\n\n**Bare names inside a namespace, qualified names outside.** Marketing's model\nfile writes `SUM(orders.order_total)` and stays a valid Ossie document any\nother OSI tool can read. The API returns `marketing.attributed_revenue`, and\naccepts a bare name when it is unambiguous. All composition metadata lives in\nour manifests, never in the customer's models.\n\n```bash\nmake teams\n```\n\nThree properties that suite demonstrates:\n\n- **Composition works.** Marketing aggregates a dataset sales exported, joined\n  across the namespace boundary.\n- **Metrics from two namespaces combine.** Each is aggregated separately and the\n  results are joined on a dimension both can reach. See below.\n- **A grant follows its column.** The revenue tag is written by sales against\n  `sales.orders.order_total`. Marketing reads that column through the import and\n  is denied by the same grant. Governance resolves against where a field was\n  defined, never the name it is used by, so aliasing an import is not a bypass.\n\nOne team cannot take the layer down for everyone: `validate` fails hard on any\nnamespace error, and `serve` loads what is valid, marks the rest unavailable in\n`health`, and refuses queries touching them with a named reason.\n\n## Facts at different grains\n\n\"Revenue and marketing spend by month\" asks for two facts that live at different\ngrains. One flat join cannot answer it: joining orders to campaigns, or an order\nheader to its lines, repeats one side and inflates any sum over it.\n\nThe engine aggregates each fact separately at the shared dimension grain and\njoins the grouped results, which is the fix `docs/03-semantic-model.md`\nprescribes for the chasm trap.\n\n```sql\nWITH \"fact_1\" AS (\n  SELECT date_trunc('month', \"orders\".\"order_date\") AS \"order_date\",\n         SUM(\"orders\".\"order_total\") AS \"order_revenue\"\n  FROM \"main\".\"orders\" AS \"orders\" GROUP BY 1\n),\n\"fact_2\" AS (\n  SELECT date_trunc('month', \"orders\".\"order_date\") AS \"order_date\",\n         SUM(\"campaigns\".\"spend\") AS \"campaign_spend\"\n  FROM \"main\".\"campaigns\" AS \"campaigns\"\n    LEFT JOIN \"main\".\"orders\" AS \"orders\" ON \"campaigns\".\"order_id\" = \"orders\".\"order_id\"\n  GROUP BY 1\n)\nSELECT COALESCE(\"fact_1\".\"order_date\", \"fact_2\".\"order_date\") AS \"order_date\",\n       \"fact_1\".\"order_revenue\", \"fact_2\".\"campaign_spend\"\nFROM \"fact_1\"\n  FULL OUTER JOIN \"fact_2\" ON \"fact_1\".\"order_date\" IS NOT DISTINCT FROM \"fact_2\".\"order_date\"\n```\n\nThree details that are not incidental. The join is **full outer**, because a\nmonth with orders and no campaigns is a real row and an inner join would\nunderstate it. The condition is **null safe**, because a null dimension value is\na real group. And the dimension is **coalesced** across parts rather than read\nfrom the first, for the same reason.\n\nParts may come from different namespaces. Two parts share a dimension when their\nfields have the same governed origin, which is the identity an imported dataset\npreserves.\n\nThe governance gate runs over **every** part before any of them is emitted.\nChecking only the first would let every later metric escape it.\n\n### What is still refused, and why\n\nA sum over the order header grouped by a product dimension:\n\n```\nrefused (fan_out_would_inflate): metric \"order_revenue\" aggregates SUM(...) over\norders, but this query repeats orders rows\n  hint: the total would be inflated, so the engine will not run it. Metrics\n  defined at the order_lines grain answer this correctly: line_revenue,\n  units_sold. ...\n```\n\nThat one is not a limitation. An order contains several products, so \"order\nrevenue for hardware\" has no answer without an allocation rule, and summing it\nacross categories would exceed the real total. The refusal names the metrics\ndefined at the grain where the question *is* well defined.\n\nLikewise, a namespace that has not imported a dataset cannot group by it, and\nthe refusal says exactly which file to edit:\n\n```\nrefused (cross_namespace_query): namespace \"marketing\" cannot group by\n\"sales.customers.region\", which belongs to \"sales\"\n  hint: add the dataset to `imports` in marketing's namespace.yaml, and to\n  `exports` in sales's, or drop this dimension\n```\n\n## Answering from a pre-aggregated table\n\nA `rollups.yaml` beside the model declares which pre-aggregated tables\nexist, what each holds, and how each proves it is current:\n\n```yaml\nversion: 1\nrollups:\n  - name: orders_by_region_month\n    source: main.rollup_orders_daily\n    dimensions:\n      - {field: customers.region,   column: region}\n      - {field: orders.order_date,  column: order_month, grain: month}\n    metrics:\n      - {metric: order_revenue, column: order_revenue}\n    freshness:\n      column: built_at\n      max_age: 26h\n```\n\nA question the planner can answer from one is; a question it cannot is\nanswered from the source tables. Nothing is ever refused because of a\nrollup.\n\nA rollup is a second copy of a number, which is the thing this engine\nexists not to do quietly, so every rule is a reason not to route:\n\n- **A freshness check is required.** No `column` and `max_age`, no load.\n  The engine asks the warehouse for `MAX(column)` before reading the table\n  and falls back when it is stale. An engine with no executor never routes,\n  because compiled output must not depend on state nothing verified.\n- **Only distributive metrics.** SUM, COUNT, MIN and MAX can be recomputed\n  from partial aggregates. `COUNT(DISTINCT ...)`, `AVG` and any ratio\n  cannot, and listing one fails the load.\n- **The recombining function is derived, not declared.** A stored COUNT\n  recombines with **SUM**, because counting the rollup's rows counts groups\n  and COUNT there returns a smaller number in the right units that nothing\n  would flag. There is nowhere in the file to get this wrong.\n- **Governance is unchanged.** The plan is built against the model and only\n  then rewritten, so a grant written on `customers.region` keeps applying\n  when a rollup carries a copy of that column.\n\nEvery answer that came from one says so, on the response, in the audit log\nand on `GET /v1/health`, which also reports when each rollup was last\nprobed and why one is not being used. Details in\n[docs/03-semantic-model.md](docs/03-semantic-model.md).\n\n## Interfaces\n\nFive surfaces, one planner, one request vocabulary.\n\n| Surface | Command |\n|---|---|\n| CLI | `truegrain validate \\| test \\| compile \\| query \\| diff \\| health` |\n| MCP (stdio or streamable HTTP) | `truegrain serve mcp` |\n| REST / JSON | `truegrain serve rest` |\n| PostgreSQL wire protocol, for BI tools | `truegrain serve postgres` |\n| GraphQL, for a front end that speaks it | `POST /graphql` |\n\nA BI tool connects to `serve postgres` as it would to any PostgreSQL. What it\nsends is parsed into a semantic request and never forwarded as text, so the\ngovernance gate, the fan-out refusal and the audit record all still apply: a\nquestion that would inflate a total comes back as an `ERROR` whose `HINT`\nnames the metric that answers it correctly.\n\nThe MCP tool surface is fixed at four tools: `list_metrics`,\n`describe_metric`, `list_dimensions`, `query`. There is no fifth, and adding one\nthat accepts SQL fails the test suite.\n\n### Queries that outlive the caller\n\nA warehouse query can run longer than the default HTTP timeout of every agent\nframework in use. Synchronously the client gives up, retries, and the warehouse\nruns and bills the query twice for an answer nobody reads.\n\n```python\nresult = client.run(metrics=[\"sales.order_revenue\"], dimensions=[\"sales.customers.region\"])\n```\n\n`run` submits, polls with backoff, collects every page and returns one result,\nso no caller writes a polling loop. Underneath it is `POST /v1/jobs`, `GET\n/v1/jobs/{id}` and `DELETE /v1/jobs/{id}`.\n\nThree properties are not incidental:\n\n- **Compilation is synchronous**, so the governance gate runs before the job\n  exists. A request you may not make is refused on submit, not discovered by\n  polling. A 202 is a promise the query was authorized.\n- **A job is readable only by the identity that submitted it.** Another caller\n  gets 404 rather than 403, so job identifiers cannot be probed for existence.\n- **Abandoning a wait cancels the job**, rather than leaving the warehouse\n  spending on an answer nobody is waiting for.\n\nMCP deliberately does not get this. An MCP client is a local process over stdio\nwith no request timeout to outlive, so a fifth tool would be cost with no\nbenefit.\n\n### Connecting Claude Code\n\n```bash\nclaude mcp add truegrain -- /path/to/truegrain serve mcp -models /path/to/models -db /path/to/demo.duckdb\n```\n\n## Pointing it at BigQuery\n\n```bash\nexport SEMANTIC_TOKEN=...   # or use a real authenticator, see below\ntruegrain serve rest \\\n  -models ./models -policy ./policy.yaml \\\n  -dialect bigquery -project my-project -location US \\\n  -impersonate -token-env SEMANTIC_TOKEN\n```\n\nCredentials are Application Default Credentials and nothing else. There is no\nkey file path in any config field, because a key path in a project file becomes\na key file in a git repository.\n\nTwo behaviours are worth knowing before the first invoice.\n\n**Every query is estimated before it runs.** A dry run reports the bytes the\nstatement would scan, and anything over the cap is refused rather than run:\n\n```\nrefused (query_scans_too_much): this query would scan 1.4 TiB, over the\n200.0 GiB limit\n  hint: narrow it with a filter on the partitioning column, use a coarser\n  grain, or raise the limit deliberately with -max-bytes-billed\n```\n\nThe refusal classifies as `modify`, so an agent adds a filter instead of giving\nup. `MaximumBytesBilled` is also set on the real job as a backstop, because the\nestimate is not exact for clustered tables and materialized views.\n\n**`-impersonate` runs each query as the calling service account**, which is what\nmakes the warehouse's own row and column security apply to the caller rather\nthan to this process. It needs `roles/iam.serviceAccountTokenCreator` on the\nidentities it acts as. Without it the engine still works and `health` says so in\nfull:\n\n```\nexecutor  bigquery (my-project); impersonation is off, so every query runs as\n          this process's own service account and the warehouse cannot tell\n          callers apart; capped at 200.0 GiB scanned per query\n```\n\nUse `-dry-run` to estimate every query and execute none, which is how to point\nthe engine at a real warehouse and validate a model against real schemas\nwithout spending anything.\n\n## Which commands need warehouse credentials\n\nMost of them do not, and that is worth knowing before wiring anything into a\npipeline. A continuous integration job that checks a model should not hold a\ncredential that can read your customers.\n\n| Command | Warehouse access | BigQuery role |\n|---|---|---|\n| `validate` | none | none |\n| `diff` | none | none |\n| `compile` | none | none |\n| `init` | metadata: names, types, keys | `roles/bigquery.metadataViewer` |\n| `doctor` | metadata: names and types | `roles/bigquery.metadataViewer` |\n| `query`, `serve` | reads rows, runs jobs | `roles/bigquery.jobUser` plus `roles/bigquery.dataViewer` |\n\nOn Postgres the same split is a database role with `SELECT` on the tables the\nmodel reads, and nothing else. Every query runs in a read-only transaction, so\nthe server refuses a write even if the emitter ever produced one.\n\n`validate`, `diff` and `compile` read YAML and emit SQL. They never open a\nconnection, which is why the pull request check below runs with no cloud\ncredential at all.\n\n`init` and `doctor` read metadata through the API. No query job runs, so they\ncost nothing and cannot read a value.\n\n### Three identities, not one\n\nA pipeline that uses one service account for everything gives its weakest job\nits strongest permissions. Split them by what each actually does:\n\n```bash\n# 1. CI: checks the model against the warehouse's shape. Metadata only.\ngcloud iam service-accounts create truegrain-ci \\\n  --display-name \"truegrain model checks\"\n\ngcloud projects add-iam-policy-binding acme-analytics \\\n  --member \"serviceAccount:truegrain-ci@acme-analytics.iam.gserviceaccount.com\" \\\n  --role roles/bigquery.metadataViewer\n\n# 2. The engine: runs queries. The only identity that reads a row.\ngcloud iam service-accounts create truegrain-engine \\\n  --display-name \"truegrain query engine\"\n\ngcloud projects add-iam-policy-binding acme-analytics \\\n  --member \"serviceAccount:truegrain-engine@acme-analytics.iam.gserviceaccount.com\" \\\n  --role roles/bigquery.jobUser\n\n# Grant data access per dataset, not project-wide.\nbq add-iam-policy-binding \\\n  --member \"serviceAccount:truegrain-engine@acme-analytics.iam.gserviceaccount.com\" \\\n  --role roles/bigquery.dataViewer \\\n  acme-analytics:main\n```\n\nThe third identity is not a service account. It is the API token a pipeline\nuses to call a **running** engine, and it is narrowed with scopes rather than\nIAM:\n\n```bash\ntruegrain serve rest -models ./models -token-env CI_TOKEN -token-scopes read:model\n```\n\nThat credential can list metrics, describe them and compile SQL. It cannot make\nthe warehouse do work. See [A token that cannot read your\ndata](#a-token-that-cannot-read-your-data).\n\nAdd `roles/iam.serviceAccountTokenCreator` to the engine's identity only if you\nrun it with `-impersonate`, which makes each query execute as the caller so the\nwarehouse's own row and column security applies to them rather than to the\nengine.\n\n### No key files in CI\n\nWorkload Identity Federation lets a GitHub Actions run authenticate to Google\ndirectly. Nothing is downloaded, nothing is stored in a repository secret, and\nthere is no key to rotate or leak:\n\n```bash\ngcloud iam workload-identity-pools create github \\\n  --location global --display-name \"GitHub Actions\"\n\ngcloud iam workload-identity-pools providers create-oidc github \\\n  --location global --workload-identity-pool github \\\n  --issuer-uri \"https://token.actions.githubusercontent.com\" \\\n  --attribute-mapping \"google.subject=assertion.sub,attribute.repository=assertion.repository\" \\\n  --attribute-condition \"assertion.repository == 'acme/analytics-models'\"\n\ngcloud iam service-accounts add-iam-policy-binding \\\n  truegrain-ci@acme-analytics.iam.gserviceaccount.com \\\n  --role roles/iam.workloadIdentityUser \\\n  --member \"principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github/attribute.repository/acme/analytics-models\"\n```\n\nThe `--attribute-condition` is the part worth getting right. Without it, any\nrepository on GitHub can mint a token for this service account.\n\n```yaml\nname: Model\non: [pull_request, schedule]\n\npermissions:\n  contents: read\n  id-token: write        # required to mint the OIDC token\n  pull-requests: write\n\njobs:\n  # No credential. Nothing here talks to a warehouse.\n  meaning:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: rk-chavali/truegrain@v0.3.0\n        with:\n          models: .\n          against: ${{ github.base_ref }}\n\n  # Metadata only, as truegrain-ci. Cannot read a row.\n  drift:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: google-github-actions/auth@v2\n        with:\n          workload_identity_provider: projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github/providers/github\n          service_account: truegrain-ci@acme-analytics.iam.gserviceaccount.com\n      - run: truegrain doctor -models . -dialect bigquery -project acme-analytics\n```\n\n`doctor` exits 5 when the warehouse disagrees with the model, so the schedule\nfinds drift by looking rather than by a caller hitting it.\n\nEverywhere else, credentials are Application Default Credentials and nothing\nelse: `gcloud auth application-default login` on a laptop, the attached service\naccount on Cloud Run or GKE, `GOOGLE_APPLICATION_CREDENTIALS` where you must.\nThere is no key file path in any config field, because a key path in a project\nfile becomes a key file in a git repository.\n\n## Who is calling\n\nEverything this engine promises about governance rests on the identity being\nthe caller's. Three authenticators ship, and they narrow as they get weaker.\n\n```bash\n# OpenID Connect: Okta, Entra, Auth0, Keycloak, Google\ntruegrain serve rest -oidc-issuer https://acme.okta.com/oauth2/default                     -oidc-audience semantic-prod -groups-claim groups\n\n# Google-signed identity tokens: Cloud Run, GKE, Compute Engine\ntruegrain serve rest -google-audience https://semantic.acme.internal\n\n# One shared token: a single tenant, and nothing more\ntruegrain serve rest -token-env SEMANTIC_TOKEN\n```\n\nVerification is delegated to `go-oidc` and Google's own verifier rather than\nhand written. Signature checking, discovery and key rotation are where JWT\nauthentication goes wrong, and a bespoke parser is the most common way to end\nup accepting a forged token.\n\nFour things fail closed rather than defaulting to permissive:\n\n- **An audience is mandatory.** Without one, every token the issuer ever minted\n  for any application would authenticate here. That is the most expensive\n  common OIDC misconfiguration, so the server refuses to start instead.\n- **Only asymmetric algorithms are accepted.** With HMAC in the list a caller\n  can sign a token using the provider's public key as the shared secret. `alg:\n  none` is rejected for the obvious reason.\n- **A plaintext issuer is refused**, because keys fetched over HTTP can be\n  substituted in transit and the verification becomes decorative.\n- **An unverified `email` claim is not an identity.** It is caller-supplied, and\n  the subject is exactly what the policy file and the warehouse grant key on.\n\nServing on anything but loopback without authentication is refused outright. A\nstatic token authenticates but maps every caller to one subject, so the audit\nlog cannot tell them apart; the server says so on a non-loopback bind.\n\nWhat none of this defends against: a stolen token works until it expires. Serve\nover TLS and keep lifetimes short.\n\n## Clients\n\nThe SDKs live in their own repositories, so a Java contributor does not clone Go\nto fix a Python bug and each one releases on its own cadence.\n\n| Language | Install | Repository | Reference |\n|---|---|---|---|\n| Python | `pip install truegrain` | [truegrain-python](https://github.com/rk-chavali/truegrain-python) | [pages](https://rk-chavali.github.io/truegrain-python/truegrain.html) |\n| TypeScript | `npm install truegrain` | [truegrain-typescript](https://github.com/rk-chavali/truegrain-typescript) | [pages](https://rk-chavali.github.io/truegrain-typescript/) |\n| Go | `go get github.com/rk-chavali/truegrain-go` | [truegrain-go](https://github.com/rk-chavali/truegrain-go) | [pkg.go.dev](https://pkg.go.dev/github.com/rk-chavali/truegrain-go) |\n\nThey are hand-written rather than generated. A generator produces a transport\nand a flat error type; the parts worth having are the ones it cannot produce,\nnamely the refusal semantics an agent branches on and a `run()` that submits a\njob and polls it so no caller writes that loop.\n\nEach client vendors `api/openapi.yaml` and checks itself against it in both\ndirections, so an operation this engine gains that a client cannot call is a\nfailing build in that client rather than a discovery made in production.\n\n## A token that cannot read your data\n\nOne bearer token used to grant everything, so a continuous integration job\nthat only validated a model held a credential that could also read every row\nits identity was allowed.\n\n```bash\ntruegrain serve rest -models ./models -token-env CI_TOKEN -token-scopes read:model\n```\n\nThat credential lists metrics, describes them and compiles SQL. It cannot\nexecute anything:\n\n```\ncode:   out_of_scope\nreason: this credential is not permitted to run queries against the warehouse\nhint:   it holds read:model; use a credential with run:query, or ask an\n        operator to widen this one\nretry:  never\n```\n\nTwo scopes, `read:model` and `run:query`, because the line worth drawing is\nbetween reading what the model says and making the warehouse do work. That is\nthe line between a cheap secret and an expensive one.\n\nNaming no scopes grants all of them, so nothing changes for a deployment that\nhad one token. Naming any makes the list exhaustive: `run:query` does not\nimply `read:model`, because \"it can already run queries so it may as well read\nmetadata\" is how a scope system turns back into one token. A scope that is not\nrecognised fails at startup rather than being dropped, since a typo would\notherwise produce a credential narrower than intended and fail somewhere far\nfrom the mistake.\n\n## Governance\n\nColumn access is resolved between the planner and the emitter. A denied request\nreturns before any SQL exists: there is no compiled statement to leak and no\nwarehouse job to cancel.\n\n```bash\ntruegrain health -models ./models -policy ./policy.yaml\n```\n\n`health` reports what the deployment actually enforces, including what it does\nnot. Overstating a governance guarantee is worse than not offering one.\n\nThree resolvers ship today:\n\n- **file** — a policy file next to the model. Real compile-time enforcement,\n  provable on any machine with no cloud account. It governs queries that go\n  through this engine and places no control on the warehouse itself.\n- **allow-all** — for local work against fixture data. It says so in `health`.\n\n- **bigquery-policy-tags** — the warehouse's own control. The tags are already\n  on your columns and the caller's own IAM decides, so nothing is restated in\n  our format and nothing drifts.\n\n```bash\ntruegrain serve rest -dialect bigquery -project my-project -policy-tags -impersonate\n```\n\nA column with no policy tag is readable, which is what BigQuery itself does. A\ntagged column requires `roles/datacatalog.fineGrainedReader`, and the check is\nmade **as the caller** rather than as this process. Evaluating the tag's IAM\npolicy locally would be wrong: it misses bindings inherited from the project,\nfolder and organization, conditional bindings, and group expansion, so it would\nreport access the warehouse refuses and refuse access it allows.\n\nA field whose expression does not reduce to a single column is treated as\nreading every tagged column in its table. That can deny a field which would\nhave been allowed. It can never allow one that should have been denied, which\nis the direction to be wrong in.\n\n`-policy-tags` and `-policy` are refused together. They are two sources of\ntruth for one decision, and silently preferring one would leave an operator\nbelieving the other was in force.\n\nThis pairs with a modelling tool that writes the tags: classify a column once,\nand the tag is enforced both in the warehouse and here, before the query is\nsent.\n\n## Does this change a number\n\nA model edit that still validates, still passes every test, and quietly moves\nevery dashboard is the failure mode this project exists to prevent. Reviewing a\nYAML diff does not catch it: the YAML change is small and the consequence is not.\n\n`truegrain diff` compares compiled SQL rather than model text, so renaming a\ndescription is invisible and changing a join, a grain or an expression is not.\n\n```bash\ngit worktree add /tmp/base origin/main\ntruegrain diff -models . -against /tmp/base\n```\n\n```\nChanged meaning (4). These compile differently than before:\n  ~ sales.shipped_revenue\n  ~ sales.shipped_revenue by sales.customers.region\n  ...\n```\n\nExit 0 when nothing changed, 4 when something did, so CI can gate on it. A\nrequest that used to be refused and now compiles counts as a change, and so does\nthe reverse.\n\n### In a pull request\n\nThe action runs the same comparison and leaves the answer where the review\nhappens. It edits one comment rather than adding a new one per push.\n\n```yaml\nname: Semantic model\non: pull_request\n\npermissions:\n  contents: read\n  pull-requests: write\n\njobs:\n  model:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: rk-chavali/truegrain@v0.3.0\n        with:\n          models: .\n          against: ${{ github.base_ref }}\n```\n\nThe comment it writes:\n\n> **4 changed meaning.** This pull request changes what the model computes.\n>\n> **Changed meaning (4)** — these still compile. They compute something different.\n> - `sales.shipped_revenue`\n> - `sales.shipped_revenue by sales.customers.region`\n\nOmit `against` to validate only. Add `fail-on-change: true` for a model nobody\nshould change without a conversation, and `full: true` to include the SQL on\nboth sides. It exposes `changed`, `removed`, `altered` and `added` as outputs,\nso a later step can decide something.\n\nBoth commands also speak JSON, for a pipeline that is not GitHub Actions:\n\n```bash\ntruegrain validate -models . -format json\ntruegrain diff -models . -against /tmp/base -format json\n```\n\n`validate` emits JSON on failure too, with one entry per problem, because the\nrun worth parsing is the one that failed.\n\n## Is the model still true\n\n`diff` compares a model against its previous self. `doctor` compares it against\nthe warehouse it claims to describe.\n\n```bash\ntruegrain doctor -models . -dialect bigquery -project my-project\n```\n\n```\n4 table(s) checked.\n\n  ! sales.customers\n      this table does not exist\n      in main.customer_dim\n      every query touching this dataset fails; fix the source or drop the dataset\n\n  ! marketing.orders.order_total\n      reads column \"order_total_amount\", which this table does not have\n      in main.orders\n      every query using this field fails\n\n2 will break a query.\n```\n\nMetadata only: it reads no rows, so it needs no more than permission to list a\nschema, and it is cheap enough to run on a schedule. Drift is found by looking\nregularly rather than by a caller hitting it.\n\nExit 0 when the warehouse agrees, 5 when it does not, and `-format json` for a\npipeline. A type the warehouse contradicts is a warning rather than an error,\nbecause the query still runs and only its meaning has moved.\n\nAn executor that cannot introspect says so rather than reporting the model as\nhealthy, which is the one answer worth never giving.\n\n### Which rows, not just which columns\n\nA tag decides whether a caller may read the email address. This decides whether\na regional manager may read another region's revenue, which until now was yes,\nbecause nothing asked.\n\n```yaml\nrows:\n  - namespace: sales\n    principal: \"@acme/northeast\"\n    description: The northeast team sees northeast orders only.\n    filter:\n      dimension: customers.region\n      op: eq\n      values: [\"NE\"]\n```\n\n```\n$ truegrain query -metric sales.order_revenue -identity sam@acme.com -groups @acme/northeast\n750.00\n\n$ truegrain query -metric sales.order_revenue -identity chris@acme.com\n885.50\n```\n\nThe filter is added to the request before it is planned, so the planner and the\ngate both see the narrowed query and no second path exists that could skip it.\nIt combines with the caller's own filters by AND, so narrowing a query can\nnever widen it, and there is no request field that names it, so it cannot be\nremoved. A filter whose dimension no longer resolves refuses the query rather\nthan being dropped: silently ignoring a restriction is a data leak wearing the\ncostume of a permissive default.\n\nA policy restricts the principals it names and nobody else, which matches how\nan untagged column is readable. `health` reports how many principals are\ncovered so a partial policy is visible rather than assumed complete, and does\nnot name them.\n\n## Correctness\n\n```bash\ngo test ./...          # everything that needs no database\nmake test-all          # adds the parity suite against real DuckDB\n```\n\n| Suite | What it proves |\n|---|---|\n| `internal/osi` | The expression parser handles every construct the Ossie spec marks REQUIRED, and the decomposability table matches the spec row for row |\n| `internal/resolve` | The engine reads a published upstream Ossie model, and refuses an ontology document by name |\n| `internal/plan` | Fan-out and chasm joins are refused; safe joins are not |\n| `internal/dialect` | Committed golden SQL per dialect, so an emitter change is a readable diff. Filter values never appear in SQL text |\n| `internal/engine` | The governance suite from `docs/06-validation.md`, minus the GCP-specific rows |\n| `internal/exec` | Real SQL against real data: totals agree with and without a grouping dimension, and metrics match hand-computed answers |\n| `internal/serve/*` | No interface exposes raw SQL; refusals carry an actionable code and hint |\n| `internal/govern` (tags) | An untagged column is readable, every tag on a column must allow, a computed field requires every tag in its table, and a lookup failure is an error rather than a decision |\n| `internal/govern` | Fail-closed on a policy outage, no cache leakage across identities or groups, a revoked grant takes effect, denials do not disclose the schema |\n| `internal/serve/rest` (auth) | A missing audience and a plaintext issuer fail to start, only asymmetric algorithms are accepted, an unverified email is not an identity, and no error echoes the credential |\n| `internal/exec` (BigQuery) | The spend cap refuses before the query runs, `health` states when impersonation is off, and bound parameters match the placeholders the emitter writes |\n| `internal/config` | Environment values are required rather than defaulted to empty |\n| `cmd/truegrain` | The built binary: exit codes, and that every command resolving access records the decision |\n| `internal/rollup` | A rollup may only hold metrics that can be recomputed from partial aggregates, a stored COUNT recombines with SUM, and a rollup with no freshness check does not load |\n| `internal/engine` (rollups) | Against real DuckDB: a rollup answers the same total as the fact tables, a stale one is never read, a grain finer than the rollup falls back, and governance still resolves against the model |\n| `internal/serve/pgwire` | A catalogue statement is answered with the caller's own filter applied or declined outright, and a hostile name is refused rather than carried |\n| `internal/govern` (rotation) | A rotation has no window where neither token works, an expired token stops working, and an unreadable file does not lock everybody out |\n| `internal/config` (environments) | An overlay changes only what it mentions, and an unknown `-env` is an error rather than a fallback to the base configuration |\n\n## Architecture\n\n```\nOssie YAML → parser → resolver → planner → governance gate → emitter → executor\n                                              ↓ denied\n                                         structured refusal\n```\n\n`internal/plan` holds no SQL strings, so correctness is unit-testable without a\ndatabase. `internal/engine` is the only thing that owns a dialect, so no\ninterface can emit SQL without the gate having run.\n\nThe engine reads Apache Ossie, which carries no grain, additivity or cardinality\nfields. All three are derived; `internal/osi/COMPLIANCE.md` records every\nconstruct, its status, and each place this engine diverges from the spec.\n\n## Not built yet\n\nStated plainly rather than implied by omission.\n\n- **No executor has ever run Databricks, ClickHouse, Athena, Trino or\n  Redshift.** All five dialects compile and their output is golden tested,\n  and not one of them has a way to execute. Each says so in its own\n  capabilities rather than leaving it to be found.\n- **The Snowflake executor has never run against a real account.** It speaks\n  the SQL REST API with a key-pair assertion, and everything about it that\n  would be this engine's fault, the assertion, the bindings, the decimal\n  scale, the partition paging, is tested. That Snowflake accepts what is\n  sent is not, and `Name()` reports it as unverified.\n- **A BigQuery round trip proven in CI.** The executor's refusal logic, spend\n  cap and parameter binding are unit tested and the emitted SQL is golden\n  tested, but no test in this repository runs a query against real BigQuery.\n  DuckDB and Postgres both do, against the same fixture, on every run.\n- **The Terraform fixture** for the policy tag resolver: two service\n  accounts, a taxonomy and a tagged column, from `docs/06-validation.md`.\n  The decision logic is unit tested against fakes; the Data Catalog round\n  trip is not.\n- **`truegrain init` for Snowflake and DuckDB.** It reads BigQuery and\n  PostgreSQL. The shape it produces is warehouse-agnostic, so each remaining\n  one is a reader rather than a redesign.\n- **Importers for anything but dbt, LookML and Cube.** Those three are read.\n  Apache Ossie is building converters upstream in `converters/`; check there\n  before starting a fourth.\n- **psql's `\\d <table>` over the wire protocol.** It sends a join across\n  half a dozen catalogue tables with regex operators. `\\dt` works, and so\n  does reading `information_schema.columns` directly. A statement the\n  catalogue cannot answer is logged with the name the client gave itself,\n  so an operator can see what their tool sent.\n- **Cumulative and conversion metrics.** Window functions parse and are\n  refused by name.\n- **Semi-additive measures.** Not expressible in Ossie; refused, not\n  guessed. This is also why a balance cannot go in a rollup.\n- **Cancelling a query on another replica.** Jobs can be shared between\n  replicas with `-job-store-dsn-env`, so submitting on one and polling\n  another works. Cancelling still only stops the warehouse query on the\n  replica running it: elsewhere the job is marked cancelled and the caller\n  stops waiting, but the query runs to its own deadline. A\n  `context.CancelFunc` is a pointer in one process's memory, and no store\n  changes that. `GET /v1/health` says which store is configured and what it\n  means.\n- **Any write path.** There is no INSERT, no CREATE, no materialization.\n  Rollups are read, never built: whatever builds them is your own pipeline,\n  and the engine's only interest in it is the freshness column it writes.\n\n## Licence\n\nApache 2.0, matching Apache Ossie.\n"}