{"id":"237a8c46-05d0-4fb5-9b27-87b252032830","slug":"semantic-junkyard","kind":"mcp","name":"semantic-junkyard","summary":"Keeps knowledge in files, databases and Git while giving AI agents one verified way to read, propose and confirm changes.","owner":"aantenore","repoUrl":"https://github.com/aantenore/semantic-junkyard","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-07-09T10:30:29.000Z","addedAt":"2026-10-21T06:23:05.141Z","logoUrl":"https://avatars.githubusercontent.com/u/50747458?u=85f5d95e018eafce83450d3f740369a921255d6b&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["data-analytics","memory-knowledge","developer-tools"],"tags":["agentic-ai","change-control","data-governance","data-lineage","local-first","typescript","semantic-federation"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-22T14:43:03.624Z","sample":false,"url":"https://conduits.dev/tools/semantic-junkyard","language":"TypeScript","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Semantic Junkyard\n\n## What this changes in the real world\n\nImportant business knowledge is often scattered across documents, databases, and repositories. An\nAI agent may find a plausible answer but lose where it came from, or update the wrong record without\nchecking the result. **Semantic Junkyard builds one searchable map across those sources while\nkeeping the originals in charge, then makes every supported change pass through a reviewable plan\nand an authoritative readback.**\n\n### A concrete example\n\nAn operations user asks an agent to mark order `ORD-1001` as dispatched. The system identifies the\nexact configured database row, records its current version, checks policy, performs the allowlisted\nupdate, and reads the row again. Only when the source itself says `dispatched` does the semantic map\npublish fresh evidence. If the row changed in the meantime, execution stops instead of overwriting\nsomeone else's work.\n\nSemantic Junkyard is for data, platform, and operations teams that want agents to work across\nexisting knowledge sources without turning a model or a new catalog into the source of truth.\n\n| Feature | Practical benefit |\n| --- | --- |\n| Federated map across files, SQLite, and Git | People and agents can discover related knowledge without moving ownership away from existing systems. |\n| Source-linked evidence and provenance | An answer can show where a fact came from and which version was observed. |\n| Reviewable semantic proposals | Model suggestions remain suggestions until deterministic checks or a person accepts them. |\n| Exact change plans with policy and optional approval | A reviewer can see the target, expected change, and authorization context before a write. |\n| Authoritative post-write readback | Success means the requested state was observed at the real source, not merely that a connector returned `200 OK`. |\n| Read-only MCP defaults | An external agent starts with bounded discovery rather than silent write authority. |\n\n> **Maturity:** This is a local-first reference product, not a production multi-tenant platform.\n> It currently implements local filesystem, SQLite, and Git connectors with a single-node SQLite\n> control plane. Production IAM, tenant isolation, source-ACL propagation, high availability, and\n> managed cloud connectors remain outside the current implementation.\n\n## Technical scope\n\nSemantic Junkyard is a local-first reference implementation of an **agent-safe semantic federation and verified semantic-action protocol**.\n\nIt is not another data catalog and it does not become the authority for connected data. It observes configured sources, preserves evidence and source identity, separates authoritative facts from reviewable semantic proposals, exposes bounded context to agents, and permits a change only through an exact capability contract:\n\n```text\nbusiness intent\n  -> typed source target and exact diff\n  -> source-version preconditions\n  -> policy and optional human approval\n  -> idempotent write\n  -> authoritative source reread\n  -> postcondition\n  -> semantic refresh only after verification\n```\n\nThe repository proves this contract against real local filesystem, SQLite, and Git sources. It is deliberately not presented as a production-ready multi-tenant platform.\n\n## Understand It In One Minute\n\nSemantic Junkyard builds a **derived map of existing data** for people and AI agents. The original files, databases, and repositories remain the systems of record.\n\n```mermaid\nflowchart LR\n  subgraph Clients[\"People and agents\"]\n    Product[\"Product workbench\"]\n    Poc[\"Conversational PoC\"]\n    Agent[\"External MCP client\"]\n  end\n\n  subgraph Plane[\"Semantic Junkyard control plane\"]\n    Api[\"REST API\"]\n    Mcp[\"MCP server\"]\n    Engine[\"HTTP runtime\"]\n    McpEngine[\"Independent MCP runtime\\nread-only tools by default\"]\n    Model[\"Optional local HF model\\ntyped suggestions only\"]\n    Control[(\"Control-plane SQLite\\nsemantic map + audit + plans\")]\n    Connectors[\"Typed source connectors\"]\n  end\n\n  subgraph Sources[\"Authoritative sources\"]\n    Files[\"Filesystem\\nread only\"]\n    Sqlite[\"SQLite\\nallowlisted row updates\"]\n    Git[\"Git\\nallowlisted contract commits\"]\n  end\n\n  Product --> Api\n  Poc --> Api\n  Agent --> Mcp\n  Api --> Engine\n  Mcp --> McpEngine\n  Engine <--> Control\n  McpEngine <--> Control\n  Model -. \"validated typed candidates\" .-> Engine\n  Model -. \"when selected for enabled sync\" .-> McpEngine\n  Engine --> Connectors\n  McpEngine -. \"source access; mutations require flags\" .-> Connectors\n  Connectors -->|\"discover and reread\"| Files\n  Connectors <-->|\"discover, bounded update, reread\"| Sqlite\n  Connectors <-->|\"discover, bounded commit, reread\"| Git\n```\n\nFour rules explain the system:\n\n1. **The semantic layer is a map, not the source of truth.** Every useful statement keeps its source and evidence identity.\n2. **Models may suggest; deterministic controls decide.** Model output can become a reviewable proposal or a typed intent, never an approval or an unrestricted write.\n3. **A plan belongs to one authorization context.** Actor, roles, clearance, and policy version are persisted in the plan identity; another principal must create a new plan.\n4. **A write is complete only after authoritative readback.** A connector success response is insufficient; the expected postcondition must be observed from the source and reflected into fresh semantic evidence.\n\nStart with [How Semantic Junkyard works](docs/how-it-works.md), then use the [Hands-on guide](docs/user-guide.md) to run the real read-only, autonomous SQLite, and approval-gated Git workflows. The [documentation index](docs/README.md) provides paths for operators, agent builders, connector authors, and reviewers. Security reviewers should begin with the concrete [threat model](docs/threat-model.md).\n\n## What The Product Is\n\nSemantic Junkyard combines two responsibilities that are usually split across several systems:\n\n1. **Semantic federation for agents.** Discover physical and semantic resources across configured sources, materialize provenance-linked evidence, expose lexical/vector/graph retrieval, and retain source authority.\n2. **Verified change control.** Resolve a business request to one configured write capability, bind its exact target and preconditions into a fingerprint, enforce policy and approval, execute idempotently, reread the authoritative source, and publish reflection evidence only when the postcondition passes.\n\nThe control plane stores observations, proposals, evidence, decisions, plans, approvals, runs, and audit events. The filesystem, operational SQLite database, and Git repository remain the authoritative sources.\n\nThe defensible product wedge is the second responsibility: a reusable, connector-conformance protocol for proving that an agent-requested business change reached the correct authoritative source and came back as governed read evidence. Search, catalogs, graphs, and MCP access are integration surfaces, not differentiation claims. The [market scan](docs/market-scan.md) explains why production deployments should integrate established context platforms instead of rebuilding them.\n\nSource-local catalog identifiers are namespaced by connection before entering the federated read model. The original identifier remains provenance metadata, so equal IDs from different domains cannot overwrite one another and deleting a connection removes only its owned assets, metrics, policies, lineage, contracts, and ontology classes.\n\nSee [Product definition](docs/product-definition.md) for scope and non-goals and [Reference workflow](docs/reference-workflow.md) for an end-to-end acceptance path.\n\n## Local Reference Product\n\nA persistent development start creates a supply-chain reference environment when no source connections exist:\n\n| Source | Discovery | Governed write behavior |\n| --- | --- | --- |\n| `Supply Chain Knowledge` | Real local files: Markdown policy, CSV reference data, OpenLineage JSON, and supported documents. | Read-only. The filesystem connector has no write method. |\n| `Operations Database` | Real SQLite schema, columns, primary/foreign keys, row counts, optional samples, sensitivity signals, and evidence. | One allowlisted `orders.status` update selected by key. A source-row hash is the optimistic precondition; readback uses a new read-only connection and exact field equality. |\n| `Semantic Contract Repository` | Real committed Git tree and YAML semantic contracts with commit/blob provenance. | Only configured semantic-contract paths. Approval is required; HEAD and blob hashes are preconditions; the connector commits only the planned path and verifies committed content and fields with `git show`. |\n\nThe filesystem connector supports `txt`, `md`, `html`, `json`, `jsonl`, `csv`, `yaml`, `yml`, and `pdf`, with file-count/size bounds and symlink rejection. `metadata_only` and `external_reference` ingestion retain no submitted payload text in the control-plane source record; they index a registration note instead.\n\nThere is no generic SQL executor, shell tool, arbitrary file writer, or write path for an unknown source. A write is possible only when a compiled connector resolves exactly one target that is exposed by explicit configuration.\n\n## Semantic Proposal Lifecycle\n\nSource synchronization distinguishes observation from interpretation:\n\n- **Authoritative source facts** such as SQLite columns, foreign keys, and declared contract-to-metric relations are accepted automatically, tagged `source_fact`, and cannot be rejected in the semantic layer. They must be changed at the source or in its authority mapping.\n- **Connector deterministic inferences** and **local-model candidates** are stored as `proposed` assertions with confidence, explanation, origin, resource IDs, and evidence chunk IDs.\n- Operators accept or reject non-authoritative proposals with a rationale in the product UI or REST API.\n- A later source sync marks assertions that are no longer emitted as `superseded` and removes them from active navigation.\n\nAcceptance never makes an inference authoritative. Pending connector proposals are visible to operators with an explicit lifecycle, but agent graph tools and graph retrieval exclude them until acceptance. Rejected and superseded relations are inactive. Direct-ingest relations are marked `proposed` and excluded from agent graph reasoning, but they do not yet have first-class proposal records; manual curation requires a rationale and creates an accepted, non-authoritative relation. See [How semantic meaning is governed](docs/how-it-works.md#5-how-semantic-meaning-is-governed).\n\nThe optional Hugging Face path can suggest concepts, classifications, relations, and conflicts from a bounded list of observed resources. Strict schemas discard malformed output, invented resource IDs, self-relations, duplicates, and over-limit candidates. A model proposal never becomes an authoritative source fact.\n\n## Product And Agent Surfaces\n\n| Surface | Default | Responsibility |\n| --- | --- | --- |\n| Product workbench (`apps/web`) | `http://localhost:5173` | Operator-facing source registry, connection test/sync, proposal review, retrieval/graph inspection, exact plan review, approval, execution, readback, and audit. |\n| External conversational PoC (`apps/poc`) | `http://localhost:5174` | Independent REST client that starts deterministic and read-only, returns an answer/claim/citation contract, and exposes opt-in plan-only and autonomous workflows. It stops for missing evidence, no writable source, policy blocks, or required approval. |\n| API (`apps/api`) | `http://127.0.0.1:8787` | HTTP control plane and generated reference OpenAPI document. |\n| MCP server (`apps/mcp`) | stdio | External agent surface over the same engine contracts. It opens the selected control-plane SQLite database directly and intentionally cannot create approvals or decide proposals. |\n\nThe two React applications do not share frontend state. The browser PoC is not an MCP client; it calls the product API through its own client module. `npm run poc:agent:mcp` is the separate real MCP client/server proof of concept.\n\n## Deterministic And Local-HF Enrichment\n\nThe default semantic path is deterministic and offline:\n\n- local parsing and stable chunking;\n- extractive summaries and pattern-based entities/relations/claims;\n- 128-dimensional signed hash embeddings;\n- SQLite FTS5 plus deterministic vector and graph score fusion;\n- deterministic source profiling, policy checks, target resolution, and postcondition evaluation.\n\nLocal Hugging Face generation through MLX is optional and has three bounded roles: source-enrichment proposals, conversational intent interpretation, and a bundled trace summary. It does not approve actions, bypass connector rules, choose arbitrary targets, or decide whether a postcondition passed. Prompts explicitly request typed output without chain-of-thought; the product records evidence, tool observations, concise explanations, and decisions, not hidden reasoning traces.\n\n`ollama` and `openai-compatible` values currently affect provider configuration reporting only. They are not injected into discovery, extraction, embeddings, planning, or policy. See [Local models](docs/local-models.md).\n\n## Quick Start\n\nPrerequisites: Node.js 20.19 or later, npm, SQLite support supplied by the lockfile, and Git for the reference contract repository.\n\n```bash\nnpm ci\nnpm run dev\n```\n\n### Install the alpha from source\n\nThe `v0.1.0-alpha.1` release is a locked source distribution, not an npm package. A clean install\nuses only the repository, the committed lockfile, Node.js 20.19 or later, npm, and Git:\n\n```bash\ngit clone https://github.com/aantenore/semantic-junkyard.git\ncd semantic-junkyard\ngit checkout v0.1.0-alpha.1\nnpm ci\nnpm run check\nnpm run benchmark:reference\nnode scripts/smoke-release.mjs\n```\n\n`npm run check` validates documentation, types, tests, and every production build. The reference\nbenchmark then gates its fixed synthetic fixture at 7/7 relevant items retrieved in the top five,\n3/3 exact action targets, 3/3 unsupported-intent abstentions, 0/6 false verified outcomes, one\nidempotent replay, and one stale-precondition rejection. These numbers are a deterministic regression\nsignal, not evidence of general retrieval quality, production safety, or comparative model\nperformance. The final command starts the built artifacts in temporary storage, checks API health,\nreadiness, and OpenAPI, then performs an MCP handshake and confirms the default tool set is read-only.\nCI repeats that release smoke on Linux and Windows and runs the browser path separately on Node.js\n22. See the [evaluation guide](docs/evaluation.md#deterministic-reference-benchmark),\n[changelog](CHANGELOG.md), and [threat model](docs/threat-model.md) before using the reference outside\na local environment.\n\n`npm run dev` builds the shared contracts and starts the API, product workbench, and conversational PoC. The persistent API start reconciles the three deterministic reference connections and retries connections that are `configured`, `syncing`, `error`, missing a last sync, or missing published resources. Set `SEMANTIC_JUNKYARD_BOOTSTRAP_REFERENCE_SOURCES=false` to disable that bootstrap.\n\nFocused commands:\n\n```bash\nnpm run dev:product       # API + operator workbench\nnpm run dev:poc           # API + external REST PoC\nnpm run build\nnpm run mcp               # built MCP stdio server\nnpm run poc:agent:mcp     # real MCP client/server reference run\n```\n\nThe API reads environment variables from its process. It does not automatically load the root `.env` file. The complete template is [.env.example](.env.example).\n\nFrom the product dashboard, **Run mission** synchronizes every selected/configured source and profiles the resulting fabric as one durable report. The equivalent operator API is `POST /api/discovery/missions`; MCP exposes `discover_sources` only when started with `--allow-sync`.\n\n## Reference Actions\n\nThe seeded SQLite action is autonomous and low-risk:\n\n```text\nSet order ORD-1001 status to dispatched\n```\n\nThe seeded Git action resolves to a configured YAML contract and requires approval:\n\n```text\nUse dispatch eligible orders as the denominator for Late Dispatch Rate and publish version 2\n```\n\nFor both paths, planning is read-only. Execution recomputes the plan against current source state; a changed target, row hash, Git HEAD, blob, diff, risk decision, or warning changes the fingerprint or fails a connector precondition. Only an exact reread that satisfies the connector postcondition can produce `verified` reflection evidence.\n\nIf the SQLite row already has the requested allowlisted value, the plan is marked as a no-op and execution performs only the precondition and authoritative reread; it does not issue a redundant `UPDATE`.\n\n## Principal HTTP Contracts\n\nThe generated OpenAPI document is at `GET /api/openapi.json`. Liveness is exposed at `GET /api/health`; `GET /api/ready` returns `503` while reference bootstrap is initializing or degraded. Principal route groups are:\n\n- Source federation: `GET/POST /api/source-connections`, connection `test` and `sync`, `GET /api/source-resources`, and `GET /api/source-sync-runs`.\n- Proposal governance: `GET /api/semantic/proposals` and `POST /api/semantic/proposals/:proposalId/decision`.\n- Retrieval: scoped `semantic_search`/`expand_context` (`domain` by default, `operational` for receipts), `source_resource_search`, `entity_lookup`, `graph_neighbors`, `find_paths`, and direct evidence reads.\n- Change control: `POST /api/business/actions/plan`, `/approve`, and `/execute`; run and approval listings are separate.\n- Agent integration: intent interpretation, manifest, MCP descriptors, discovery runs, audit events, and the bundled local PoC route.\n\nRequest bodies are strict Zod contracts. Unknown keys, including a caller-supplied `approved` flag, are rejected.\nHTML ingestion also fails closed when configurable input-length, tree-depth, or child-node limits are exceeded; the safe defaults are documented in `.env.example`.\n\nEvery business-action plan is persisted before it can be approved or executed. Its identity includes the authenticated actor, normalized roles, clearance, and policy version. Execution by a different principal returns `PLAN_PRINCIPAL_MISMATCH`; a missing or stale plan cannot be reconstructed from caller input alone.\n\n## MCP Contract\n\nBuild before starting the stdio server:\n\n```bash\nnpm run build\nnpm run mcp\n```\n\nExample client configuration:\n\n```json\n{\n  \"mcpServers\": {\n    \"semantic-junkyard\": {\n      \"command\": \"node\",\n      \"args\": [\"/absolute/path/to/semantic-junkyard/apps/mcp/dist/server.js\"]\n    }\n  }\n}\n```\n\nThe MCP server is read-only by default. It exposes bounded tools for permission explanation, resource/semantic search, entity and graph navigation, context/evidence retrieval, proposal listing, and action planning. Persisted discovery, configured-source synchronization, and business execution are registered only with `--allow-discovery`, `--allow-sync`, and `--allow-write` respectively. It exposes no tool for connection creation, proposal decisions, or approval creation. An `approvalId` may be consumed only if a human-facing API channel created it for the exact plan.\n\nUse `--db <relative-path>` or `SEMANTIC_JUNKYARD_DB=<relative-path>` to select a control-plane database inside the product-owned `apps/api/data` root. The default launchers do not accept an alternate root from the environment or command line; absolute, drive-qualified, traversal, file-URI, reserved-device, symbolic-link, multiply linked, and orphaned rollback/WAL sidecar layouts fail closed. Embedding hosts that need another location must call `openControlPlaneDatabase({ authorizedRoot, databasePath })` with an absolute, pre-existing real root and a relative database path. Use `--memory` for an in-memory seeded runtime and `--no-seed` to disable that memory seed. Grant mutation flags independently and only to a trusted client. The bundled MCP PoC explicitly starts its private server with `--allow-write`. MCP does not inherit REST authentication or CORS. The spawned process has its operating-system filesystem authority, including access to local source paths stored in the selected database.\n\nSee [Agent contract](docs/agent-contract.md).\n\n## Trust Boundaries\n\n- Connected content is untrusted data. It cannot redefine tool policy or connector configuration.\n- The local source is authoritative for source facts and postconditions; the control-plane read model is derived.\n- Capability templates describe possible integrations but are not executable. Without a managed connector, a target is blocked and cannot claim source verification.\n- Non-authoritative semantic assertions require review and remain distinguishable by lifecycle and origin.\n- The browser API boundary and the MCP process boundary are different. REST tokens do not constrain a local MCP process.\n- The default tokenless loopback profile grants a development-only local owner role. Authenticated mode requires distinct agent, operator, and approver bearer tokens, but these static tokens are not production IAM.\n- Local idempotency is enforced in the control-plane SQLite database. It is not a distributed transaction or durable exactly-once guarantee across crashes.\n\n## Current Limitations\n\n- Single-node SQLite is the only control-plane store. There is no production clustering, tenancy, migration service, backup orchestration, or high-availability design.\n- Only local filesystem, SQLite, and Git connectors are implemented. There are no production DataHub, OpenMetadata, cloud object-store, warehouse, ticketing, or remote Git-provider connectors.\n- There is no production IAM, tenant isolation, source-ACL propagation, approval delegation/expiry/revocation workflow, or secrets manager.\n- Synchronization and actions run in process. Per-connection SQLite leases prevent overlapping syncs across runtime instances, and deterministic observation replacement is transactional, but there is no durable job queue, scheduler, retry service, outbox, or crash-recovery reconciler.\n- Control-plane transactions and source-native writes are not one distributed transaction. In-process ambiguous outcomes are persisted as `reconciliation_required` and cannot reuse a consumed approval. A process crash after a source commit but before control-plane persistence can still leave no run record; there is no reconciliation worker.\n- Arbitrary unknown-source writes are intentionally unsupported. SQLite updates require an allowlisted table, key, and columns; Git writes require an allowlisted semantic-contract path; filesystem is read-only.\n- Deterministic extraction, entity resolution, intent parsing, and hash embeddings are reference-quality implementations, not production semantic quality.\n- Local-HF execution is Apple Silicon/MLX-oriented, depends on a pre-cached compatible model and runtime packages, and has no model-faithfulness release gate.\n- The product does not request or persist hidden chain-of-thought. Auditability is based on observable evidence, typed artifacts, policy decisions, tool events, diffs, and source readback.\n- Static catalog/metadata/ticketing capability templates remain as configuration examples. They are blocked until a managed adapter exists and must not be described as external integrations.\n\nSee [Architecture](docs/architecture.md) for the full boundary analysis.\n\n## Verification\n\n```bash\nnpm run typecheck\nnpm test\nnpm run build\nnpm run test:e2e\n```\n\n`npm run check` runs type checking, Vitest, and the production build; Playwright remains a separate command. See [Evaluation](docs/evaluation.md) and the acceptance checklist in [Reference workflow](docs/reference-workflow.md).\n\n## Repository Layout\n\n```text\napps/api           HTTP control plane, semantic runtime, source connectors, local model harness\napps/mcp           MCP stdio server and external MCP PoC client\napps/poc           independent conversational REST client\napps/web           operator product workbench\npackages/shared    strict shared Zod contracts\ndocs               product, architecture, contracts, workflow, evaluation, and market context\n```\n\n## Documentation\n\n- [Documentation index](docs/README.md)\n- [How Semantic Junkyard works](docs/how-it-works.md)\n- [Hands-on guide](docs/user-guide.md)\n- [Product definition](docs/product-definition.md)\n- [Reference workflow](docs/reference-workflow.md)\n- [Architecture](docs/architecture.md)\n- [Agent contract](docs/agent-contract.md)\n- [Adapter contracts](docs/adapter-contracts.md)\n- [Local models](docs/local-models.md)\n- [Evaluation](docs/evaluation.md)\n- [Market scan](docs/market-scan.md)\n- [Contributing](CONTRIBUTING.md)\n- [Security](SECURITY.md)\n"}