mcp server

bluefield

Local first web scraper that turns any URL into clean Markdown or JSON and signs every capture with an offline verifiable Ed25519 provenance manifest called BPM. Git like history and diff for web data

danielvhofmann/bluefieldcommunitylistedunreviewed

Use tool

Set up

Runs on your computer. Pick your agent and operating system.

This project does not publish an install method we can turn into a config. Follow the README instead.

README

synced Oct 23, 2026

Bluefield

Prove what your agent read. An open-source, local-first web scraper where every capture is a signed, offline-verifiable Ed25519 provenance manifest.

Bluefield turns any URL into clean Markdown or structured JSON, tracks how a page changes over time, and signs every capture with an offline-verifiable Ed25519 provenance manifest. Months later, with no network and no trust in us, anyone can prove exactly what a URL served and when.

Repository: https://github.com/danielvhofmann/bluefield

Try it in one line — no install, no signup, no key

$ npx @bluefields/cli scrape https://example.com

That captures the page into a content-addressed store on your own disk and signs it with a local key. No server, no account, no hosted database, no telemetry — grep the source. Install it for keeps with npm install -g @bluefields/cli (then bf --help), or build from source: git clone https://github.com/danielvhofmann/bluefield && cd bluefield && pnpm install && pnpm -r build.

Add to your agent (MCP) in 30 seconds

Give an assistant like Claude web access as native tools. The @bluefields/mcp-server exposes scrape, map, crawl, search, extract, and an offline verify — so your agent can not just read the web, but prove what it read.

Add this to claude_desktop_config.json (on macOS: ~/Library/Application Support/Claude/claude_desktop_config.json). The same block works in Claude Desktop, Claude Code, Cursor, and VS Code:

{
  "mcpServers": {
    "bluefields": {
      "command": "npx",
      "args": ["-y", "@bluefields/mcp-server"],
      "env": {
        "BLUEFIELDS_API_KEY": "bf_your_key_here"
      }
    }
  }
}

The MCP server is a thin client to the hosted Bluefield API — a free, hard-capped reference demo, not a commercial product — so its data-plane tools need a BLUEFIELDS_API_KEY. The one exception is verify, which runs entirely offline against a pinned public key: no key, no network, no trust in the server. See packages/mcp/README.md for the tool list and configuration.

Tamper detection, in one demo

The whole point is that a signed capture is fail-closed: flip a single byte of the stored page and bf verify catches it — offline, with a non-zero exit.

The animated cast was recorded before bf verify gained its capture method line (2026-07-26); the transcript below is a verbatim run of the current build.

$ bash examples/tamper-demo.sh
· throwaway store: /var/folders/…/T/bf-tamper-demo.aVQcKP   (your ~/.bluefield is untouched)

1) Capture https://example.com/
   → this is the ONLY step that needs network; everything below is offline.
captured https://example.com/
  snapshot 2a909528-8a32-430b-8780-aba17348b55b
  sha256   ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d
  signed   yes    changed: yes

2) Verify the untouched capture — expect ✓ VERIFIED (content hash + signature)
✓ VERIFIED  2a909528
  content hash matches : yes
  signature valid      : yes
  keyId                : ur6Z7L3GVCzaRX2EIVxG15lCIwHgkm5KlPRiOY8L6io
  capture method       : http-v1 — no JavaScript executed; 559 B

3) Tamper: flip ONE byte of the stored page on disk
   object: objects/ff/ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d
   byte 0: 60 → 28   (file length unchanged — a silent edit)

4) Verify again — expect ✗ FAILED with a non-zero exit (fail-closed)
✗ FAILED  2a909528
  content hash matches : NO
  signature valid      : yes
  keyId                : ur6Z7L3GVCzaRX2EIVxG15lCIwHgkm5KlPRiOY8L6io
  capture method       : http-v1 — no JavaScript executed; 559 B
  content hash mismatch;

✓ Tamper detected: one flipped byte → 'content hash mismatch' → verify exited non-zero.
  The signed hash proved the bytes changed, entirely offline. That is the point.

The same proof, site-scale

bf crawl captures a whole site — every page through the same signed path. So the proof scales: crawl a site, flip one byte of one stored page, re-verify everything, and verification pinpoints exactly which page was touched (examples/site-proof-demo.sh, a real run):

$ bf crawl https://bluefields-data.com --max-pages 6 --depth 2
crawl done — job 582fb1d0-…
  captured 6  blocked 0  skipped 0  failed 0

# …flip ONE byte of ONE stored page, then re-verify ALL six:
✓ PASS  77da7a4d  https://bluefields-data.com/about
✓ PASS  bc986259  https://bluefields-data.com/pricing
✓ PASS  adc7a39f  https://bluefields-data.com/integrations
✓ PASS  de0c2f76  https://bluefields-data.com/playground
✓ PASS  6b2d2caa  https://bluefields-data.com/products
✗ FAIL  550acd5d  https://bluefields-data.com/

✓ 6 pages, 6 signatures. One byte changed. The proof finds it.

Crawling is robots-respecting by design (no flag to disable it), paced per host, https-only, and resumable — Ctrl-C pauses, --resume continues.

Why Bluefield vs the alternatives

Bluefield is not the fastest or the most battle-tested scraper. It is the one that lets you prove what you captured. Here is the honest tradeoff:

	Bluefield	Firecrawl	Crawl4AI	Jina Reader
Open source	✓	✓	✓	partial
Signed, offline-verifiable provenance	✓	✗	✗	✗
Local-first (no account, runs on your disk)	✓	✗ (hosted)	✓	✗ (hosted)
Site crawl (every page a signed snapshot)	✓	✓ (unsigned)	✓ (unsigned)	✗
Sitemap map / URL discovery	✓	✓	✓	✗
MCP server	✓	✓	✓	✗
Watch + diff a page over time	✓	✗	✗	✗
Search endpoint	✗	✓	✗	✓
Coverage on hard / anti-bot sites	roughly a tie	mature, strong	fair	fair
Tail latency on the hardest targets	fair	best	fair	good
Price	free (self-host)	paid tiers	free	cheapest hosted
Community maturity	new	large	large	medium

Where the others genuinely win: Firecrawl has more mature coverage, better tail latency on the hardest targets, and a much larger community. Crawl4AI has a large community too. Jina is the cheapest hosted reader. If those are what you need, use them — we would rather be honest than lose your trust.

Where Bluefield wins: provenance you can verify offline, a local-first design that never phones home, a first-class MCP server, and built-in watch + diff. If "prove what your agent read" matters, this is the point of the project.

The Bluefield Provenance Manifest (BPM)

Every signed capture conforms to the Bluefield Provenance Manifest (BPM) — a small, permanent, offline-verifiable record: an Ed25519 signature over a JCS-canonical manifest binding the content hash, the URL, the fetch time, and the robots posture.

BPM is a proposed specification, not an adopted or ratified standard. The format, signature scheme, and verification procedure are written up in packages/attest/spec/SPEC.md, with conformance test vectors under packages/attest/spec/test-vectors/. Feedback and independent verifier implementations are welcome.

The CLI runs the same audited engine (fetcher / extractor / differ / attest) as the hosted Bluefield demo, wired to an embedded database (PGlite) and a filesystem object store — so nothing leaves your machine beyond fetching the page itself; verify, log, diff, and export are fully offline.

# Capture a page (needs network; https-only, blocks localhost/file:// by design)
$ bf scrape https://example.com/
# Prove it — recomputes the hash and checks the signature, fully offline
$ bf verify 3f9a2b
# Diff, history, and a portable proof bundle anyone can re-verify
$ bf diff https://example.com/
$ bf log https://example.com/
$ bf export 3f9a2b --out ./proof     # writes <id>.content + <id>.bundle.json

Packages

Package	What it is	License
`@bluefields/cli`	The `bf` command-line tool + local library	AGPL-3.0
`@bluefields/fetcher`	Tiered HTML capture (HTTP → headless browser), SSRF-guarded	AGPL-3.0
`@bluefields/extractor`	HTML/PDF → clean Markdown / structured JSON	AGPL-3.0
`@bluefields/differ`	Classifies and diffs successive captures	AGPL-3.0
`@bluefields/attest`	The BPM reference implementation + spec + verifier	Apache-2.0
`@bluefields/primitives`	Canonicalization, SSRF guard, robots.txt, env, logging	Apache-2.0
`@bluefields/db`	Drizzle schema definitions (table shapes only)	Apache-2.0
`@bluefields/sdk`	TypeScript client + offline attestation verifier	Apache-2.0
`bluefields` (Python)	Python client SDK	Apache-2.0
`@bluefields/mcp-server`	MCP server exposing scrape/verify tools to AI agents	Apache-2.0

License map

Bluefield is dual-licensed by layer:

Engine + headline tool — AGPL-3.0-or-later: fetcher, extractor, differ, cli. If you run a modified version as a network service, the AGPL requires you to offer your users its source.
Spec, verifier, SDKs, primitives, MCP, schema — Apache-2.0: attest (BPM spec + reference verifier), primitives, db, sdk-js, sdk-py, mcp. Permissively licensed so that verifying a Bluefield attestation, or building on the spec, carries no copyleft obligation.

Full texts: LICENSE-APACHE and LICENSE-AGPL. Every source file carries an SPDX header naming its license.

Self-hosting

Self-hosting Bluefield means running the bf CLI (or importing the library) with your own keys — no Bluefield account, no hosted API. See SELFHOST.md.

Honest positioning

We would rather you trust us because the claims are checkable than because the copy is loud.

What Bluefield is best at: provenance. Every result is a signed, offline-verifiable record — you can prove what your agent read, months later, with no network and no trust in us. If "prove what you scraped" matters, this is the point of the project.
Coverage and speed: plain HTTP fetching handles most static sites with zero vendor keys. JS-heavy and anti-bot ("fortress") sites need a proxy and an unlocker you bring, and some will still return a challenge. We do not claim to beat commercial scraping APIs on raw reachability or tail latency on the hardest targets — mature hosted services (Firecrawl and friends) invest heavily there, and on those axes they often win.
Extraction quality: good on articles, docs, and product pages; weaker on dense listing, forum, and some structured pages. It is an honest engine, not a magic one.
No telemetry. The CLI phones home to nothing. This is a design property, not a promise — the source is here to check.

Development

Requirements: Node 22+, pnpm 11+.

pnpm install         # install + generate the lockfile
pnpm -r build        # build every package to dist/
pnpm -r test         # run the full test suite
pnpm typecheck       # type-check every package
pnpm lint            # biome

See CONTRIBUTING.md.

The Python SDK (packages/sdk-py) is not part of the pnpm workspace; build and test it with your Python toolchain (pip install -e packages/sdk-py[dev] then pytest).