cyberfrog

Features

• Overview first: after a crawl you land on an SEO health score (0–100), the key numbers and the issues to fix first; every number opens its report.
• Quick search (⌘K): jump to any report, issue or action by typing.
• Sidebar of reports grouped by Overview, Pages, AI & GEO, Content, Links and Resources, each with a badge showing how many issue types it has.
• Issues report: every problem found, ranked High / Medium / Low, with the number of URLs affected and how to fix it. Click an issue to jump to the matching report, already filtered.
• Crawling: same-host breadth-first crawl, robots.txt (obeyed or ignored), redirects recorded, configurable speed, URL limit, user agent (CyberFrog, Googlebot, Bingbot or custom) and exclude patterns. Live progress, with a Stop button.
• On-page: titles, meta descriptions and H1 (missing, duplicate, too long, too short), word count, canonical, meta robots and X-Robots-Tag.
• Indexability: noindex/nofollow, canonicalised pages, canonicals pointing to non-200 URLs.
• Links: inlinks and outlinks with anchor text for every URL, external link checking, and broken links listed with the page they're on.
• Images: missing or empty alt text, missing width/height, file size (flagged above 100 KB).
• XML sitemaps: reads sitemap indexes from robots.txt, crawls sitemap-only URLs, finds orphan pages and indexable pages missing from the sitemap. It can also generate a sitemap.xml.
• Performance: response time and page size, median response time.
• Hreflang: invalid codes, missing return links, non-200 alternates, missing self-reference and x-default.
• Structured data: JSON-LD (including @graph) and microdata types, invalid JSON, missing properties Google requires for common rich results.
• Duplicate content: exact duplicates and near duplicates (SimHash, ≥ 90% similar) on the main content, plus low-content pages.
• Crawl comparison: compares each crawl with the previous one of the same site — new and removed pages, status and indexability changes, changed titles, meta descriptions, H1 and canonicals, and how each issue moved.
• Custom extraction and search: pull any value from every page with a CSS selector (text or attribute) or a regex — authors, prices, GA4 IDs — or check which pages contain or lack a piece of text or code.
• Redirect chains: every redirect followed to its final URL, with chains, loops and non-200 destinations.
• Uncrawlable links: links search engines may not follow (JavaScript navigation, href outside <a>) and invalid URLs.
• Link Score and site structure: internal PageRank on a 0–100 scale for every URL, URLs per click depth, and the folder tree with URL counts, indexable pages and the strongest Link Score in each folder.
• JavaScript rendering (optional): pages are also opened in a headless Chrome, Chromium, Brave or Edge already on your Mac. The JavaScript report shows content and links that exist only after rendering, and pages that failed to render.
• AI assistant: suggests titles and meta descriptions, explains a page's issues and writes prioritised action plans. It runs through your own Claude Code or Codex (your Claude or ChatGPT plan, no API cost), or an Anthropic API key kept in the macOS Keychain. Settings show whether each engine is connected and how to connect it. Page data is sent only when you ask.
• AI visibility tracker: for the questions your customers ask, an AI with web search answers, and CyberFrog records whether it cites your site, in which position, and which sites it cites instead, with history over time.
• Google Search Console: sign in with Google (read-only) or use a service account, then see clicks, impressions, CTR and position for every page next to the crawl: low-CTR pages, quick wins close to page one, pages Google sees that the crawl didn't find, and queries where two of your pages compete. An AI can rewrite titles and metas for the low-CTR ones.
• Competitor analysis: crawl a competitor with your settings and compare 13 measures side by side, see the topics they cover and you don't (and the reverse), and get the steps to overtake them, plus an AI plan.
• Generators: an llms.txt built from your topics and pages, and FAQPage structured data from a page's question headings.
• Client report: one self-contained HTML report, branded and ready to print to PDF.
• Built-in help: a "?" on every report explains what it shows, how to read it and what to do.
• List mode: paste a list of URLs (a migration plan, a spreadsheet column) and CyberFrog checks only those, on any site, plus where they redirect.
• Scheduled crawls: every day or every week at a set time, even with the app closed. A macOS notification tells you what changed since the previous run (new and removed pages, status changes, new errors).
• GEO – AI visibility: how ready the site is for AI answer engines (ChatGPT, Claude, Perplexity, AI Overviews). Screenshot
  • which of 12 AI crawlers robots.txt lets in, split into search, on-demand and training bots
  • llms.txt presence and format, and Content-Signal preferences
  • a 0–100 citability score per page: answer up front, question subheadings, lists and tables, facts, author, date, sources, structured data
  • pages AI crawlers can't read because their content is built by JavaScript
• Topics & links: local TF-IDF analysis of titles, H1s and text:
  • topic clusters built around their strongest page (the hub)
  • pages that cannibalize each other's searches, with the words they share
  • the internal links to add, with suggested anchor text
• MCP server: Claude and other assistants can run crawls and query the data (see below).
• Export and reuse: CSV export of any view (Excel-friendly), crawls saved as single SQLite files you can save, reopen and query.
• English and Italian interface.

Install

macOS (Apple silicon): download the .dmg from the Releases page and drag CyberFrog to Applications.

First launch on macOS

CyberFrog isn't notarised by Apple yet, so the first time macOS blocks it with "Apple could not verify CyberFrog is free of malware". To allow it (once):

1. Open CyberFrog from Applications. When the warning appears, click Done.
2. Open System Settings → Privacy & Security, scroll to the Security section and click Open Anyway next to the CyberFrog message.
3. Confirm with Open Anyway and your password or Touch ID.

On macOS 14 (Sonoma) and earlier you can instead right-click the app and choose Open.

Prefer the Terminal? This removes the download quarantine flag:

xattr -dr com.apple.quarantine /Applications/CyberFrog.app

Use with AI assistants (MCP)

CyberFrog includes a Model Context Protocol server, so assistants like Claude can crawl sites and analyse the results for you. Ask things like "Crawl example.com and tell me what to fix first" or "Which pages have duplicate titles?".

Claude Code:

claude mcp add cyberfrog -- /Applications/CyberFrog.app/Contents/MacOS/cyberfrog --mcp

Claude Desktop: add this to claude_desktop_config.json (Settings → Developer → Edit Config):

{
  "mcpServers": {
    "cyberfrog": {
      "command": "/Applications/CyberFrog.app/Contents/MacOS/cyberfrog",
      "args": ["--mcp"]
    }
  }
}

Tool	What it does
crawl_site	Crawls a site (default limit 500 URLs) and returns an SEO summary; optional custom extraction/search rules, JavaScript rendering (render_js) and list mode (urls)
list_crawls	Lists saved crawls, newest first
crawl_summary	Status codes, indexability, title/meta/H1 problems, broken links, slow pages, images, plus GEO and topics counts
query_crawl	Runs a read-only SQL SELECT on a crawl (up to 200 rows)
geo_report	AI crawler access per bot, llms.txt check, Content-Signal lines and per-page citability signals (up to 200 pages)
topics_report	Topic clusters, cannibalization pairs and internal link suggestions (up to 200 each)
content_gap	Compares two crawls (mine, theirs): topics the competitor covers that you don't, your strengths, and where their page goes deeper
page_details	Metadata, inlinks and outlinks of one URL

Crawls made through MCP are saved alongside the app's crawls, so you can open them in CyberFrog too.

Build from source

Requirements: Node.js 20+, Rust 1.88+, Xcode Command Line Tools.

npm install
npm run tauri dev          # run in development
npx tauri build            # build CyberFrog.app and the .dmg

Crawl from the terminal without the UI:

cd src-tauri
cargo run --example crawl -- https://example.com crawl.sqlite

Tests:

cd src-tauri && cargo test     # crawler, reports, settings
node scripts/pages.check.ts    # filters, issues and exports

How it works

Part	Technology
Desktop shell	Tauri 2
Crawler	Rust: tokio, reqwest, scraper, texting_robots; optional headless Chromium for JavaScript
AI assistant	Claude Code (claude -p) or Codex (codex exec) CLIs, isolated from tools and the user's config, or the Claude Messages API with a key in the macOS Keychain
Scheduled crawls	launchd agents running cyberfrog --scheduled <id>
Storage	one SQLite file per crawl
Interface	React + TypeScript, TanStack Table and Virtual

The crawler writes every URL, link, image and structured-data item to SQLite while it runs. The reports read from that file, so a crawl can be closed, reopened or shared later.

Be a good citizen

Crawl sites you own or have permission to audit. Keep the default speed unless you know the server can take more, and leave "Respect robots.txt" on.

Not affiliated with Screaming Frog

CyberFrog is an independent, clean-room project inspired by desktop SEO crawlers such as Screaming Frog SEO Spider. It contains no code, assets or data from Screaming Frog, and was written only from public documentation and observable behaviour. "Screaming Frog" and "SEO Spider" are trademarks of their respective owner.

License

Apache License 2.0. If you redistribute CyberFrog or a modified version, keep the NOTICE file with the attribution to the author and state the changes you made.

---