{"id":"3390ec2b-4a4a-4f22-a5bb-3cee5ba38a4c","slug":"arcfetch","kind":"skill","name":"arcfetch","summary":"URL fetching and article extraction tool that converts web pages to clean markdown","owner":"briansunter","repoUrl":"https://github.com/briansunter/arcfetch","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"skill","createdAt":"2025-12-25T06:23:56.000Z","addedAt":"2026-10-21T16:31:57.459Z","logoUrl":"https://avatars.githubusercontent.com/u/2816289?u=b6e37628662434cd4b8ede727bd07f37950be6ea&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["files-documents"],"tags":[],"runtime":"node","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-25T20:59:18.564Z","sample":false,"url":"https://conduits.dev/tools/arcfetch","language":"TypeScript","transports":["stdio"],"install":{"npm":"arcfetch"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"<p align=\"center\">\n  <img src=\"docs/banner.png\" alt=\"ArcFetch: fetch web pages, extract articles, get clean markdown\" width=\"320\">\n</p>\n\n# arcfetch\n\n[![npm version](https://badge.fury.io/js/arcfetch.svg)](https://www.npmjs.org/package/arcfetch)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Zero-config URL fetching** that converts web pages to clean markdown with automatic JavaScript rendering fallback.\n\nPerfect for AI workflows, research, and documentation. Fetches URLs, extracts article content using Mozilla Readability, and caches as markdown with 90-95% token reduction.\n\n## Why arcfetch?\n\n| Problem | Solution |\n|---------|----------|\n| **JS-heavy sites return blank** | Auto-detects and retries with Playwright |\n| **Login walls / error pages scored as content** | Boilerplate detection (22 patterns) catches them |\n| **Too much HTML clutter** | Mozilla Readability extracts just the article |\n| **High token costs for LLMs** | 90-95% token reduction vs raw HTML |\n| **No good caching story** | Temp → Docs workflow for easy curation |\n| **Hard to integrate** | Works as CLI or MCP server with zero setup |\n\n## Features\n\n- **Smart Fetching**: Simple HTTP first, automatic Playwright fallback for JS-heavy sites\n- **Quality Gates**: Scoring (0-100) with boilerplate, login wall, paywall, and error page detection\n- **Content-to-Source Ratio**: Catches JS-rendered or gated content by comparing extracted text to source HTML size\n- **Anti-Bot Detection**: Stealth plugin, viewport/timezone/locale rotation, realistic headers, navigator overrides\n- **Clean Markdown**: Mozilla Readability + Turndown for 90-95% token reduction\n- **Temp → Docs Workflow**: Cache to temp folder, promote to docs when ready\n- **Link Extraction**: Extract and batch-fetch all links from a cached reference\n- **CLI & MCP**: Available as command-line tool and MCP server (6 tools)\n- **Multiple Output Formats**: Plain text, JSON, filepath, or summary\n\n## Quick Start\n\n### No Installation Required\n\n```bash\n# Fetch and display markdown\nbunx arcfetch fetch https://example.com/article\n\n# Get just the filepath (for scripts)\nbunx arcfetch fetch https://example.com -o path\n\n# With pretty output\nbunx arcfetch fetch https://example.com --pretty\n```\n\narcfetch is a Bun-based executable. `npx arcfetch` also works when `bun` is installed on `PATH`.\n\n### Global Installation\n\n```bash\nbun add --global arcfetch\n\n# Then use directly\narcfetch fetch https://example.com/article\n```\n\n### Development\n\n```bash\ngit clone https://github.com/briansunter/arcfetch.git\ncd arcfetch\nbun install\nbun run cli.ts fetch https://example.com\n```\n\n## CLI Usage\n\n### Commands\n\n```bash\n# Fetch a URL and save to temp folder\narcfetch fetch https://example.com/article\n\n# List all cached references\narcfetch list\n\n# Extract links from a cached reference\narcfetch links my-article\n\n# Fetch all links from a cached reference (parallel)\narcfetch fetch-links my-article\n\n# Promote from temp to permanent docs\narcfetch promote my-article\n\n# Delete a cached reference\narcfetch delete my-article\n\n# Show current configuration\narcfetch config\n\n# Start MCP server\narcfetch mcp\n```\n\n### Output Formats\n\n```bash\n# Plain text (LLM-friendly, default)\narcfetch fetch https://example.com -o text\n\n# Just the filepath (for scripts)\narcfetch fetch https://example.com -o path\n\n# Summary: slug|filepath\narcfetch fetch https://example.com -o summary\n\n# Structured JSON\narcfetch fetch https://example.com -o json\n\n# Human-friendly with emojis\narcfetch fetch https://example.com --pretty\n```\n\n### Advanced Options\n\n```bash\n# Add search query as metadata\narcfetch fetch https://example.com -q \"machine learning\"\n\n# Set minimum quality threshold (default: 60)\narcfetch fetch https://example.com --min-quality 80\n\n# Force Playwright (skip simple fetch)\narcfetch fetch https://example.com --force-playwright\n\n# Use faster wait strategy for simple sites\narcfetch fetch https://example.com --wait-strategy load\n\n# Re-fetch even if URL already cached\narcfetch fetch https://example.com --refetch\n\n# Custom directories\narcfetch fetch https://example.com --temp-dir .cache --docs-dir content\n\n# Verbose output for debugging\narcfetch fetch https://example.com -v\n```\n\n## MCP Server\n\n### Installation (Recommended: bunx)\n\nAdd to your Claude Code MCP configuration:\n\n```json\n{\n  \"mcpServers\": {\n    \"arcfetch\": {\n      \"command\": \"bunx\",\n      \"args\": [\"arcfetch\"]\n    }\n  }\n}\n```\n\nOr using `npx` when `bun` is installed on `PATH`:\n\n```json\n{\n  \"mcpServers\": {\n    \"arcfetch\": {\n      \"command\": \"npx\",\n      \"args\": [\"arcfetch\"]\n    }\n  }\n}\n```\n\n### Local Development\n\n```json\n{\n  \"mcpServers\": {\n    \"arcfetch\": {\n      \"command\": \"bun\",\n      \"args\": [\"run\", \"/path/to/arcfetch/index.ts\"],\n      \"cwd\": \"/path/to/arcfetch\"\n    }\n  }\n}\n```\n\n### MCP Tools\n\n| Tool | Parameters | Description |\n|------|------------|-------------|\n| `fetch_url` | `url`, `query?`, `minQuality?`, `refetch?`, `outputFormat?`, `waitStrategy?`, `forcePlaywright?` | Fetch URL with auto JS fallback |\n| `list_cached` | `tempDir?` | List all cached references |\n| `promote_reference` | `refId`, `tempDir?`, `docsDir?` | Move from temp to docs folder |\n| `delete_cached` | `refId`, `tempDir?` | Delete a cached reference |\n| `extract_links` | `refId`, `tempDir?`, `outputFormat?` | Extract links from a cached reference |\n| `fetch_links` | `refId`, `tempDir?`, `docsDir?`, `refetch?`, `outputFormat?` | Fetch all links from a cached reference |\n\n## Configuration\n\n### Config File\n\nCreate `arcfetch.config.json` in your project root:\n\n```json\n{\n  \"quality\": {\n    \"minScore\": 60,\n    \"jsRetryThreshold\": 85\n  },\n  \"paths\": {\n    \"tempDir\": \".tmp/arcfetch\",\n    \"docsDir\": \"docs/ai/references\"\n  },\n  \"playwright\": {\n    \"timeout\": 30000,\n    \"waitStrategy\": \"networkidle\"\n  }\n}\n```\n\nConfig files checked (in order): `arcfetch.config.json`, `.arcfetchrc`, `.arcfetchrc.json`\n\n### Environment Variables\n\n```bash\nARCFETCH_MIN_SCORE=60\nARCFETCH_JS_RETRY_THRESHOLD=85\nARCFETCH_TEMP_DIR=.tmp/arcfetch\nARCFETCH_DOCS_DIR=docs/ai/references\n```\n\nLegacy `SOFETCH_*` names are still supported for existing setups.\n\n### Priority Order\n\nCLI arguments > Environment variables > Config file > Built-in defaults\n\n## How It Works\n\n![ArcFetch pipeline: fetch URL with Playwright fallback, extract with Mozilla Readability + Turndown, run quality gates that detect boilerplate / paywalls / login walls, output clean markdown or JSON](docs/explainer.png)\n\nThe pipeline runs four stages in order — fetch, extract, quality-gate, output — with a Playwright fallback when simple HTTP returns blank or low-quality content.\n\n### Quality Scoring\n\nScore starts at 100, deductions apply:\n\n| Check | Deduction |\n|-------|-----------|\n| Blank content | Score = 0 |\n| Content < 50 chars | -50 |\n| Content < 300 chars | -15 |\n| HTML tags > 100 | -40 |\n| HTML tags > 50 | -20 |\n| HTML ratio > 30% | -25 |\n| Extraction ratio < 0.5% (large page) | -35 |\n| Extraction ratio < 2% (large page) | -20 |\n| Boilerplate detected | -40 |\n| Script/style tags | -10 to -15 |\n\n### Boilerplate Detection\n\nOn short content (< 2000 chars), 22 patterns are checked:\n\n- **Error pages**: \"something went wrong\", \"an error occurred\"\n- **404 pages**: \"page not found\"\n- **Login walls**: \"log in to continue\", \"please log in\", \"sign in to continue\"\n- **Paywalls**: \"subscribe to continue reading\"\n- **Bot detection**: \"are you a robot\", \"complete the captcha\"\n- **Access denied**, **JS-required**, **unsupported browser**\n\nLong articles (>= 2000 chars) are not checked for boilerplate to avoid false positives.\n\n### Playwright Wait Strategies\n\n| Strategy | Speed | Reliability | Best For |\n|----------|-------|-------------|----------|\n| `networkidle` | Slowest | Highest | JS-heavy apps, dynamic content |\n| `domcontentloaded` | Medium | Medium | Most SPAs, modern sites |\n| `load` | Fastest | Basic | Static sites, simple pages |\n\n## File Format\n\nCached files use markdown with YAML frontmatter:\n\n```markdown\n---\ntitle: \"Article Title\"\nsource_url: \"https://example.com/article\"\nfetched_date: 2026-02-06\ntype: web\nstatus: temporary\nquery: \"optional search query\"\n---\n\n# Article Title\n\nExtracted markdown content...\n```\n\n- **Ref IDs** are slugified titles (e.g., `how-to-build-react-apps`)\n- **Temp storage**: `.tmp/arcfetch/<slug>.md` (status: temporary)\n- **Permanent storage**: `docs/ai/references/<slug>.md` (status: permanent, after promote)\n- **Duplicate detection**: re-fetching same URL returns existing ref unless `--refetch`\n\n## Real-World Examples\n\n### AI Research Workflow\n\n```bash\n# Fetch multiple articles for research\narcfetch fetch https://arxiv.org/abs/2301.00001 -q \"LLM research\"\narcfetch fetch https://openai.com/research/gpt-4 -q \"GPT-4\"\n\n# Review all cached references\narcfetch list --pretty\n\n# Promote the good ones to docs\narcfetch promote llm-research-paper\narcfetch promote gpt-4-technical-report\n```\n\n### Link Crawling Workflow\n\n```bash\n# Fetch a page with lots of links\narcfetch fetch https://example.com/resources --pretty\n\n# See what links it contains\narcfetch links resources --pretty\n\n# Fetch all of them in parallel\narcfetch fetch-links resources --pretty\n```\n\n### Script Integration\n\n```bash\n#!/bin/bash\n# Fetch and get filepath\nfilepath=$(arcfetch fetch https://example.com -o path)\n\n# Process with other tools\ncat \"$filepath\" | other-tool\n\n# Or get JSON for structured processing\narcfetch fetch https://example.com -o json | jq '.quality'\n```\n\n### Handling JS-Heavy Sites\n\n```bash\n# Modern React/Vue/Angular apps\narcfetch fetch https://spa-example.com --force-playwright\n\n# Simple blogs (use faster strategy)\narcfetch fetch https://blog.example.com --wait-strategy load\n\n# Unknown site (let arcfetch decide automatically)\narcfetch fetch https://unknown-site.com -v\n```\n\n## Troubleshooting\n\n### Low Quality Score\n\n```bash\n# Lower the threshold temporarily\narcfetch fetch https://example.com --min-quality 40\n\n# Or force Playwright (often produces better results)\narcfetch fetch https://example.com --force-playwright\n\n# Check what's happening with verbose mode\narcfetch fetch https://example.com -v\n```\n\n### Timeout on Slow Sites\n\n```bash\n# Use faster wait strategy\narcfetch fetch https://example.com --wait-strategy load\n\n# Combine with force-playwright for JS sites\narcfetch fetch https://example.com --force-playwright --wait-strategy domcontentloaded\n```\n\n## Architecture\n\n```\n┌──────────────────────────────────────────────────────┐\n│                  CLI / MCP Interface                  │\n└──────────────────────────────────────────────────────┘\n                           │\n                           ▼\n┌──────────────────────────────────────────────────────┐\n│                   Core Pipeline                       │\n│  1. Simple HTTP Fetch (browser-like UA)               │\n│  2. Extract with Readability + Turndown               │\n│  3. Quality Score + Boilerplate Detection             │\n│  4. Conditional Playwright Retry (with stealth)       │\n│  5. Cache with YAML Frontmatter                       │\n└──────────────────────────────────────────────────────┘\n                           │\n           ┌───────────────┼───────────────┐\n           ▼               ▼               ▼\n     ┌───────────┐  ┌───────────┐  ┌───────────┐\n     │   Cache   │  │ Playwright│  │ Quality   │\n     │  Manager  │  │  Manager  │  │ Validator │\n     └───────────┘  └───────────┘  └───────────┘\n```\n\n## Contributing\n\n```bash\ngit clone https://github.com/briansunter/arcfetch.git\ncd arcfetch\nbun install\nbun test          # Run tests\nbun run typecheck # Type checking\nbun run check     # Lint + format check\n```\n\n## License\n\nMIT License - see LICENSE file for details\n"}