{"id":"b7d0e535-d2ca-48b0-abf1-f0cf5045eae9","slug":"officialspicrawl-agent-plugins","kind":"mcp","name":"agent-plugins","summary":"Web scraping plugins for AI coding agents (Claude Code, Cursor, Codex, Gemini CLI, OpenCode, Copilot CLI, Factory Droid, Devin) and Dify: the Spicrawl MCP server, packaged. URLs to Markdown, batch job","owner":"OfficialSpicrawl","repoUrl":"https://github.com/OfficialSpicrawl/agent-plugins","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-10-05T06:45:48.000Z","addedAt":"2026-10-21T05:15:54.165Z","logoUrl":"https://avatars.githubusercontent.com/u/337738096?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["developer-tools","browser-web","files-documents"],"tags":["claude-code-plugin","gemini-cli-extension","opencode-plugin","web-scraping","ai-agents","claude-code","codex","cursor"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-27T01:52:32.512Z","sample":false,"url":"https://conduits.dev/tools/officialspicrawl-agent-plugins","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Spicrawl agent plugins: web scraping for Claude Code, Cursor, Codex and Gemini CLI\n\n[![smithery badge](https://smithery.ai/badge/spicrawl/web-scraper)](https://smithery.ai/servers/spicrawl/web-scraper)\n\n**How do I give Claude Code, Cursor, Codex or Gemini CLI web scraping?** Connect the agent to\nSpicrawl's hosted [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server at\n`https://mcp.spicrawl.com/mcp`. This repository packages that connection as a plugin or extension\nfor each tool, so setup is one install command plus an API key from <https://app.spicrawl.com>.\nThrough the full server the agent gets 24 `spicrawl_*` tools: fetch a web page as clean Markdown\n(or HTML, text or JSON), render pages that need JavaScript, extract fields with CSS selectors, run\nbatch jobs over lists of URLs, keep login sessions, and read its own request history and usage. For\nClaude Code, run `claude plugin marketplace add OfficialSpicrawl/agent-plugins && claude plugin install spicrawl@spicrawl-plugins`.\nFor Gemini CLI, run `gemini extensions install https://github.com/OfficialSpicrawl/agent-plugins`.\nCursor and Codex are in the table below.\n\nSpicrawl's plugins for coding agents and workflow tools, in one repository. For coding agents it\nconnects the agent to the hosted Spicrawl MCP server and, in the tools that support it, gives the\nagent a skill that explains how to use it. For Dify and n8n it adds Spicrawl as a plugin or node\nthat calls the Spicrawl API from a workflow.\n\nSpicrawl is a web scraping API for AI agents. Through the MCP server an agent can:\n\n- scrape a web page to Markdown, HTML, text or JSON;\n- run batch jobs over lists of URLs;\n- keep persistent login sessions (cookies and storage);\n- read its request history and its usage;\n- search Spicrawl's docs.\n\n## Supported tools\n\nEach tool needs a Spicrawl API key; see [Get an API key](#get-an-api-key) and the section for your\ntool under [Install](#install).\n\n| Tool | Install | What you get |\n|---|---|---|\n| Claude Code | `claude plugin marketplace add OfficialSpicrawl/agent-plugins && claude plugin install spicrawl@spicrawl-plugins` | plugin (MCP server + skill); Claude Code asks for the key when you enable it |\n| Cursor | `git clone https://github.com/OfficialSpicrawl/agent-plugins.git && mkdir -p ~/.cursor/plugins/local && cp -r agent-plugins/plugins/spicrawl ~/.cursor/plugins/local/spicrawl` | plugin (MCP server + skill); restart Cursor, then set `SPICRAWL_API_KEY` under Plugins, Configure |\n| Codex | `codex plugin marketplace add OfficialSpicrawl/agent-plugins` | plugin (skill + MCP server entry), in OpenAI's portable format; then install `spicrawl` from `/plugins`. The key goes in your Codex config (see [ChatGPT and Codex](#chatgpt-and-codex)) |\n| ChatGPT | not in the directory yet | the same plugin, connecting through OAuth at `/chatgpt/mcp` once that endpoint is live (see [ChatGPT and Codex](#chatgpt-and-codex)) |\n| Gemini CLI | `gemini extensions install https://github.com/OfficialSpicrawl/agent-plugins` | extension (MCP server + context file); asks for the key as a setting |\n| OpenCode | `\"plugin\": [\"opencode-spicrawl\"]` in `opencode.json` | npm plugin `opencode-spicrawl` (adds the MCP server); reads `SPICRAWL_API_KEY` |\n| Factory Droid | `droid plugin marketplace add OfficialSpicrawl/agent-plugins && droid plugin install spicrawl@spicrawl-plugins --scope user` | plugin (MCP server + skill); reads `SPICRAWL_AUTH` |\n| GitHub Copilot CLI | `copilot plugin marketplace add OfficialSpicrawl/agent-plugins && copilot plugin install spicrawl@spicrawl-plugins` | plugin (skill), plus one command for the MCP server (see [GitHub Copilot CLI](#github-copilot-cli)) |\n| Devin | `devin plugins install --local ./plugins/spicrawl` | plugin (MCP server + skill), run from a clone of this repository |\n| Dify | Plugins, Marketplace, search \"Spicrawl\" | plugin (Scrape, Extract and Batch scrape tools), from the [Dify Marketplace](https://marketplace.dify.ai/plugin/spicrawl/spicrawl) |\n| n8n | `npm install n8n-nodes-spicrawl` in `~/.n8n/nodes`, or Settings, Community Nodes, Install | community node `n8n-nodes-spicrawl` (Scrape URL and Batch Scrape), on npm; source in [OfficialSpicrawl/n8n-nodes-spicrawl](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl) |\n| Any tool that reads Agent Skills | `npx skills add OfficialSpicrawl/agent-plugins` | the skill only |\n| Any MCP client | `https://mcp.spicrawl.com/mcp` (Streamable HTTP, header `Authorization: Bearer <key>`) | the MCP server, by hand |\n\n## Where to find Spicrawl\n\n- Dify Marketplace: <https://marketplace.dify.ai/plugin/spicrawl/spicrawl>\n- Smithery: <https://smithery.ai/servers/spicrawl/web-scraper>\n- Cursor Directory: <https://cursor.directory/plugins/spicrawl>\n- npm: [`@spicrawl/mcp`](https://www.npmjs.com/package/@spicrawl/mcp) (the MCP server),\n  [`opencode-spicrawl`](https://www.npmjs.com/package/opencode-spicrawl) and\n  [`n8n-nodes-spicrawl`](https://www.npmjs.com/package/n8n-nodes-spicrawl)\n- Official MCP Registry: [`io.github.OfficialSpicrawl/mcp`](https://registry.modelcontextprotocol.io/v0/servers?search=io.github.OfficialSpicrawl/mcp)\n- Glama: <https://glama.ai/mcp/connectors/com.spicrawl.mcp>\n\n## Get an API key\n\nCreate a key at <https://app.spicrawl.com>. Keys look like `spicrawl_live_...` or\n`spicrawl_test_...`. The usage tools (`spicrawl_usage`, `spicrawl_usage_summary`,\n`spicrawl_usage_reconciliation`) need a key with the `read` scope.\n\nNever commit a key. None of the files in this repository contain one.\n\n## Install\n\nThe MCP server is `https://mcp.spicrawl.com/mcp` (Streamable HTTP). It authenticates with the\nheader `Authorization: Bearer <key>`. This endpoint does not use OAuth. (ChatGPT uses a separate\nOAuth endpoint, `https://mcp.spicrawl.com/chatgpt/mcp`; see [ChatGPT and Codex](#chatgpt-and-codex).)\nEach tool below takes the key in the way its plugin format allows.\n\n### Claude Code\n\n```sh\nclaude plugin marketplace add OfficialSpicrawl/agent-plugins\nclaude plugin install spicrawl@spicrawl-plugins\n```\n\nClaude Code asks for the API key when the plugin is enabled and stores it as a sensitive value.\n\n### ChatGPT and Codex\n\nOpenAI's plugin lives in its own folder, [`plugins/spicrawl-openai`](plugins/spicrawl-openai), in\nOpenAI's portable (Agent Plugins) format: a root `plugin.json`, an `mcp.json`, a `skills/` folder\nand `assets/`. That format has no field for a key, so the plugin carries none.\n\n```sh\ncodex plugin marketplace add OfficialSpicrawl/agent-plugins\n```\n\nThen run `/plugins` in Codex, pick the `Spicrawl` marketplace and install `spicrawl`. This installs\nthe `fetch-web-pages` skill and declares the Spicrawl MCP server, but Codex cannot give that\ndeclaration your key. The plugin's own declaration points at the ChatGPT endpoint\n(`/chatgpt/mcp`, OAuth, a restricted set of tools), not at the full `/mcp` server. To use your API\nkey and every tool, add the server yourself in `~/.codex/config.toml`, with the key read from the\nenvironment, and switch off the plugin's own copy of it:\n\n```toml\n[mcp_servers.spicrawl]\nurl = \"https://mcp.spicrawl.com/mcp\"\nbearer_token_env_var = \"SPICRAWL_API_KEY\"\n\n[plugins.\"spicrawl@spicrawl-plugins\".mcp_servers.spicrawl]\nenabled = false\n```\n\n```sh\nexport SPICRAWL_API_KEY=spicrawl_live_...   # in the shell you start Codex from\n```\n\nChatGPT: the plugin is not in the ChatGPT directory yet. ChatGPT cannot send a custom API key, so\nit does not use `/mcp`. It connects through OAuth at `https://mcp.spicrawl.com/chatgpt/mcp` once\nthat endpoint is live: you sign in to your Spicrawl account and approve the connection, and no key\nis pasted into ChatGPT. That endpoint offers a restricted set of tools: `spicrawl_scrape` (GET\nrequests only), `spicrawl_batch_submit`, `spicrawl_batch_status`, `spicrawl_batch_results` (up to\n25 URLs per job) and `spicrawl_docs_search` and `spicrawl_docs_read`. Until it is live, ChatGPT\ncannot connect.\n\n### Cursor\n\nInstall the `spicrawl` plugin, then give it your key: in Cursor open Plugins, Configure on\nSpicrawl, and set `SPICRAWL_API_KEY`. The plugin declares it as a required variable, and Cursor\nsubstitutes it into the `Authorization` header of the MCP server. It is a plugin variable that you\nset in Cursor, not a shell environment variable.\n\nUntil the plugin is listed in the Cursor marketplace, copy `plugins/spicrawl` to\n`~/.cursor/plugins/local/spicrawl` and restart Cursor.\n\nTo skip the plugin and add only the server, put this in `~/.cursor/mcp.json`. Here the key does\ncome from the environment Cursor is started from, so export it first:\n\n```sh\nexport SPICRAWL_API_KEY=spicrawl_live_...\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"spicrawl\": {\n      \"url\": \"https://mcp.spicrawl.com/mcp\",\n      \"headers\": { \"Authorization\": \"Bearer ${env:SPICRAWL_API_KEY}\" }\n    }\n  }\n}\n```\n\n### OpenCode\n\n```sh\nexport SPICRAWL_API_KEY=spicrawl_live_...\n```\n\nThen add the plugin to `opencode.json` (project) or `~/.config/opencode/opencode.json` (global):\n\n```json\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"plugin\": [\"opencode-spicrawl\"]\n}\n```\n\nThe plugin (source in [`opencode/`](opencode), on npm as\n[`opencode-spicrawl`](https://www.npmjs.com/package/opencode-spicrawl)) adds the Spicrawl remote\nMCP server to OpenCode's config in memory, with OAuth off and the key from `SPICRAWL_API_KEY` as\nthe bearer token. It writes nothing to your config files. Without the key it adds nothing and logs\na warning. A `mcp.spicrawl` entry that you write by hand wins over the plugin. It installs no\nskill. Check the connection with `opencode mcp list`.\n\n### Factory Droid\n\n```sh\nexport SPICRAWL_AUTH=\"Bearer spicrawl_live_...\"   # the whole header value, including \"Bearer \"\ndroid plugin marketplace add OfficialSpicrawl/agent-plugins\ndroid plugin install spicrawl@spicrawl-plugins --scope user\n```\n\nDroid expands `${SPICRAWL_AUTH}` from the shell environment when it connects. OAuth is disabled\nfor this server in the plugin's `mcp.json`.\n\n### Devin\n\nAdd it as a personal plugin. In Devin Cloud: Customize, Plugins, Add plugin, then \"From repository\"\n(this repository, subdirectory `plugins/spicrawl`) or \"Upload .zip\" (a zip of `plugins/spicrawl`).\nLocally: `devin plugins install --local ./plugins/spicrawl`. Then connect the\n`SPICRAWL_AUTHORIZATION` credential from the plugin's Connect button; its value is the whole header\nvalue: `Bearer spicrawl_live_...`. The manifest is `plugins/spicrawl/.devin-plugin/plugin.json`.\n\nTo reference the plugin from your own Devin plugin manifest, use a `git-subdir` source with\n`\"url\": \"https://github.com/OfficialSpicrawl/agent-plugins.git\"` and `\"path\": \"plugins/spicrawl\"`, pinned\nto a commit `sha`.\n\n### Gemini CLI\n\n```sh\ngemini extensions install https://github.com/OfficialSpicrawl/agent-plugins\n```\n\nGemini asks for one setting, \"Spicrawl authorization header\". Enter the whole header value:\n`Bearer spicrawl_live_...`. To change it later: `gemini extensions config spicrawl`.\nThe extension loads `GEMINI.md` as context.\n\n### GitHub Copilot CLI\n\n```sh\ncopilot plugin marketplace add OfficialSpicrawl/agent-plugins\ncopilot plugin install spicrawl@spicrawl-plugins\n```\n\nThis installs the skill. Copilot CLI's docs describe no environment-variable expansion for remote MCP server\nsettings, so a plugin cannot carry the key. Add the server with:\n\n```sh\ncopilot mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \\\n  --header \"Authorization: Bearer $SPICRAWL_API_KEY\"\n```\n\nYour shell expands the variable, so the key is written to Copilot's MCP config file on your\nmachine.\n\n### Dify\n\nThe Dify plugin (source in [`dify/`](dify)) adds three tools to workflows and agents: Scrape,\nExtract and Batch scrape. It calls the Spicrawl API directly, not the MCP server, so it takes the\nAPI key (with the `scrape` scope, plus `batch` for Batch scrape) in the plugin's authorization\nsettings.\n\n[![Spicrawl for Dify: install, add your key, then Scrape, Extract and Batch scrape in workflows and agents](assets/dify-demo.webp)](https://docs.spicrawl.com/agents/dify)\n\nWatch the full video on the [Dify docs page](https://docs.spicrawl.com/agents/dify), or download [`assets/dify-demo.mp4`](assets/dify-demo.mp4).\n\n1. In Dify open Plugins, Marketplace, search for Spicrawl and select Install, or open the\n   [Spicrawl listing](https://marketplace.dify.ai/plugin/spicrawl/spicrawl). It needs Dify 1.14.2\n   or later.\n2. Choose To Authorize on the Spicrawl plugin and paste your API key.\n\nTo install a specific release, or on a Dify that cannot reach the Marketplace, download\n`spicrawl-<version>.difypkg` from the [`dify-v*` releases](https://github.com/OfficialSpicrawl/agent-plugins/releases)\nof this repository and pick it under Plugins, Install plugin. The package on GitHub is unsigned, so\na self-hosted Dify with signature verification on rejects it; install from the Marketplace there.\n\nSee [`dify/README.md`](dify/README.md) for the tools and their parameters.\n\n### n8n\n\nThe n8n node (source in its own repository, [OfficialSpicrawl/n8n-nodes-spicrawl](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl), on npm as\n[`n8n-nodes-spicrawl`](https://www.npmjs.com/package/n8n-nodes-spicrawl)) is a community node with\ntwo operations, Scrape URL and Batch Scrape. It calls the Spicrawl API directly, not the MCP\nserver.\n\n1. In n8n open Settings, Community Nodes, Install, and enter `n8n-nodes-spicrawl`. On a\n   self-hosted n8n without that screen, run `npm install n8n-nodes-spicrawl` in `~/.n8n/nodes`\n   and restart n8n.\n2. Add a **Spicrawl API** credential and paste your API key into API Key.\n3. Add the **Spicrawl** node to a workflow, or let an AI Agent node call it as a tool.\n\nSee the [n8n node README](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl#readme) for the operations and options.\n\n### The skill alone (Agent Skills tools)\n\n```sh\nnpx skills add OfficialSpicrawl/agent-plugins\n```\n\nThis installs `plugins/spicrawl/skills/spicrawl/SKILL.md` into the agents you pick. The skill\ndescribes the Spicrawl tools, the CLI and the HTTP API. It does not connect the MCP server.\n\n### Any MCP client\n\n| | |\n|---|---|\n| URL | `https://mcp.spicrawl.com/mcp` |\n| Transport | Streamable HTTP |\n| Header | `Authorization: Bearer <your key>` |\n| OAuth | not used; turn automatic OAuth off if the client tries it |\n\n## FAQ\n\n### How do I add web scraping to Claude Code?\n\nInstall the plugin: `claude plugin marketplace add OfficialSpicrawl/agent-plugins`, then\n`claude plugin install spicrawl@spicrawl-plugins`. Or add only the server:\n\n```sh\nclaude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \\\n  --header \"Authorization: Bearer $SPICRAWL_API_KEY\"\n```\n\nThen ask Claude Code to read a URL; it calls `spicrawl_scrape`.\n\n### How do I add web scraping to Cursor?\n\nInstall the `spicrawl` plugin and set `SPICRAWL_API_KEY` under Plugins, Configure (see\n[Cursor](#cursor)), or put the server in `~/.cursor/mcp.json`. The plugin is also listed on the\n[Cursor Directory](https://cursor.directory/plugins/spicrawl).\n\n### How do I add web scraping to Codex or Gemini CLI?\n\nCodex: `codex plugin marketplace add OfficialSpicrawl/agent-plugins`, then add the server with your\nkey to `~/.codex/config.toml` (see [ChatGPT and Codex](#chatgpt-and-codex)). Gemini CLI:\n`gemini extensions install https://github.com/OfficialSpicrawl/agent-plugins`.\n\n### What is MCP, and which server do these plugins use?\n\nMCP, the Model Context Protocol, is an open protocol that lets an AI app call external tools. The\nplugins connect to Spicrawl's hosted server at `https://mcp.spicrawl.com/mcp` over Streamable HTTP,\nwith your API key as a bearer token. Spicrawl fetches the page and returns the content, so nothing\nelse runs on your machine.\n\n### What can the agent do with Spicrawl?\n\nFetch one URL as Markdown, HTML, text or JSON; extract named fields with CSS selectors; take a\nscreenshot; run batch jobs over many URLs; keep a session's cookies and storage between requests;\nread its request history and credit usage; and search Spicrawl's documentation. The Markdown is the\npage's main content, which keeps token use down.\n\n### Does it work on pages that need JavaScript?\n\nYes. If a plain fetch comes back empty, the agent can ask for JavaScript rendering. A plain fetch\ncosts 1 credit and JavaScript rendering costs 3.\n\n### Can it crawl a whole website?\n\nNo. Spicrawl fetches the URLs you give it. To cover a site, collect the URLs (from a sitemap, for\nexample) and submit them as a batch job.\n\n### Can it read pages behind a login?\n\nA session keeps cookies and storage between requests, so the agent can sign in once and read\nfurther pages with the same session. Use it only for sites and accounts you are allowed to access.\n\n### How much does it cost?\n\nCredits are charged only for successful requests, and a failed request costs 0. See\n<https://docs.spicrawl.com/credits> for the price of each kind of request.\n\n### Where does my API key go?\n\nOnly to Spicrawl. See [What is sent where](#what-is-sent-where). No file in this repository contains\na key.\n\n### Which other tools does it support?\n\nChatGPT, OpenCode, Factory Droid, GitHub Copilot CLI, Devin, Dify, n8n, and any tool that reads\nAgent Skills or speaks MCP. See [Supported tools](#supported-tools).\n\n### What license is it under?\n\nThe plugins in this repository are Apache-2.0. Spicrawl itself is a hosted service that needs an\naccount and an API key.\n\n## What is sent where\n\nThe API key is sent only to Spicrawl: to `mcp.spicrawl.com` in the `Authorization` header of MCP\nrequests, and, for the Dify plugin and the n8n node, to `api.spicrawl.com` in the `Authorization`\nheader of API requests. Nothing here contains a key. The plugins for Claude Code, Codex and ChatGPT, Cursor, Factory Droid,\nDevin, Gemini CLI and Copilot CLI run no code of their own: they are JSON manifests, a skill file\n(Markdown) and a logo. The OpenCode plugin is a small JavaScript package (`opencode/`) that reads\n`SPICRAWL_API_KEY` and hands it to OpenCode as the MCP header; it makes no network requests\nitself. The Dify plugin (`dify/`, Python) and the n8n node ([its own repository](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl), TypeScript) make HTTPS requests to\n`https://api.spicrawl.com` and nowhere else; they read the key from Dify's plugin authorization\nand from an n8n credential. Each tool reads the key from where its format allows: a sensitive\nplugin setting (Claude Code, Gemini CLI), a plugin variable you set in Cursor, an environment variable (Codex, through\nyour own `config.toml` entry; Factory Droid; OpenCode), a credential you enter in Devin, or the\ncommand you type (Copilot CLI, any MCP client).\n\nThe pages an agent asks Spicrawl to scrape, and the results, go through the same server.\n\n## Docs and contact\n\n- MCP server docs: <https://docs.spicrawl.com/agents/mcp>\n- Website: <https://spicrawl.com>\n- SDK: <https://github.com/OfficialSpicrawl/sdk>, CLI: <https://github.com/OfficialSpicrawl/cli> (Apache-2.0)\n- Contact: dev@spicrawl.com\n\nLicensed under the Apache License 2.0. See [LICENSE](LICENSE).\n\n## Maintainer notes\n\nThere are two plugin folders:\n\n- `plugins/spicrawl`: Claude Code, Cursor, Factory Droid, Devin and Copilot CLI, and the skill that\n  `npx skills add` installs. Every one of these needs a key in the MCP definition (or, for Copilot\n  CLI, no MCP definition at all).\n- `plugins/spicrawl-openai`: ChatGPT and Codex only, in OpenAI's portable format. It is the folder\n  that goes into the ZIP for OpenAI's plugin portal, and `.agents/plugins/marketplace.json`\n  points at it. See [The OpenAI folder](#the-openai-folder).\n\nShared files in `plugins/spicrawl`:\n\n- `skills/spicrawl/SKILL.md`: the canonical skill. `name` equals its directory.\n  The skills CLI finds it because `.claude-plugin/marketplace.json` lists `./plugins/spicrawl`,\n  so no copy or symlink at the repository root is needed.\n- `logo.png`: 180x180, used by the Cursor and Devin manifests. `plugins/spicrawl-openai/assets/logo.png`\n  is a byte-identical copy (a ZIP must be self-contained, and a symlink is not accepted). Change both together.\n- `.devin-plugin/plugin.json` is the manifest prepared for Devin's marketplace.\n\nThere is one MCP definition per format, in `plugins/spicrawl/`. They are separate files on\npurpose: the same server needs a different credential syntax in each tool.\n\n| File | Read by | Credential syntax |\n|---|---|---|\n| `mcp/claude.json` | Claude Code (`mcpServers` in `.claude-plugin/plugin.json`) | `${user_config.spicrawl_api_key}`, from `userConfig` with `sensitive: true` |\n| `mcp/cursor.json` | Cursor (`mcpServers` in `.cursor-plugin/plugin.json`) | `${SPICRAWL_API_KEY}`, from `variables` in `.cursor-plugin/plugin.json`; the user sets it under Plugins, Configure |\n| `mcp.json` | Factory Droid (fixed root path) | `${SPICRAWL_AUTH}` as the whole header value, `oauth: false` |\n| inline in `.devin-plugin/plugin.json` | Devin | `${SPICRAWL_AUTHORIZATION}` as the whole header value |\n| inline in `gemini-extension.json` | Gemini CLI | `$SPICRAWL_MCP_BEARER` from the extension `settings` |\n| `opencode/index.js` (npm `opencode-spicrawl`) | OpenCode | `Bearer <value of SPICRAWL_API_KEY>`, built in code |\n| `plugins/spicrawl-openai/mcp.json` | ChatGPT, Codex (Agent Plugins `mcp.json`) | none in the file: ChatGPT authenticates with OAuth at `/chatgpt/mcp`; Codex users who want a key add `/mcp` themselves in `~/.codex/config.toml` |\n\nNotes:\n\n- No file is named `.mcp.json`. It is Claude Code's default name, but Copilot CLI also reads it by\n  default and Droid translates it, and neither expands `${user_config...}`.\n- Cursor's manifest points at `mcp/cursor.json` so its default `mcp.json` discovery does not pick\n  up the Droid file. Every `${VAR}` in that file must be declared under `variables` in the\n  manifest, and Cursor does not read it from the shell: `${env:...}` is valid only in the user's\n  own `~/.cursor/mcp.json`.\n- `opencode/` is the source of the npm package `opencode-spicrawl` (OpenCode has no plugin\n  marketplace format). It has its own `package.json` version and is published to npm separately\n  from these manifests, so its version does not have to match `0.2.2`.\n- The collection version is `version` in `gemini-extension.json` and in the manifests under\n  `plugins/` (`.claude-plugin`, `.cursor-plugin`, `.factory-plugin`, `.github/plugin` and\n  `spicrawl-openai/plugin.json`). It is `0.2.2` (0.2.0 added the Dify and n8n integrations; 0.2.1 aligned the skill with the public\n  rendering tiers; 0.2.2 added the plugin `README.md` and `LICENSE`, and the category and tags in the Cursor manifest, for Cursor marketplace review) and these files are bumped together. `opencode/` and `dify/` are versioned on their own, and so is the n8n node, which lives in its own repository.\n- `dify/` is the Dify plugin. The n8n community node is not in this repository: its source is\n  [OfficialSpicrawl/n8n-nodes-spicrawl](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl), with the package at that repository's root because\n  n8n's verification looks for `credentials/` there. Neither is a coding-agent plugin: they call\n  the REST API (`https://api.spicrawl.com`), not the MCP server, and no marketplace file in this\n  repository lists them. `dify/` is Python (tests: `cd dify && python -m pytest`). In a batch\n  result line, `result.content` is the page as a plain string, and JSON text only when the job's\n  `response_format` is `json`; both integrations use it that way.\n- The Gemini variable name must not contain `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `AUTH`,\n  `CREDENTIAL`, `CERT` or `PRIVATE`: Gemini CLI removes such variables from the environment before\n  it expands header values.\n- There is no root `plugin.json` and no Agent Plugins `mcp.json` in `plugins/spicrawl`, and there\n  must not be. Checked again against the current docs:\n  - The standard (agent-plugins.org, section 7.2.1) says \"Clients MUST NOT perform placeholder or\n    environment-variable expansion in url, header names, or header values\" and \"Plugins MUST NOT\n    embed credentials or other secrets in headers\", and defines no OAuth or credential-reference\n    fields. A portable package cannot carry a bearer key.\n  - Copilot CLI's reference says \"A root plugin.json that targets Agent Plugins takes precedence\n    over .plugin/plugin.json and .claude-plugin/plugin.json\". Adding one would hide\n    `.github/plugin/plugin.json` and the Claude manifest.\n  - OpenAI's packaging guide says portable packages \"always discover skills in skills/ and MCP\n    servers in mcp.json\", and that an MCP declaration in an OpenAI overlay \"can't replace, disable,\n    or add to those components\". A root manifest in this folder would drop Codex's\n    `bearer_token_env_var`.\n  - Factory Droid already owns the root `mcp.json` path in this folder, with a different shape\n    (`type: http`, `${SPICRAWL_AUTH}`). The Agent Plugins `mcp.json` needs the same path with\n    `type: streamable-http` and a `$schema`.\n\n  So the portable package is a separate folder, `plugins/spicrawl-openai`, and no marketplace\n  other than `.agents/plugins/marketplace.json` lists it. The `skills/` layout in `plugins/spicrawl`\n  follows the standard anyway.\n- `plugins/spicrawl/.github/plugin/plugin.json` gives Copilot CLI a manifest without an MCP\n  server. Without it Copilot CLI would read the Claude manifest and send the literal header.\n- Manifests list only what the server offers today.\n- Coming soon, so absent from every manifest and skill step: AI extraction, stealth mode, the managed proxy pool, and the remote browser (`spicrawl_browser_connect_url`).\n- After changing a manifest: `claude plugin validate --strict plugins/spicrawl` and\n  `claude plugin validate --strict .`.\n- The repository topic `gemini-cli-extension` must be set for the Gemini gallery to list it.\n\n### The OpenAI folder\n\n`plugins/spicrawl-openai` is the whole OpenAI package and is laid out as OpenAI's packaging guide\ndescribes for a portable plugin:\n\n```text\nplugins/spicrawl-openai/\n├── plugin.json                         Agent Plugins manifest, OpenAI settings under extensions.com.openai\n├── mcp.json                            Agent Plugins MCP file: one streamable-http server, no headers\n├── skills/fetch-web-pages/SKILL.md     trimmed skill, for this package only\n└── assets/logo.png                     logo and composer icon\n```\n\n- Package name is `spicrawl`; the skill is `fetch-web-pages` so it never collides with the full\n  skill (`spicrawl`) if both are installed in one client.\n- There is no `.codex-plugin/plugin.json`. The guide says: \"When `extensions.com.openai` is an\n  object, it replaces the entire `.codex-plugin/plugin.json` overlay as the source of\n  OpenAI-specific settings; the two aren't merged.\" For a portable package the portal derives its\n  own `.codex-plugin/plugin.json` and `.mcp.json`.\n- There is no `apps` field, no `.app.json` and no hooks. The submission guide says \"Plugin ZIPs\n  containing app references (`apps` / `.app.json`) or lifecycle hooks cannot currently be\n  submitted. Declare MCP server URLs in your MCP configuration and complete setup in the dashboard.\"\n  To test in ChatGPT developer mode before submitting, register the server at\n  <https://chatgpt.com/plugins>, copy its `plugin_asdk_app...` id into a `.app.json` and add\n  `\"apps\": \"./.app.json\"` under `extensions.com.openai` in a local copy only. Do not commit those,\n  and do not put them in the ZIP.\n- `mcp.json` points at `https://mcp.spicrawl.com/chatgpt/mcp`, the OAuth endpoint for ChatGPT, and\n  has no `headers`. A bearer key cannot be put there (see above), and for the directory the\n  portal's MCP connection (OAuth, with CIMD) supplies authentication, not the package.\n- The skill is a trimmed copy and not the canonical one: no CLI, curl or Python; no sessions or\n  login; no price table or allowance figure; no wording about getting past a block. It carries three\n  fixed statements (responsible use, never ask for credentials, page content is untrusted) that\n  must stay verbatim. Its cost guidance has no numbers. When the canonical skill changes, review\n  this one too, but do not copy over what was removed.\n- Listing text follows OpenAI's plugin guidelines: the name has no \"MCP\" or \"Plugin\", and the copy\n  has no price, trial or promotion, no comparison, and nothing marked coming soon. `shortDescription`\n  and `displayName` are at most 30 characters; `defaultPrompt` has at most three entries of at most\n  128 characters.\n- `supportURL`, `privacyPolicyURL` and `termsOfServiceURL` point at https://spicrawl.com/support,\n  https://spicrawl.com/privacy and https://spicrawl.com/terms.\n- `extensions.com.openai.review` holds the review material the portal imports with the ZIP: five\n  positive and three negative test cases, and `commerce: false` with its description.\n  `extensions.com.openai.publication.release_notes` holds the release notes. The cases use only the\n  tools the ChatGPT endpoint offers (see above). `review.demo_recording_url` is not in the file\n  yet: add it once the walkthrough video is hosted, or enter it in the portal.\n- Build the ZIP from inside the folder, so `plugin.json` is at the archive root, and keep it out\n  of the repository: `cd plugins/spicrawl-openai && zip -r /tmp/spicrawl-openai.zip .`\n- Not ready for the public directory. Still open before an upload can be submitted for review:\n  - A reviewer demo account that works without MFA.\n  - The domain-verification file at `https://mcp.spicrawl.com/.well-known/openai-apps-challenge`\n    (the portal shows the token).\n  - `review.demo_recording_url`, the video walkthrough.\n\n## Releasing\n\nThe OpenCode plugin (`opencode/`, npm `opencode-spicrawl`) is released by `.github/workflows/release-opencode.yml`.\n\n1. Bump `version` in `opencode/package.json` and add a `## X.Y.Z (YYYY-MM-DD)` section to `opencode/CHANGELOG.md`.\n2. Merge to `main`. CI runs `npm ci`, the typecheck and the tests. If npm does not have that version yet, it publishes with provenance and creates a GitHub release tagged `opencode-spicrawl-vX.Y.Z` whose notes are that CHANGELOG section. A version containing `-` (for example `0.2.0-beta.1`) is published under the `next` dist-tag and marked as a prerelease.\n3. The `NPM_TOKEN` repository secret is a granular npm token for the `spicrawl` account with publish rights and 2FA bypass. It lasts at most 90 days, so rotate it before then. Move to OIDC trusted publishing before January 2027, when npm ends direct token publishing.\n\nThe n8n node (npm `n8n-nodes-spicrawl`) is released from its own repository, [OfficialSpicrawl/n8n-nodes-spicrawl](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl), by that repository's `.github/workflows/publish.yml`: bump `version` and the changelog, merge, then push the tag `vX.Y.Z`. It has its own `NPM_TOKEN` secret.\n\nThe Dify plugin (`dify/`) is released by `.github/workflows/release-dify.yml`.\n\n1. Bump `version` in `dify/manifest.yaml` (the `meta.version` field is the manifest format version; leave it).\n2. Merge to `main`, then push the tag `dify-vX.Y.Z` on the merged commit. The tag must match the manifest. The workflow packages `spicrawl-X.Y.Z.difypkg` with the Dify plugin CLI (pinned to one release and checked against its SHA-256) and attaches it to a GitHub release of that tag.\n\nThe other tools install straight from this repository, so they need no release step: a change is live once it is on `main`.\n\nThe `validate` workflow runs on every push and pull request. It parses all JSON files, runs the OpenCode tests, runs the Dify plugin's pytest, and fails if any file contains something matching `spicrawl_(live|test)_[a-z0-9]{20,}`.\n"}