{"id":"8f010750-80c3-4850-8a46-6fe9b0f908f9","slug":"sukauto","kind":"mcp","name":"Sukauto","summary":"AI-powered market analysis platform that provides in-depth insights acorss multiple business domains.","owner":"rexxAman","repoUrl":"https://github.com/rexxAman/Sukauto","license":"MIT","licenseTier":"permissive","trust":"community","stars":4,"match":null,"fewStars":true,"format":"mcp","createdAt":"2025-07-13T12:23:00.000Z","addedAt":"2026-10-21T15:33:13.636Z","logoUrl":"https://avatars.githubusercontent.com/u/141953689?u=a68a9452364622a37e54296086443e3d49b58bb2&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["data-analytics","browser-web","search"],"tags":["market-intelligence","prompt-engineering","multi-agent-systems","scraping","rag-pipeline"],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T19:36:30.693Z","sample":false,"url":"https://conduits.dev/tools/sukauto","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Architectural Documentation: MarketScout\n\nThis document provides a comprehensive, production-grade architectural and technical analysis of the **MarketScout** system. It details the multi-agent orchestration state machine, tool execution layer via the Model Context Protocol (MCP), multi-model LLM routing, real-time evidence persistence, dual-storage backend, and step-by-step developer workflows.\n\n---\n\n## 1. Project Overview\n\n### What problem does this project solve?\nTraditional market research is slow, fragmented, and prone to hallucinations. Analysts must search search engines, monitor social platforms (X/Twitter), inspect video transcripts (YouTube), and scrape web pages to extract insights. \n\n**MarketScout** automates deep market intelligence by orchestrating a multi-agent AI system that plans research, executes specialized tools via MCP, verifies and synthesizes evidence, drafts comprehensive reports, reflects on knowledge gaps through a LangGraph Map-Reduce loop, and exports polished PDF documents.\n\n### Research Analysis Workflows\nMarketScout supports 6 distinct market research workflows:\n1. **Industry Analysis** (`industry_analysis` / `Industry Report`): Market size, growth drivers, value chain, regulatory landscape, and trends.\n2. **Competitor Analysis** (`competitor_analysis` / `Competitor Report`): Competitor profiling, feature matrices, pricing models, strengths, weaknesses, and market positioning.\n3. **Market Gap Analysis** (`market_gap_analysis` / `Market Gap Report`): Unmet customer needs, underserved segments, technological voids, and product-market fit opportunities.\n4. **Target Market Segmentation** (`target_market_analysis` / `Target Market Report`): Demographic, firmographic, psychographic, and behavioral customer segmentation.\n5. **Barrier Assessment** (`barrier_analysis` / `Barrier Report`): Regulatory barriers, capital requirements, distribution moats, switching costs, and IP hurdles.\n6. **Sales Forecasting** (`sales_forecasting` / `Sales Forecast Report`): Multi-scenario demand projections (Bear/Base/Bull), unit economics, pricing elasticity, and revenue drivers.\n\n### High-Level Workflow\n```mermaid\nflowchart TD\n    A[User Request / Topic] --> B[FastAPI Backend :8000]\n    B --> C[(Storage Backend: Local JSON / MongoDB)]\n    B --> D[Step 1: Planner Agent]\n    D --> E[Step 2: Tool Routing ReAct Agent]\n    E --> F[FastMCP Tool Gateway :5001]\n    F --> G[Google, Crawl4AI, YouTube, X/Twitter]\n    G --> E\n    E --> H[Atomic Evidence & URL Persistence]\n    H --> I[Step 3: Evidence Analysis & Synthesis]\n    I --> J[Step 4: Initial Report Drafting]\n    J --> K[Step 5: LangGraph Reflection & Gap Filling Loop]\n    K -->|Find Gaps -> Map: Fill Gaps -> Reduce: Merge| K\n    K --> L[Markdown-to-PDF Generator]\n    L --> M[Download Report & Real-Time Logs]\n```\n\n---\n\n## 2. Overall Architecture\n\nMarketScout is built around a **Modular Micro-Services & Agentic Architecture**. It separates the execution of the agent state machine (the orchestrator) from the tool execution layer (data collectors) using the **Model Context Protocol (MCP)** over Server-Sent Events (SSE).\n\n```\n                            ┌────────────────┐\n                            │     Users      │\n                            └───────┬────────┘\n                                    │ HTTP / WebSocket\n                                    ▼\n                      ┌──────────────────────────────┐\n                      │   Frontend (React/Next.js)   │\n                      │    or CLI / Terminal Tool    │\n                      └──────────────┬───────────────┘\n                                     │\n                        HTTP POST    │ WebSocket\n                        /analysis    │ /ws/research/{id}\n                                     ▼\n                      ┌──────────────────────────────┐\n                      │      Backend (FastAPI)       │\n                      │          (Port 8000)         │\n                      └─────┬────────┬────────┬──────┘\n                            │        │        │\n                            │        │        │ Run LangGraph Engine\n                Write/Read  │        │        └──────────────┐\n                Metadata &  │        │                       │\n                Evidence    │        │ Serve PDF             ▼\n                            ▼        │ /reports/... ┌─────────────────┐\n                      ┌───────────┐  │              │    LangGraph    │\n                      │  Storage  │  │              │  Agent workflow │\n                      │  Backend  │  │              └────────┬────────┘\n                      │(Local/DB) │  │                       │\n                      └───────────┘  ▼                       │ Run Tools\n                              ┌─────────────┐                │ (MCP SSE Client)\n                              │ Local PDF   │◄───────────────┤ \n                              │ Storage     │ (Saves PDF)    │\n                              └─────────────┘                ▼\n                                                    ┌─────────────────┐\n                                                    │   MCP Gateway   │\n                                                    │   (FastAPI)     │\n                                                    │   (Port 5001)   │\n                                                    └────────┬────────┘\n                                                             │ SSE Transports\n                                                             ▼\n                                                    ┌─────────────────┐\n                                                    │   FastMCP Tool  │\n                                                    │     Servers     │\n                                                    └────────┬────────┘\n                                                             │\n                      ┌──────────────────────────────────────┼──────────────────────────────────────┐\n                      │                      │               │                      │               │\n                      ▼                      ▼               ▼                      ▼               ▼\n              ┌──────────────┐        ┌──────────────┐ ┌───────────┐        ┌──────────────┐\n              │ google_tools │        │ scraper_tools│ │youtube_tools │     │   x_tools    │\n              └──────┬───────┘        └─────┬────────┘ └──────┬───────┘     └──────┬───────┘\n                     │                      │                 │                    │\n                     ▼                      ▼                 ▼                    ▼\n                 Google APIs            Crawl4AI            YouTube           X API v2 &\n               (Search, Trends,        Scraper API        Transcripts       SERP Fallback\n               Shopping, News)\n```\n\n### Architectural Tiers:\n1. **Frontend / Terminal Client**: \n   - Interactive React/Next.js UI or terminal test harness (`tests/run_terminal_research.py`).\n   - Submits structured analysis requests, listens to real-time execution logs via WebSockets, and downloads generated PDF reports.\n2. **Backend Gateway (FastAPI, Port 8000)**: \n   - REST API handling request validation (`CreateAnalysisRequest`), storage persistence, PDF file serving, and WebSocket streaming.\n   - Spawns background worker tasks via `asyncio.create_task` and dispatches streaming updates through an in-memory queue.\n3. **Pluggable Storage Backend (`database/storage.py`)**:\n   - Dual-mode storage: **`LocalFileStorage`** (zero-dependency JSON storage in `data/analyses/`) or **`MongoStorage`** (MongoDB Atlas/community).\n   - Real-time atomic persistence of search evidence, citation URLs, and intermediate report drafts.\n   - Global query cache (`data/search_cache.json` / Mongo `search_cache`) to avoid duplicate external API calls.\n4. **Agent Orchestrator & State Machine (`agents/`)**:\n   - Linear preparation pipeline: **Planner** $\\rightarrow$ **Tool Routing ReAct Agent** $\\rightarrow$ **Evidence Synthesizer** $\\rightarrow$ **Report Writer**.\n   - Cyclic self-correction engine: **LangGraph Map-Reduce loop** that discovers gaps, executes targeted tool searches in parallel, and merges findings up to $k$ iterations based on `research_depth`.\n5. **Model Context Protocol (MCP) Gateway (Port 5001)**:\n   - FastAPI server mounting FastMCP servers via Server-Sent Events (`/mcp/{server}/sse`).\n   - Integrates Google Search & Shopping, Crawl4AI web scrapers, YouTube transcript fetchers, and X (Twitter) social analyzers.\n6. **Multi-Model LLM Routing**:\n   - Supports Google Gemini, Ollama (local models like `llama3.2:3b`, `phi3:mini`, `gemma3:1b`), Groq, Jan AI, and Hugging Face.\n   - Distinct model configurations per pipeline stage (`PLANNER_MODEL`, `TOOL_ROUTING_MODEL`, `EVIDENCE_ANALYSIS_MODEL`, `REPORT_WRITING_MODEL`, `REPORT_REVIEW_MODEL`).\n\n---\n\n## 3. End-to-End Execution Flow\n\n```\nUser/Client       FastAPI (8000)       Storage Backend      Planner & ReAct       MCP Gateway (5001)    LangGraph Loop\n    │                   │                    │                     │                      │                  │\n    │──[1] POST /anal.─>│                    │                     │                      │                  │\n    │                   │──[2] Save Analysis>│                     │                      │                  │\n    │                   │      (Pending)     │                     │                      │                  │\n    │<──[3] Return ID───│                    │                     │                      │                  │\n    │                   │──[4] Spawn Task ────────────────────────>│                      │                  │\n    │──[5] Open WS ────>│                    │                     │                      │                  │\n    │                   │                    │                     │──Step 1: Plan───────>│                  │\n    │                   │                    │                     │──Step 2: Tool Run ──>│                  │\n    │                   │                    │                     │                      │──Run Tools──────>│\n    │                   │                    │                     │<─Tool Results────────│<──Data/URLs──────│\n    │                   │                    │<──Push Evidence─────│                      │                  │\n    │<──[6] WS Logs ────│<──Queue Stream─────│                     │──Step 3: Synthesis──>│                  │\n    │                   │                    │<──Save Draft────────│──Step 4: Draft──────>│                  │\n    │                   │                    │                     │                      │                  │\n    │                   │                    │                     │──Step 5: Invoke Graph──────────────────>│\n    │                   │                    │                     │                      │                  │──Find Gaps\n    │                   │                    │                     │                      │                  │──Parallel Fill\n    │                   │                    │                     │                      │                  │──Merge Gaps\n    │                   │                    │                     │                      │                  │  (k iterations)\n    │                   │                    │<──Update Draft──────│<─Final Report───────────────────────────│\n    │                   │                    │                     │──Compile PDF         │                  │\n    │                   │                    │<──Set Completed─────│                      │                  │\n    │<──[7] Close WS ───│<──Sentinel None────│                     │                      │                  │\n    │                   │                    │                     │                      │                  │\n    │──[8] GET /rep.───>│                    │                     │                      │                  │\n    │<──[9] Return PDF──│                    │                     │                      │                  │\n```\n\n### Detailed Pipeline Steps:\n1. **Submission (`POST /analysis`)**: Client submits market topic, analysis type, geography, research depth (`quick`, `comprehensive`, `deep_research`), objective, and context.\n2. **Storage Initialized**: Storage creates document with status `pending`, generates a 24-character hexadecimal ID, and initializes the in-memory queue.\n3. **Step 1 — Planner Agent**: The `PLANNER_MODEL` generates a structured research plan, questions to answer, required metrics, and search queries.\n4. **Step 2 — Tool Routing (Evidence Gathering)**: A ReAct agent runs tools via MCP to collect web searches, social posts, video transcripts, and scraped articles. Tool outputs are automatically summarized, and every result + extracted URL is **atomically persisted** to storage.\n5. **Step 3 — Evidence Analysis & Synthesis**: The `EVIDENCE_ANALYSIS_MODEL` extracts key quantitative statistics, player market shares, regulatory facts, and links with citation verification.\n6. **Step 4 — Initial Report Drafting**: The `REPORT_WRITING_MODEL` produces the initial comprehensive draft adhering to domain prompt guidelines. The draft is immediately saved to disk/DB.\n7. **Step 5 — LangGraph Map-Reduce Reflection Loop**:\n   - `find_gaps`: The `REPORT_REVIEW_MODEL` analyzes the draft and outputs structured knowledge gaps.\n   - `continue_to_fill_gaps`: Maps each gap into parallel execution branches using LangGraph's `Send` API.\n   - `fill_gaps`: Parallel ReAct agent instances fetch missing metrics and data.\n   - `merge_filled_gaps`: Reduces and synthesizes newly discovered data back into the main report text.\n   - Evaluates iteration limit ($k=2$ for `quick`, $k=3$ for `comprehensive`, $k=5$ for `deep_research`). Repeats if $k$ is not reached.\n8. **PDF Generation & Completion**: `MarkdownPdf` builds a PDF in `reports/{analysis_id}/{type}.pdf`. Storage status is updated to `completed` and the download path is emitted over WebSocket.\n\n---\n\n## 4. Component Breakdown\n\n* **FastAPI Gateway (`server.py`)**:\n  - Handles `/analysis`, `/analysis/{id}`, `/reports/...`, and `/ws/research/{id}`.\n  - Manages asynchronous worker tasks and WebSocket queue piping.\n  - Houses the `PROMPTS_REGISTRY` mapping each `AnalysisType` to domain prompts.\n* **Storage Abstraction Layer (`database/storage.py`)**:\n  - Abstract base class `StorageBackend` with concrete implementations: `LocalFileStorage` and `MongoStorage`.\n  - Thread-safe local file operations using mutex locks.\n  - Persistent query cache (`search_cache`) to prevent redundant tool executions.\n* **Real-Time Memory & Evidence Tracker (`database/memory.py`)**:\n  - `record_tool_evidence`: Persists every search call with timestamp, tool name, query/URL, content preview, and extracted URLs.\n  - `record_draft_report`: Persists intermediate drafts so progress is never lost during long runs.\n* **Agent Factory (`agents/create_agent.py`)**:\n  - Orchestrates the 5-step research pipeline and handles PDF generation and failure states.\n* **LangGraph Engine (`agents/graph.py`)**:\n  - Defines the cyclical StateGraph (`find_gaps` $\\rightarrow$ `fill_gaps` $\\rightarrow$ `merge_filled_gaps` $\\rightarrow$ `route_loop`).\n  - Implements robust JSON parsing with fallback line extraction for gap reflection.\n* **Model Context Protocol Servers (`mcp_servers/`)**:\n  - `google_tools`: Google Search, Shopping, News, and Google Trends via SerpAPI.\n  - `scraper_tools`: Async Crawl4AI scraper converting web pages into clean markdown.\n  - `youtube_tools`: Robust YouTube video ID parsing and transcript extraction.\n  - `x_tools`: X (Twitter) API v2 client with fallback to SERP web indexing.\n* **Prompt Engineering Hub (`prompts/`)**:\n  - Individual prompt suites for Industry, Competitor, Market Gap, Target Market, Barrier, and Sales Forecast reports, plus Planner instructions.\n\n---\n\n## 5. Folder Structure\n\n```\nserver/\n├── server.py                   # FastAPI main entrypoint, REST & WebSocket endpoints\n├── requirements.txt            # Python dependencies (FastAPI, LangGraph, Crawl4AI, etc.)\n├── .env.example                # Example environment variables template\n├── config/\n│   └── settings.py             # Centralized settings (LLM provider, storage, rate limits, keys)\n├── agents/                     # LLM multi-agent core\n│   ├── create_agent.py         # 5-step research pipeline orchestrator & PDF generation\n│   ├── graph.py                # LangGraph StateGraph (map-reduce gap reflection)\n│   ├── state.py                # AgentState schema & custom list reducers with reset\n│   └── utils.py                # LLM model resolution, rate limiting, and backoff retries\n├── database/                   # Data layer & persistence\n│   ├── db.py                   # MongoDB client connection\n│   ├── memory.py               # Real-time search evidence recording & URL extractor\n│   ├── schema.py               # Pydantic schemas (CreateAnalysisRequest, AnalysisSchema, Enums)\n│   └── storage.py              # Dual storage backend: LocalFileStorage & MongoStorage\n├── data/                       # Local file storage directory (when STORAGE_BACKEND=\"local\")\n│   ├── analyses/               # Individual analysis JSON files ({analysis_id}.json)\n│   └── search_cache.json       # Query deduplication cache\n├── mcp_servers/                # Model Context Protocol tier (Port 5001)\n│   ├── main.py                 # Mounts tool servers onto unified SSE endpoints\n│   ├── google_tools/           # Google Search, Trends, News, and Shopping\n│   ├── scraper_tools/          # Crawl4AI web crawler\n│   ├── youtube_tools/          # YouTube transcript retrieval\n│   └── x_tools/                # X (Twitter) search & engagement metrics\n├── prompts/                    # Domain-specific prompt templates\n│   ├── barrier_assessment.py   # Barrier to entry report prompts\n│   ├── competitive_analysis.py # Competitor analysis report prompts\n│   ├── industry.py             # Industry analysis report prompts\n│   ├── market_gap.py           # Market gap report prompts\n│   ├── planner.py              # Master research planner prompt\n│   ├── sales_forecast.py       # Sales forecast & demand projection prompts\n│   └── target_market_segmentation.py # Target market & segmentation prompts\n├── reports/                    # Output directory for generated PDF files\n└── tests/                      # Testing & CLI tools\n    ├── run_terminal_research.py# Terminal research runner with live WebSocket logs\n    ├── test_models.py          # Model testing & validation benchmark\n    ├── test_graph.py           # Graph logic tests\n    └── view_db.py              # Database/storage inspection script\n```\n\n---\n\n## 6. Data Contracts & API Design\n\n### 1. `POST /analysis`\nCreates and kicks off a research task asynchronously.\n\n**Request Payload (`CreateAnalysisRequest`):**\n```json\n{\n  \"market_topic\": \"B2B Remote Collaboration Software\",\n  \"analysis_type\": \"sales_forecasting\",\n  \"geography\": \"Global\",\n  \"research_depth\": \"comprehensive\",\n  \"objective\": {\n    \"type\": \"estimate_future_demand\",\n    \"description\": \"Estimate 3-year market demand and pricing trends\"\n  },\n  \"decision_question\": \"Should we enter the remote team productivity space in 2026?\",\n  \"context\": {\n    \"target_customer\": \"Mid-market tech companies\",\n    \"business_stage\": \"exploring\",\n    \"time_horizon\": \"3 Years\",\n    \"competitors\": [\"Slack\", \"Notion\", \"Miro\"],\n    \"forecast_period\": \"3 Years\"\n  },\n  \"model_name\": null\n}\n```\n\n* **Supported `analysis_type` values**:\n  - `industry_analysis` (`Industry Report`)\n  - `competitor_analysis` (`Competitor Report`)\n  - `market_gap_analysis` (`Market Gap Report`)\n  - `target_market_analysis` (`Target Market Report`)\n  - `barrier_analysis` (`Barrier Report`)\n  - `sales_forecasting` (`Sales Forecast Report`)\n\n* **Supported `research_depth` values**:\n  - `quick`: 2 reflection iterations ($k=2$)\n  - `comprehensive`: 3 reflection iterations ($k=3$, default)\n  - `deep_research`: 5 reflection iterations ($k=5$)\n\n**Response:**\n```json\n{\n  \"id\": \"da52f47236fcaa595a21de1e\",\n  \"status\": \"created\"\n}\n```\n\n### 2. `GET /analysis/{analysis_id}`\nRetrieves full record status, parameters, intermediate drafts, report path, and real-time evidence collected.\n\n### 3. `WS /ws/research/{request_id}`\nStreams real-time agent thoughts, tool invocations, argument payloads, reflection progress, and completion events.\n- Emits `__OUTPUT_FILE__reports/{id}/{file}.pdf` on completion.\n- Emits `__ERROR__{message}` if pipeline encounters an unrecoverable failure.\n\n### 4. `GET /reports/{rid}/{file_id}`\nSafely serves generated PDF reports with built-in path-traversal validation.\n\n---\n\n## 7. Storage Architecture & Evidence Persistence\n\nMarketScout implements a pluggable storage layer via `database/storage.py`.\n\n```\n                  ┌───────────────────────────────┐\n                  │      StorageBackend (ABC)     │\n                  └───────────────┬───────────────┘\n                                  │\n                  ┌───────────────┴───────────────┐\n                  ▼                               ▼\n       ┌────────────────────┐          ┌────────────────────┐\n       │  LocalFileStorage  │          │    MongoStorage    │\n       │(Default, Zero-Dep) │          │  (MongoDB Atlas)   │\n       └────────────────────┘          └────────────────────┘\n```\n\n### 1. `LocalFileStorage` (Default)\n- Stores analyses as formatted JSON documents in `data/analyses/{analysis_id}.json`.\n- Requires **no external database** to run locally.\n- Thread-safe file writes using Python `threading.Lock`.\n- Automatically falls back to `LocalFileStorage` if MongoDB is configured but unreachable.\n\n### 2. Real-Time Evidence Logging (`database/memory.py`)\nEvery tool execution records a `SearchEvidenceItem`:\n```json\n{\n  \"timestamp\": \"Sun Aug 23 13:02:15 2026\",\n  \"stage\": \"initial_research\",\n  \"tool_name\": \"search_google_news\",\n  \"query_or_url\": \"b2b remote collaboration software trends\",\n  \"content_snippet\": \"Market growth projections indicate a 14.2% CAGR...\",\n  \"extracted_urls\": [\n    \"https://www.pcmag.com/picks/the-best-online-collaboration-software\",\n    \"https://www.techradar.com/best/best-cloud-storage\"\n  ]\n}\n```\n* **Why this matters**:\n  - **Zero data loss**: If a rate limit or network timeout occurs later in the run, all research evidence and links collected up to that point remain saved on disk.\n  - **Auditability**: Analysts can inspect raw snippets and verified citation sources.\n  - **Search Caching**: Search results are cached by normalized keys (`{tool_name}:{query}`) in `data/search_cache.json` to prevent repetitive API calls.\n\n---\n\n## 8. Multi-Model Pipeline & Configuration\n\nMarketScout features a multi-model architecture where different steps in the research lifecycle can be assigned to different LLMs based on performance and budget:\n\n| Pipeline Stage | Environment Variable | Default Model | Responsibility |\n|---|---|---|---|\n| **Fallback / Multipurpose** | `MULTIPURPOSE_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | General tasks & defaults |\n| **Planner** | `PLANNER_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | Outlining research strategy & sub-queries |\n| **Tool Routing** | `TOOL_ROUTING_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | ReAct agent tool invocation & evidence gathering |\n| **Tool Summarizer** | `SUMMARIZATION_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | Compressing lengthy scraping outputs |\n| **Evidence Analysis** | `EVIDENCE_ANALYSIS_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | Fact synthesis, numerical extraction & URL checks |\n| **Report Writing** | `REPORT_WRITING_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | Drafting sections & merging gap results |\n| **Report Review (Gaps)**| `REPORT_REVIEW_MODEL` | `gemini-2.0-flash` / `llama3.2:3b` | Critic LLM identifying knowledge gaps |\n\n### Supported Providers:\n1. **Google Gemini**: Set `LLM_PROVIDER=\"google\"` and supply `GOOGLE_API_KEY`. (Default model: `gemini-2.0-flash`).\n2. **Ollama (Local Models)**: Set `LLM_PROVIDER=\"ollama\"`, `OLLAMA_BASE_URL=\"http://localhost:11434\"`.\n   - Recommended local models:\n     - `llama3.2:3b` (Fast, high quality, minimal RAM pressure)\n     - `gemma3:1b` (Ultra-lightweight)\n     - `phi3:mini` (Fast and capable)\n3. **Groq**: Set `LLM_PROVIDER=\"groq\"`, `GROQ_API_KEY=\"...\"`, `GROQ_MODEL=\"llama-3.3-70b-versatile\"`.\n4. **Jan AI**: Set `LLM_PROVIDER=\"jan\"`, `JAN_API_BASE=\"http://localhost:1337/v1\"`.\n5. **Hugging Face**: Set `LLM_PROVIDER=\"huggingface\"`, `HUGGINGFACEHUB_API_TOKEN=\"...\"`.\n\n### Gemini Rate Limiting & Transient Failure Handling\nTo avoid `429 Quota Exceeded` errors when operating on Google Free Tier accounts:\n- **`InMemoryRateLimiter`**: Regulates outgoing requests (configurable via `GOOGLE_RATE_LIMIT_RPS`, default `0.5` rps).\n- **Exponential Backoff**: `call_llm_with_backoff` automatically retries requests on transient failures (up to 7 retries with jitter).\n\n---\n\n## 9. Getting Started & Local Development\n\n### Prerequisites\n- **Python**: 3.10, 3.11, or 3.12.\n- **Node.js**: (Optional) For the frontend UI.\n- **Ollama**: (Optional) If running local models (`brew install ollama && ollama run llama3.2:3b`).\n- **MongoDB**: (Optional) Only required if `STORAGE_BACKEND=\"mongodb\"`.\n\n### 1. Installation\n```bash\n# Navigate to server\ncd server\n\n# Create and activate virtual environment\npython3 -m venv .venv\nsource .venv/bin/activate\n\n# Install dependencies\npip install --upgrade pip\npip install -r requirements.txt\n```\n\n### 2. Environment Configuration\nCopy `.env.example` to `.env` and fill in your keys:\n```bash\ncp .env.example .env\n```\nKey variables to review:\n```env\n# Storage (Default is local JSON in data/)\nSTORAGE_BACKEND=\"local\"\nDATA_DIR=\"data\"\n\n# LLM Provider: 'google', 'ollama', 'groq', 'jan', or 'huggingface'\nLLM_PROVIDER=\"google\"\nGOOGLE_API_KEY=\"your-google-api-key\"\n\n# Search APIs\nSERP_DEV_API_KEY=\"your-serp-api-key\"\n```\n\n### 3. Starting the Services\n\n**Terminal 1 — Run the FastMCP Gateway (Port 5001):**\n```bash\npython -m mcp_servers.main\n```\n\n**Terminal 2 — Run the FastAPI Server (Port 8000):**\n```bash\npython server.py\n```\n\nFastAPI will start at `http://localhost:8000`. You can inspect interactive API documentation at `http://localhost:8000/docs`.\n\n---\n\n## 10. CLI Research Runner & Testing\n\nMarketScout includes built-in terminal utilities to run and test research without needing the web UI.\n\n### Running Research via Terminal\nRun an end-to-end research query with real-time streaming logs:\n```bash\npython tests/run_terminal_research.py \\\n  --topic \"Autonomous Drone Delivery for Groceries\" \\\n  --type sales_forecasting \\\n  --depth quick\n```\nOptions:\n- `--topic`: Any market research question or business topic.\n- `--type`: `sales_forecasting`, `industry_analysis`, `competitor_analysis`, `market_gap_analysis`, `target_market_analysis`, or `barrier_analysis`.\n- `--depth`: `quick` ($k=2$), `comprehensive` ($k=3$), or `deep_research` ($k=5$).\n\n### Testing LLM Models\nValidate your configured LLM models (cloud or local Ollama/Jan models):\n```bash\npython tests/test_models.py\n```\n\n### Inspecting Stored Data\nView saved analyses from local storage or MongoDB:\n```bash\npython tests/view_db.py\n```\n\n---\n\n## 11. Security, Resilience & Production Scaling\n\n1. **Path Traversal Guard**: Endpoint `/reports/{rid}/{file_id}` verifies that file paths cannot traverse out of the designated output directory.\n2. **Atomic Evidence Persistence**: Search results are committed before calling subsequent LLM chains, safeguarding research against memory crashes.\n3. **Structured Logging**: Rich console logging with contextual loggers (`market_scout`, `market_scout.agent`, `market_scout.graph`, `market_scout.storage`).\n4. **Scaling to Production**:\n   - **Task Workers**: Replace in-process `asyncio.create_task` with **Celery / Redis** or **Arq** workers for horizontal scaling.\n   - **Distributed Storage**: Transition `reports/` to an **Amazon S3** or **Google Cloud Storage** bucket.\n   - **Proxy Rotation**: Route Crawl4AI scraping requests through rotating proxies (e.g., Bright Data) to prevent rate limits.\n   - **Redis Pub/Sub**: Stream WebSocket logs across multi-instance backend containers using a Redis Pub/Sub channel.\n\n---\n\n## 12. Interview Explanation Guide\n\nUse this structure to explain MarketScout in technical interviews:\n\n> \"I architected **MarketScout**, an autonomous multi-agent market intelligence platform that automates complex market research workflows—such as competitor analysis, industry reports, and sales forecasting.\n>\n> **The Problem**: Traditional market research is manual, fragmented across web search, social platforms, and video transcripts, and easily prone to LLM hallucinations when not grounded in verified data.\n>\n> **The Architecture**: \n> 1. I built a decoupled system using **FastAPI** and the **Model Context Protocol (MCP)**. The core AI brain connects to independent FastMCP tool microservices running Google Search, Crawl4AI web scrapers, YouTube transcript extractors, and X (Twitter) APIs over Server-Sent Events (SSE).\n> 2. For execution, I designed a multi-stage pipeline: a **Planner Agent** designs a search strategy, a **ReAct Agent** executes tools with real-time atomic evidence and citation logging, an **Evidence Synthesizer** verifies facts and links, and a **Writer Agent** produces the initial report.\n> 3. To guarantee depth and self-correction, I implemented a **LangGraph Map-Reduce loop**: a critic node discovers knowledge gaps, branches out parallel workers using the LangGraph `Send` API to research missing points via MCP, and merges findings back into the report across configurable research depths ($k=2, 3, 5$).\n> 4. I introduced a pluggable storage layer supporting both local file-based JSON storage for zero-dependency local use and MongoDB Atlas for production, combined with multi-provider LLM support across Google Gemini, Ollama local models, and Groq.\"\n"}