{"id":"c9d67095-5b8a-4bbb-86d2-d71aa8687791","slug":"know-ctl","kind":"mcp","name":"know-ctl","summary":"Transform LinkedIn saved posts into an interactive, interconnected Obsidian knowledge graph. Features a zero-config web dashboard, hybrid LLM routing (Groq + local vLLM/llama.cpp), vector embeddings,","owner":"akashrai2003","repoUrl":"https://github.com/akashrai2003/know-ctl","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-05-14T18:13:01.000Z","addedAt":"2026-10-29T05:00:16.422Z","logoUrl":"https://avatars.githubusercontent.com/u/93964391?u=2eb8c6523f9055f2fb4616420c2bd09c15e802a9&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["files-documents","memory-knowledge","search"],"tags":["fastapi","knowledge-graph","obsidian","python","semantic-search","vllm"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-29T05:00:46.695Z","sample":false,"url":"https://conduits.dev/tools/know-ctl","language":"Python","transports":["stdio"],"install":{"pypi":"social-graph"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Social Graph (know-ctl)\n\nTransform saved posts from social media platforms into an interactive, interconnected Obsidian knowledge graph.\n\n[![CI](https://github.com/akashrai2003/know-ctl/actions/workflows/ci.yml/badge.svg)](https://github.com/akashrai2003/know-ctl/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python Version](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/downloads/)\n\n---\n\n## Overview\n\nSocial Graph bridges social content consumption and personal knowledge management (PKM). It turns each saved post into an evidence-backed briefing by combining the original post, linked articles, high-signal discussion claims, questions, and community-shared resources. It then maps those briefings into a searchable topic and similarity graph and exports them to Obsidian.\n\nWhile LinkedIn is the initial production connector, Social Graph is engineered as a universal social media knowledge aggregator. Notes are structured in platform-specific subtrees (`linkedin/`, `reddit/`, `x/`, `substack/`) with shared cross-platform topic graphs and semantic similarity links. **Reddit integration is next on the roadmap**, followed by X (Twitter) and Substack.\n\nThe project includes a web application with an onboarding wizard, interactive settings dashboard, encrypted credential store, live pipeline runner terminal, and a D3 force-directed knowledge graph. No `.env` file or manual configuration required.\n\n---\n\n## Features\n\n- **Web Dashboard**: Browser interface powered by FastAPI. Configure API keys, upload data files, inspect topics, authors, and graph clusters directly in your browser.\n- **Encrypted Local Storage**: Machine-derived Fernet AES encryption stores credentials securely in local SQLite (`.socialgraph/socialgraph.db`).\n- **Multi-Platform Architecture**: Platform-partitioned knowledge storage (`linkedin/`, `reddit/`, `x/`) unified by cross-platform topic taxonomies and semantic search.\n- **Hybrid LLM Pipeline**:\n  - **Local 27B Reasoning on 8GB VRAM (Bonsai 2)**: Run full 27B reasoning models locally on consumer laptops (RTX 3070/4060, Apple Silicon) with ~5.6 GB ternary weights (`PTQ1_0`), 24k context window, and FlashAttention via PrismML's `llama-server`. Also supports standard OpenAI-compatible servers (vLLM, llama.cpp, Ollama, LM Studio).\n  - **Groq API**: Cross-source reasoning over the post, article, and thread to produce structured briefings with a thesis, takeaways, community claims, resources, and open questions. Groq also provides a functional fallback when no local model is configured.\n- **Dual Ingestion Modes**: Upload official data archive JSON exports or live-scrape posts and comments using Playwright (credentials or session cookie).\n- **Comment Intelligence**: Scans up to 80 comments by default, expands long bodies and replies, scores information value, and filters applause or promotional noise before synthesis.\n- **Deep URL and Comment Enrichment**: Crawls links from both posts and useful comments with `trafilatura`, then summarizes the extracted article text.\n- **Freshness-Aware Briefings**: Evidence hashes automatically invalidate a briefing when its post, article, or useful thread context changes.\n- **Local Vector Embeddings**: Generates embeddings locally using `sentence-transformers` with vectorized matrix similarity calculations.\n- **Obsidian Vault Synthesis**: Writes clean Markdown files with YAML frontmatter, bidirectional wikilinks (`[[post_...]]`), clickable topic/subtopic Maps of Content (MOCs), author profiles, and a combined newest-first post timeline.\n- **Model Context Protocol (MCP)**: Query structured briefings directly with `get_briefing`, alongside semantic search, topic, author, graph, and timeline tools.\n\n---\n\n## Platform Support & Roadmap\n\n| Platform | Ingestion Method | Status | Target Path |\n|:---|:---|:---|:---|\n| **LinkedIn** | JSON archive export or live Playwright scraping | Supported (v0.1) | `vault/linkedin/` |\n| **Reddit** | Saved posts & saved comments (JSON / Reddit API) | Up Next (v0.2) | `vault/reddit/` |\n| **X (Twitter)** | Bookmarks export & thread scraping | Planned | `vault/x/` |\n| **Substack** | Reading list & saved newsletters | Planned | `vault/substack/` |\n\n---\n\n## Quick Start (Web UI)\n\nNo `.env` file required. You can configure and run everything from the browser.\n\n### 1. Clone and Install\n\n```bash\n# Clone the repository\ngit clone https://github.com/akashrai2003/know-ctl.git\ncd know-ctl\n\n# Create virtual environment (Python >= 3.10)\npython3 -m venv venv\nsource venv/bin/activate\n\n# Install package\npip install -e .\n\n# Or install with dev/test dependencies\npip install -e \".[dev]\"\n```\n\n### 2. Initialize and Launch\n\n```bash\n# Initialize local database tables and workspace folders\nsg init\n\n# Launch the Web Dashboard\nsg web\n```\n\nOpen `http://localhost:8080` in your browser.\n\n### 3. Onboarding Wizard\n\nOn first launch, an interactive wizard will guide you through:\n1. **Welcome**: Overview of features.\n2. **Groq API Key**: Enter your free API key from [console.groq.com](https://console.groq.com) with an instant inline connectivity test.\n3. **Local Model Server** (optional): Connect vLLM, llama.cpp, or Ollama (supports auto-batch detection).\n4. **LinkedIn Credentials** (optional): Email/password or session cookie (`li_at`) for live scraping.\n5. **Ready**: Upload your archive or trigger the pipeline.\n\n### 4. Upload Data Export\n\nIn the **Settings** tab, upload your saved posts JSON export:\n> LinkedIn -> Settings & Privacy -> Data Privacy -> Get a copy of your data -> select \"Saved items\".\n\n### 5. Run the Pipeline\n\nHead to the **Pipeline** page and click **Run Pipeline**. Follow real-time execution in the embedded terminal stream.\n\n---\n\n## CLI Setup (.env)\n\nFor headless servers or automation scripts, you can supply credentials via environment variables or a `.env` file:\n\n```bash\ncp .env.example .env\n# Edit .env with your Groq API key and optional endpoints\nsg init\nsg run\n```\n\nFor an existing installation, apply the latest schema before running the new intelligence stages:\n\n```bash\nalembic upgrade head\n```\n\nValues configured via the Web UI are stored in SQLite and take precedence over `.env` defaults.\n\n---\n\n## Architecture\n\n```text\nSaved posts\n  -> ingest\n  -> collect comments and replies\n  -> rank useful claims / questions / resources\n  -> fetch links from the post and useful comments\n  -> classify with the combined context\n  -> embed and organize into subtopics\n  -> synthesize an evidence-backed AI briefing\n  -> build semantic and topic graph edges\n  -> write Obsidian notes, topic MOCs, and author pages\n```\n\n### LLM Task Matrix\n\n| Component | Responsibility | Recommended Model | Mode |\n|:---|:---|:---|:---|\n| **Local Model (27B)** | Article summaries, comment ranking, classification, title/subtopic generation, local briefings | **Bonsai 2 27B** (`Ternary-Bonsai-2-27B-PTQ1_0`) — *Runs on 8GB VRAM* or `Qwen/Qwen3.5-9B-FP8` | Local (CUDA / Metal / CPU) |\n| **Groq API** | Cloud multi-source briefing synthesis and fast fallbacks | `llama-3.3-70b-versatile` / `qwen/qwen3.8-27b` | Cloud API |\n| **Local Embeddings** | Vector similarity & cosine graph edges | `Qwen/Qwen3-Embedding-0.6B` | Local (CUDA/CPU) |\n\n---\n\n## 💻 Local 27B AI on 8GB VRAM (Bonsai 2 Integration)\n\nSocial Graph natively supports running **27-billion parameter reasoning models 100% locally on standard 8GB VRAM consumer laptops and GPUs** via [Bonsai 2](https://github.com/PrismML-Eng/Bonsai-demo).\n\nBy using ternary-weight quantization (`PTQ1_0` at 1.75 bits per weight), the entire 27B model weighs only **5.6 GB** while retaining **98.2% of FP16 intelligence**. Combined with FlashAttention and 8-bit quantized KV caching, you can run a **24,576 token context window** completely inside 8GB VRAM with zero cloud API dependencies.\n\n### 1. Launch Bonsai 2 Local Server\n\nClone the [Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) repository and run its setup:\n\n```bash\ngit clone https://github.com/PrismML-Eng/Bonsai-demo.git\ncd Bonsai-demo\n./setup.sh\n```\n\nLaunch the CUDA `llama-server` (tuned for 8GB VRAM):\n\n```bash\ncd ~/Desktop/Bonsai-demo\n\n./bin/cuda/llama-server \\\n  -m models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf \\\n  -ngl 99 \\\n  -c 24576 \\\n  -fa on \\\n  --cache-type-k q8_0 \\\n  --cache-type-v q8_0 \\\n  --jinja \\\n  --reasoning-effort medium \\\n  --reasoning-format auto \\\n  --host 0.0.0.0 \\\n  --port 8080\n```\n\n### 2. Connect Social Graph\n\nIn `.env` (or via **Web UI -> Settings -> Local Model Server**):\n\n```env\nSG_VLLM_BASE_URL=http://127.0.0.1:8080\nSG_VLLM_MODEL=models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf\n# Avoid port collision with llama-server on 8080:\nSG_WEB_PORT=3000\n```\n\n### 3. Run Pipeline with 100% Local Inference\n\n```bash\n# Run local post categorization\nsg classify --force\n\n# Generate granular subtopics\nsg subtopic --force\n\n# Synthesize full AI briefings locally (no cloud API needed!)\nsg insights --provider local\n\n# Launch Web UI on port 3000\nsg web --port 3000\n```\n\n> 📖 **Full Guide**: See [BONSAI_SETUP.md](BONSAI_SETUP.md) for detailed memory budgets, Apple Silicon instructions, and performance optimization tips.\n\n---\n\n## CLI Reference\n\nSocial Graph provides the `sg` command-line interface:\n\n```bash\n# Core and Status\nsg init                          # Initialize database schema and workspace\nsg status                        # Display post counts by stage and daemon status\n\n# Pipeline Execution\nsg run                           # Run full pipeline end-to-end\nsg run --from classify           # Resume starting from a specific stage\nsg run --stage enrich            # Execute only one specific stage\nsg run --live                    # Use live Playwright scraper instead of JSON\n\n# Individual Stages\nsg ingest --json <file>          # Ingest posts from a JSON archive\nsg scrape                        # Launch browser to scrape saved posts live\nsg comments                      # Fetch post comments and discussion threads\nsg rank-comments                 # Score comments and remove applause/promo noise\nsg enrich                        # Crawl and summarize linked URLs\nsg comment-enrich                # Fetch and summarize links inside comments\nsg classify                      # Categorize posts according to topic taxonomy\nsg classify --force --local-model <served-model-id>  # Safely replace all classifications\nsg embed                         # Compute vector embeddings for all posts\nsg subtopic                      # Detect granular subtopics per category\nsg subtopic --force --local-model <served-model-id>  # Rebuild primary-topic hierarchy\nsg insights                      # Generate missing/stale evidence-backed briefings\nsg insights --provider local     # Generate briefings entirely with the local model\nsg insights --provider local --local-model <served-model-id>  # Select Gemma/Qwen per run\nsg brief <urn>                   # Regenerate one briefing and one Obsidian note\nsg brief <urn> --fetch-comments  # Refresh its thread before briefing\nsg semantic-edges                # Build cosine similarity graph edges\nsg build-graph                   # Cluster communities using Leiden algorithm\nsg vault-write                   # Generate Markdown files in Obsidian vault\n\n# Discovery and Analytics\nsg search \"vector databases\"     # Semantic search over your posts\nsg similar \"urn:li:activity:...\" # Find semantically related posts\nsg graph stats                   # Inspect topic and author distributions\nsg graph co-occurrence           # Analyze topic co-occurrence patterns\nsg graph timeline                # Monthly post activity breakdown\n\n# Background Daemon and Web Server\nsg schedule start                # Start background scheduler (syncs every 6h)\nsg schedule status               # Check background daemon health\nsg schedule stop                 # Stop background scheduler\nsg web                           # Launch web dashboard (http://localhost:8080)\nsg web --no-open --port 3000     # Run web dashboard on custom port headlessly\nsg mcp-serve                     # Start Model Context Protocol server (stdio/http)\n```\n\n---\n\n## Configuration Reference\n\nSettings can be managed in **Web UI -> Settings** or set in `.env`:\n\n| Setting | Env Variable | Default | Description |\n|:---|:---|:---|:---|\n| **Groq API Key** | `SG_GROQ_API_KEY` | `\"\"` | Required for briefing synthesis |\n| **Groq Model** | `SG_GROQ_MODEL` | `llama-3.3-70b-versatile` | Primary reasoning model |\n| **Local Model URL** | `SG_VLLM_BASE_URL` | `\"\"` | Local OpenAI-compatible server URL (e.g. `http://127.0.0.1:8080`) |\n| **Local Model ID** | `SG_VLLM_MODEL` | `Qwen/Qwen3.5-9B-FP8` | Model ID (e.g. `models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf`) |\n| **Embedding Model** | `SG_EMBEDDING_MODEL` | `Qwen/Qwen3-Embedding-0.6B` | Sentence-transformers model |\n| **Embedding Device** | `SG_EMBEDDING_DEVICE` | `cuda` | `cuda`, `cpu`, or `mps` |\n| **LinkedIn Email** | `SG_LINKEDIN_EMAIL` | `\"\"` | Account email for live scraping |\n| **LinkedIn Password** | `SG_LINKEDIN_PASSWORD` | `\"\"` | Account password for live scraping |\n| **LinkedIn Cookie** | `SG_LINKEDIN_COOKIE` | `\"\"` | `li_at` session cookie for MFA bypass |\n| **Batch Size** | `SG_BATCH_SIZE` | `10` | Concurrency batch size for LLM calls |\n| **Max Comments** | `SG_MAX_COMMENTS` | `80` | Maximum thread comments scanned per post; only high-signal comments are kept in briefings |\n| **Obsidian Vault** | `SG_OBSIDIAN_VAULT_PATH` | `./vault` | Output directory for Markdown notes |\n| **Web Port** | `SG_WEB_PORT` | `8080` | Local dashboard server port |\n\n---\n\n## Repository Structure\n\n```text\nknow-ctl/\n├── socialgraph/\n│   ├── agents/          # Pipeline agents (Ingest, Classify, Embed, VaultWrite, etc.)\n│   ├── cli/             # Typer CLI subcommands (pipeline, server, schedule, graph)\n│   ├── config/          # Pydantic Settings and configuration loader\n│   ├── connectors/      # Playwright browser automation and platform connectors\n│   ├── knowledge/       # Obsidian formatting, graph layout, semantic search\n│   ├── llm/             # Hybrid client, Groq client, prompt templates, factory\n│   ├── storage/         # SQLAlchemy models, SQLite migrations, encrypted config store\n│   └── web/             # FastAPI app, SSE pipeline runner, static assets (HTML/CSS/JS)\n├── tests/               # Pytest suite (unit, API, integration)\n├── vault/               # Generated Obsidian knowledge vault (git-ignored)\n├── .socialgraph/        # Application database and logs (git-ignored)\n├── BONSAI_SETUP.md      # Running 27B local LLM on 8GB VRAM laptops\n├── SETUP.md             # In-depth credential and cookie setup guide\n├── RUNBOOK.md           # Operational commands and execution guide\n└── pyproject.toml       # Project metadata, dependencies, and tools\n```\n\n---\n\n## Contributing\n\nContributions are welcome. Please check out [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on code formatting (`ruff format`), linting (`ruff check`), and running the test suite (`pytest`).\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n"}