know-ctl

Social Graph (know-ctl)

Transform saved posts from social media platforms into an interactive, interconnected Obsidian knowledge graph.

---

Overview

Social Graph bridges social content consumption and personal knowledge management (PKM). It turns each saved post into an evidence-backed briefing by combining the original post, linked articles, high-signal discussion claims, questions, and community-shared resources. It then maps those briefings into a searchable topic and similarity graph and exports them to Obsidian.

While LinkedIn is the initial production connector, Social Graph is engineered as a universal social media knowledge aggregator. Notes are structured in platform-specific subtrees (linkedin/, reddit/, x/, substack/) with shared cross-platform topic graphs and semantic similarity links. Reddit integration is next on the roadmap, followed by X (Twitter) and Substack.

The project includes a web application with an onboarding wizard, interactive settings dashboard, encrypted credential store, live pipeline runner terminal, and a D3 force-directed knowledge graph. No .env file or manual configuration required.

---

Features

• Web Dashboard: Browser interface powered by FastAPI. Configure API keys, upload data files, inspect topics, authors, and graph clusters directly in your browser.
• Encrypted Local Storage: Machine-derived Fernet AES encryption stores credentials securely in local SQLite (.socialgraph/socialgraph.db).
• Multi-Platform Architecture: Platform-partitioned knowledge storage (linkedin/, reddit/, x/) unified by cross-platform topic taxonomies and semantic search.
• Hybrid LLM Pipeline:
  • Local 27B Reasoning on 8GB VRAM (Bonsai 2): Run full 27B reasoning models locally on consumer laptops (RTX 3070/4060, Apple Silicon) with ~5.6 GB ternary weights (PTQ1_0), 24k context window, and FlashAttention via PrismML's llama-server. Also supports standard OpenAI-compatible servers (vLLM, llama.cpp, Ollama, LM Studio).
  • Groq API: Cross-source reasoning over the post, article, and thread to produce structured briefings with a thesis, takeaways, community claims, resources, and open questions. Groq also provides a functional fallback when no local model is configured.
• Dual Ingestion Modes: Upload official data archive JSON exports or live-scrape posts and comments using Playwright (credentials or session cookie).
• Comment Intelligence: Scans up to 80 comments by default, expands long bodies and replies, scores information value, and filters applause or promotional noise before synthesis.
• Deep URL and Comment Enrichment: Crawls links from both posts and useful comments with trafilatura, then summarizes the extracted article text.
• Freshness-Aware Briefings: Evidence hashes automatically invalidate a briefing when its post, article, or useful thread context changes.
• Local Vector Embeddings: Generates embeddings locally using sentence-transformers with vectorized matrix similarity calculations.
• Obsidian Vault Synthesis: Writes clean Markdown files with YAML frontmatter, bidirectional wikilinks ([[post_...]]), clickable topic/subtopic Maps of Content (MOCs), author profiles, and a combined newest-first post timeline.
• Model Context Protocol (MCP): Query structured briefings directly with get_briefing, alongside semantic search, topic, author, graph, and timeline tools.

---

Platform Support & Roadmap

Platform	Ingestion Method	Status	Target Path
LinkedIn	JSON archive export or live Playwright scraping	Supported (v0.1)	vault/linkedin/
Reddit	Saved posts & saved comments (JSON / Reddit API)	Up Next (v0.2)	vault/reddit/
X (Twitter)	Bookmarks export & thread scraping	Planned	vault/x/
Substack	Reading list & saved newsletters	Planned	vault/substack/

---

Quick Start (Web UI)

No .env file required. You can configure and run everything from the browser.

1. Clone and Install

# Clone the repository
git clone https://github.com/akashrai2003/know-ctl.git
cd know-ctl

# Create virtual environment (Python >= 3.10)
python3 -m venv venv
source venv/bin/activate

# Install package
pip install -e .

# Or install with dev/test dependencies
pip install -e ".[dev]"

2. Initialize and Launch

# Initialize local database tables and workspace folders
sg init

# Launch the Web Dashboard
sg web

Open http://localhost:8080 in your browser.

3. Onboarding Wizard

On first launch, an interactive wizard will guide you through:

1. Welcome: Overview of features.
2. Groq API Key: Enter your free API key from console.groq.com with an instant inline connectivity test.
3. Local Model Server (optional): Connect vLLM, llama.cpp, or Ollama (supports auto-batch detection).
4. LinkedIn Credentials (optional): Email/password or session cookie (li_at) for live scraping.
5. Ready: Upload your archive or trigger the pipeline.

4. Upload Data Export

In the Settings tab, upload your saved posts JSON export:

LinkedIn -> Settings & Privacy -> Data Privacy -> Get a copy of your data -> select "Saved items".

5. Run the Pipeline

Head to the Pipeline page and click Run Pipeline. Follow real-time execution in the embedded terminal stream.

---

CLI Setup (.env)

For headless servers or automation scripts, you can supply credentials via environment variables or a .env file:

cp .env.example .env
# Edit .env with your Groq API key and optional endpoints
sg init
sg run

For an existing installation, apply the latest schema before running the new intelligence stages:

alembic upgrade head

Values configured via the Web UI are stored in SQLite and take precedence over .env defaults.

---

Architecture

Saved posts
  -> ingest
  -> collect comments and replies
  -> rank useful claims / questions / resources
  -> fetch links from the post and useful comments
  -> classify with the combined context
  -> embed and organize into subtopics
  -> synthesize an evidence-backed AI briefing
  -> build semantic and topic graph edges
  -> write Obsidian notes, topic MOCs, and author pages

LLM Task Matrix

Component	Responsibility	Recommended Model	Mode
Local Model (27B)	Article summaries, comment ranking, classification, title/subtopic generation, local briefings	Bonsai 2 27B (Ternary-Bonsai-2-27B-PTQ1_0) — Runs on 8GB VRAM or Qwen/Qwen3.5-9B-FP8	Local (CUDA / Metal / CPU)
Groq API	Cloud multi-source briefing synthesis and fast fallbacks	llama-3.3-70b-versatile / qwen/qwen3.8-27b	Cloud API
Local Embeddings	Vector similarity & cosine graph edges	Qwen/Qwen3-Embedding-0.6B	Local (CUDA/CPU)

---

💻 Local 27B AI on 8GB VRAM (Bonsai 2 Integration)

Social Graph natively supports running 27-billion parameter reasoning models 100% locally on standard 8GB VRAM consumer laptops and GPUs via Bonsai 2.

By using ternary-weight quantization (PTQ1_0 at 1.75 bits per weight), the entire 27B model weighs only 5.6 GB while retaining 98.2% of FP16 intelligence. Combined with FlashAttention and 8-bit quantized KV caching, you can run a 24,576 token context window completely inside 8GB VRAM with zero cloud API dependencies.

1. Launch Bonsai 2 Local Server

Clone the Bonsai-demo repository and run its setup:

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./setup.sh

Launch the CUDA llama-server (tuned for 8GB VRAM):

cd ~/Desktop/Bonsai-demo

./bin/cuda/llama-server \
  -m models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  -ngl 99 \
  -c 24576 \
  -fa on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --reasoning-effort medium \
  --reasoning-format auto \
  --host 0.0.0.0 \
  --port 8080

2. Connect Social Graph

In .env (or via Web UI -> Settings -> Local Model Server):

SG_VLLM_BASE_URL=http://127.0.0.1:8080
SG_VLLM_MODEL=models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf
# Avoid port collision with llama-server on 8080:
SG_WEB_PORT=3000

3. Run Pipeline with 100% Local Inference

# Run local post categorization
sg classify --force

# Generate granular subtopics
sg subtopic --force

# Synthesize full AI briefings locally (no cloud API needed!)
sg insights --provider local

# Launch Web UI on port 3000
sg web --port 3000

📖 Full Guide: See BONSAI_SETUP.md for detailed memory budgets, Apple Silicon instructions, and performance optimization tips.

---

CLI Reference

Social Graph provides the sg command-line interface:

# Core and Status
sg init                          # Initialize database schema and workspace
sg status                        # Display post counts by stage and daemon status

# Pipeline Execution
sg run                           # Run full pipeline end-to-end
sg run --from classify           # Resume starting from a specific stage
sg run --stage enrich            # Execute only one specific stage
sg run --live                    # Use live Playwright scraper instead of JSON

# Individual Stages
sg ingest --json <file>          # Ingest posts from a JSON archive
sg scrape                        # Launch browser to scrape saved posts live
sg comments                      # Fetch post comments and discussion threads
sg rank-comments                 # Score comments and remove applause/promo noise
sg enrich                        # Crawl and summarize linked URLs
sg comment-enrich                # Fetch and summarize links inside comments
sg classify                      # Categorize posts according to topic taxonomy
sg classify --force --local-model <served-model-id>  # Safely replace all classifications
sg embed                         # Compute vector embeddings for all posts
sg subtopic                      # Detect granular subtopics per category
sg subtopic --force --local-model <served-model-id>  # Rebuild primary-topic hierarchy
sg insights                      # Generate missing/stale evidence-backed briefings
sg insights --provider local     # Generate briefings entirely with the local model
sg insights --provider local --local-model <served-model-id>  # Select Gemma/Qwen per run
sg brief <urn>                   # Regenerate one briefing and one Obsidian note
sg brief <urn> --fetch-comments  # Refresh its thread before briefing
sg semantic-edges                # Build cosine similarity graph edges
sg build-graph                   # Cluster communities using Leiden algorithm
sg vault-write                   # Generate Markdown files in Obsidian vault

# Discovery and Analytics
sg search "vector databases"     # Semantic search over your posts
sg similar "urn:li:activity:..." # Find semantically related posts
sg graph stats                   # Inspect topic and author distributions
sg graph co-occurrence           # Analyze topic co-occurrence patterns
sg graph timeline                # Monthly post activity breakdown

# Background Daemon and Web Server
sg schedule start                # Start background scheduler (syncs every 6h)
sg schedule status               # Check background daemon health
sg schedule stop                 # Stop background scheduler
sg web                           # Launch web dashboard (http://localhost:8080)
sg web --no-open --port 3000     # Run web dashboard on custom port headlessly
sg mcp-serve                     # Start Model Context Protocol server (stdio/http)

---

Configuration Reference

Settings can be managed in Web UI -> Settings or set in .env:

Setting	Env Variable	Default	Description
Groq API Key	SG_GROQ_API_KEY	""	Required for briefing synthesis
Groq Model	SG_GROQ_MODEL	llama-3.3-70b-versatile	Primary reasoning model
Local Model URL	SG_VLLM_BASE_URL	""	Local OpenAI-compatible server URL (e.g. http://127.0.0.1:8080)
Local Model ID	SG_VLLM_MODEL	Qwen/Qwen3.5-9B-FP8	Model ID (e.g. models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf)
Embedding Model	SG_EMBEDDING_MODEL	Qwen/Qwen3-Embedding-0.6B	Sentence-transformers model
Embedding Device	SG_EMBEDDING_DEVICE	cuda	cuda, cpu, or mps
LinkedIn Email	SG_LINKEDIN_EMAIL	""	Account email for live scraping
LinkedIn Password	SG_LINKEDIN_PASSWORD	""	Account password for live scraping
LinkedIn Cookie	SG_LINKEDIN_COOKIE	""	li_at session cookie for MFA bypass
Batch Size	SG_BATCH_SIZE	10	Concurrency batch size for LLM calls
Max Comments	SG_MAX_COMMENTS	80	Maximum thread comments scanned per post; only high-signal comments are kept in briefings
Obsidian Vault	SG_OBSIDIAN_VAULT_PATH	./vault	Output directory for Markdown notes
Web Port	SG_WEB_PORT	8080	Local dashboard server port

---

Repository Structure

know-ctl/
├── socialgraph/
│   ├── agents/          # Pipeline agents (Ingest, Classify, Embed, VaultWrite, etc.)
│   ├── cli/             # Typer CLI subcommands (pipeline, server, schedule, graph)
│   ├── config/          # Pydantic Settings and configuration loader
│   ├── connectors/      # Playwright browser automation and platform connectors
│   ├── knowledge/       # Obsidian formatting, graph layout, semantic search
│   ├── llm/             # Hybrid client, Groq client, prompt templates, factory
│   ├── storage/         # SQLAlchemy models, SQLite migrations, encrypted config store
│   └── web/             # FastAPI app, SSE pipeline runner, static assets (HTML/CSS/JS)
├── tests/               # Pytest suite (unit, API, integration)
├── vault/               # Generated Obsidian knowledge vault (git-ignored)
├── .socialgraph/        # Application database and logs (git-ignored)
├── BONSAI_SETUP.md      # Running 27B local LLM on 8GB VRAM laptops
├── SETUP.md             # In-depth credential and cookie setup guide
├── RUNBOOK.md           # Operational commands and execution guide
└── pyproject.toml       # Project metadata, dependencies, and tools

---

Contributing

Contributions are welcome. Please check out CONTRIBUTING.md for guidelines on code formatting (ruff format), linting (ruff check), and running the test suite (pytest).

License

This project is licensed under the MIT License.