live2d-Live-Assistant

Hyori — Live2D Desktop AI Companion

An AI voice companion powered by Google Gemini Live with a Live2D animated character, real-time audio processing, an autonomous self-evolving agent engine (Prime RLM), remote WhatsApp control, and a WebUI dashboard.

Showcase

Visual demonstration of the Hyori floating overlay, high-fidelity audio visualizers, and reactive interface: view on youtube https://youtu.be/dZQYgihwA54

Features

🎙️ Voice Companion

• Real-time voice conversation — mic → Gemini Live → playback pipeline with pitch shifting and barge-in interruptions
• Unlimited talk time on free APIs — proactive context management: tracks real token usage per turn, auto-summarizes the conversation via the cheap task model when approaching limits, and seamlessly recycles the session carrying a memory briefing forward (context_manager.py)
• Live2D character — animated Hiyori model with emotion-reactive expressions ([HAPPY], [SMUG]…), lip-sync, voice-reactive visuals
• Voice cloning — capture your voice and speak as you (Fish Audio)
• Adaptive noise gate + sustained-hit barge-in detection + ALSA self-healing mic recovery
• Live translator mode — speech-to-speech translation between languages
• Offline fallback — local MiniCPM5 stack when API/quota is out

🤖 Prime Agent — self-evolving RLM kernel (engine/)

Inspired by Prime Intellect's prime-agent philosophy:

• Context-as-variables — working data lives inside the kernel's ctx store; the model explores it programmatically (peek/grep/chunk) instead of burning tokens re-reading blobs
• Live persistent Python kernel — state, imports, and functions survive across turns; execution never blocks the voice loop
• Recursive subagents — rlm(...) spawns child agents from kernel code
• RLVR verification — every step checked by grounded verifiers (file exists, syntax, exit code, LLM eval); failures roll back kernel snapshots and refine avoidance heuristics
• Skill synthesis — successful tasks are distilled into reusable Python skills in skills/learned/
• Say "run prime task: …" to trigger it by voice

💬 WhatsApp Remote Control (free, unofficial)

./whatsapp.sh        # one command: installs deps, asks your number once, shows QR

• Chat with Hyori from anywhere — persona brain with rolling memory
• 🎤 Voice note in → text transcript + text reply + voice note reply (free edge-tts → Opus)
• 🖼️ Image analysis, !shot PC screenshots (+ visual analysis), !cmd <shell> remote execution, !status
• Fail-closed security: only numbers listed in [whatsapp] allowed_users get responses; everything binds to 127.0.0.1
• No Meta API key, no per-message cost (Baileys multi-device protocol)

🛠️ 60+ built-in tools

Web search · browser automation (Kimi WebBridge) · screen analysis/vision · shell commands with tiered safety approval · app launching · media control · semantic vector memory ("what's my girlfriend's name?") · fact memory graph · reminders/deadlines/calendar · daily briefings · GitHub repo watching · RSS watchlists · full system administration (packages/services/users/processes) · audit log · MCP server support · coding delegation (Antigravity/OpenCode) with workspace self-healing

🌐 WebUI dashboard (--webui)

Hologram HUD: Live2D view, speech box, music/search/task panels, system dials, token + context usage meters, knowledge graph, persona/background/theme switching, weapon wheel, Ultron 3D interface. Cached page assembly, gzip compression, ETag revalidation.

🔧 Also

Scheduler daemons (system guardian, calendar agent, GitHub sentinel) · ESP32-CAM physical robot body (PhysicalHyori/) · ambient screen awareness (local SmolVLM vision) · curses TUI mode · personas (hyori/baymax/jarvis)

Architecture

┌──────────────────────────────────────────────────────┐
│                       main.py                        │
│   Gemini Live WebSocket (asyncio TaskGroup)          │
│   ├── mic/send/recv/play audio      ← PyAudio       │
│   ├── recv_audio()  tool dispatch + context track   │
│   ├── context_manager.py  token-aware compaction    │
│   └── monitor_system_resources()                    │
│                                                      │
│   Tool Layer: search/shell/browser/memory/media     │
│   Prime Harness: PersistentREPL + verifiers +       │
│                  skill synthesizer + subagents      │
│                                                      │
│   bridge/whatsapp_server.py ←→ whatsapp_bridge/      │
│        (FastAPI brain)         (Node Baileys link)  │
└───────────────┬──────────────────────────────────────┘
                │ UDP :10088/:10090
                ▼
   live2d_gui.py ─ Live2D PIXI overlay  ⇄  WebUI :8080

Quick Start

chmod +x install.sh && ./install.sh
cp .env.example .env          # add GOOGLE_API_KEY / TASK_API_KEY

Run

./run.sh                      # TUI + tools (lightweight)
python main.py --live2d       # Live2D desktop GUI
python main.py --webui        # browser dashboard at http://127.0.0.1:8080
python main.py --webui --network --port 9090   # LAN access, custom port
./whatsapp.sh                 # WhatsApp remote (QR link on first run)

Configuration

config.toml — voice, models, noise gate, ambient vision, physical device, context limits ([context] max_prompt_tokens), WhatsApp allowlist ([whatsapp] allowed_users). Personas live in personas/*.toml.

Privacy: config.toml will contain your phone number after WhatsApp setup — it stays local. .gitignore blocks all databases, session creds, and personal data from being committed.

Key bindings (WebUI)

Project Structure

livepythongemini/
├── main.py                 # Core async loop, Gemini session, tool schemas
├── context_manager.py      # Token tracking + conversation compaction
├── engine/                 # Prime Agent: RLM kernel, verifiers, skills
│   ├── repl_engine.py      #   persistent kernel + ContextStore
│   ├── prime_harness.py    #   plan → execute → verify → evolve
│   ├── subagent.py         #   recursive rlm() agents
│   ├── skill_synthesizer.py#   learned skills in skills/learned/
│   └── verifiers.py        #   RLVR grounded checks
├── bridge/whatsapp_server.py # WhatsApp brain (text/voice/image/!commands)
├── whatsapp_bridge/        # Node Baileys bridge (QR link, send API)
├── webui/server.py         # FastAPI dashboard + WebSocket bridge
├── audio/                  # pipeline, tasks, fillers, voice cloner
├── tools/                  # agent tool implementations
├── scheduler/              # briefing/calendar/github/system daemons
├── vision/                 # screen capture, SmolVLM ambient observer
├── memory/                 # vector store + fact graph
├── personas/               # hyori/baymax/jarvis profiles
├── PhysicalHyori/          # ESP32-CAM robot firmware + UDP bridge
└── config.toml             # runtime configuration

Credits

• Google Gemini Live — realtime AI conversation
• Prime Intellect — prime-agent RLM philosophy inspiration
• Baileys — unofficial WhatsApp protocol library
• edge-tts / gTTS — free voice synthesis
• Live2D Cubism / PIXI.js — avatar rendering
• Kimi WebBridge — browser automation