animetta

Animetta is an open-source framework for building AI virtual companions and VTubers — characters that talk, listen, remember, emote through a Live2D avatar, and act in the world (chat, livestream, Minecraft). It orchestrates ASR → LLM → TTS → emotion as a single LangGraph state machine, with swappable @ProviderRegistry providers, hybrid memory (Chroma + SQLite FTS5 + Markdown wiki), a Vue 3 / Electron desktop app, and full-chain observability.

Why Animetta? Most "AI VTuber" projects hardcode one provider pipeline. Animetta makes every layer — LLM, ASR, TTS, VAD, memory, tools — a swappable plugin via @ProviderRegistry, with full observability built in.

Contents

• Highlights
• Architecture
• Codebase Tour
• Quick Start
• Core Modules
• Project Structure
• Extending
• Documentation
• Contributing
• Tech Stack
• License

---

✨ Highlights

Animetta is not just another "ChatGPT + TTS" glue. It is an engineered AI companion framework built around three principles: configurable, observable, extensible.

• LangGraph state-graph orchestration — not a linear pipeline, but a directed graph with conditional routing, tool-calling loops, and interrupt/resume.
• Plugin provider architecture — register new vendors via the @ProviderRegistry decorator, zero core-code intrusion.
• Hybrid memory system — Chroma vector search (70%) + SQLite FTS5 keyword match (30%) + Markdown wiki knowledge base.
• Live2D emotion-driven — LLM output → emotion analysis → Live2D parameter mapping; expressions change in real time with the conversation.
• Full-chain observability — OpenTelemetry distributed tracing + Prometheus metrics + built-in Stats Dashboard.

---

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                      Frontend (Vue 3 + Vite)                      │
│               Live2D Renderer · Chat UI · Stats Dashboard         │
└─────────────────────────────┬────────────────────────────────────┘
                              │ Socket.IO / REST
┌─────────────────────────────▼────────────────────────────────────┐
│                WebSocket Server (Starlette + Socket.IO ASGI)      │
│               Session Mgmt · Desktop App · Live2D Events          │
└─────────────────────────────┬────────────────────────────────────┘
                              │
┌─────────────────────────────▼────────────────────────────────────┐
│                   LangGraph Orchestration Engine                   │
│                                                                   │
│  ┌─────────┐   ┌─────────┐   ┌──────────┐   ┌──────────────┐    │
│  │ ASR Node│ → │ Persona │ → │ LLM Node │ → │ Emotion Node │    │
│  │         │   │  Node   │   │  + RAG   │   │ → Live2D Map │    │
│  └─────────┘   └─────────┘   └────┬─────┘   └──────────────┘    │
│                                  │                                │
│                          ┌───────▼───────┐   ┌─────────────┐     │
│                          │  Tool Node   │   │ Output Node │     │
│                          │ MC/MCP/Custom│   │TTS + Memory │     │
│                          └──────────────┘   └─────────────┘     │
└───────────────────────────────────────────────────────────────────┘
                              │
         ┌────────────────────┼────────────────────┐
         ▼                    ▼                    ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│   Services    │   │    Memory     │   │   Tracing     │
│ LLM/ASR/TTS   │   │Chroma+SQLite  │   │ OTel + Stats  │
│ Live2D / VAD  │   │+ Wiki + Meme  │   │+ Prometheus   │
└───────────────┘   └───────────────┘   └───────────────┘

Deeper architecture detail: docs/architecture/overview.md.

Architecture map

The codebase decomposes into 15 layers across four areas (node counts from the architecture knowledge graph):

Backend runtime

Layer	What it covers	Key paths
LangGraph Orchestration (63)	The state-graph engine — nodes, Starlette + Socket.IO ASGI server, prompting sources, routes. The only orchestration mechanism in the project.	src/animetta/orchestration/
Provider Services (129)	Swappable LLM / ASR / TTS / VAD / singing providers following interface → implementation → factory → export with @ProviderRegistry.	src/animetta/services/
Product Tools & Minecraft (142)	Runtime product tools (incl. the Node.js Minecraft adapter) and MCP client integration exposed to the orchestrator.	src/animetta/tools/
Persona & Effective Config (64)	Persona definitions and the EffectiveConfig / registry that resolves runtime configuration.	src/animetta/config/
Memory & Live2D Avatar (32)	Hybrid memory (Chroma vector + SQLite FTS5 + wiki, per ADR-005) and the Live2D avatar / emotion mapping domain.	src/animetta/memory/v2/ · src/animetta/avatar/
Backend Platform Core (70)	Cross-cutting foundations: shared runtime core, observability/tracing, inspection, notifier, utils, acceptance, and host TTS/RVC contracts.	src/animetta/core/ · src/animetta/observability/
Backend Package & External Hosts (10)	Backend package roots and host-side service packages (Qwen TTS, RVC host) that run on the Windows host, not in containers.	src/animetta_qwen_tts/ · src/animetta_rvc_host/

Frontend

Layer	What it covers	Key paths
Frontend Application (138)	Vue 3 + Vite application code: components, views, stores, router, composables, Live2D perf, and feature modules (live streaming, Minecraft gameplay, review, TTS failover).	frontend/src/
Frontend Assets (56)	Static public assets bundled with the desktop app — Live2D models, backgrounds, danmaku test data.	frontend/public/
Frontend Shell & Build (54)	Electron main/preload, sites worker, build/smoke scripts, and Vite/Uno/tsconfig/Electron-builder configuration plus entry HTML.	frontend/electron/ · frontend/scripts/

Configuration & infrastructure

Layer	What it covers	Key paths
Runtime Configuration (30)	Declarative runtime configuration: personas, features, demo data, program scripts, plus root manifests and environment templates.	config/ · .env.example
Infrastructure & CI/CD (22)	Container definitions, Compose topology, GitHub Actions pipelines, and host-side observability stack config.	docker/ · observability/ · .github/workflows/

Developer surface

Layer	What it covers	Key paths
Dev Tooling & Scripts (78)	Quality planner, dev-agent MCP servers, and the runtime lifecycle / operational scripts.	tooling/ · scripts/
Evaluations & Contracts (28)	Evaluation harnesses/fixtures and interface contracts (gamebot, Minecraft).	evaluations/ · contracts/
Project Skills & Docs (17)	In-repo agent skills and top-level documentation.	.agents/skills/ · docs/

---

🗺️ Codebase Tour

A ten-step reading path through the actual code, from boot to deep internals:

1. Project Overview — Start here: this README plus docs/architecture/overview.md for the purpose and shape of the system.
2. Frontend Entry Point — frontend/src/main.ts mounts the Vue 3 app (Vite + Electron); frontend/src/App.vue wires the shell that hosts the Live2D renderer, chat UI, and dashboard.
3. The LangGraph Orchestration Engine — The heart of the backend. orchestrator.py builds the directed state graph with conditional routing and tool-calling loops; state.py defines the shared AgentState that flows ASR → Persona → LLM → Emotion; llm_node.py is where generation happens.
4. Prompts & Persona — prompting/sources.py assembles the persona- and guard-aware system prompt; config/__init__.py loads character definitions and the EffectiveConfig that parameterizes every node.
5. Realtime Server (Starlette + Socket.IO) — The ASGI server bridges frontend and orchestrator: websocket.py manages sessions and streams events; routes.py declares the Socket.IO/REST route handlers.
6. Swappable Provider Services — Every capability is a plugin via @ProviderRegistry. The package roots expose the LLM, TTS, and ASR provider factories (interface → implementation → factory → export): services/llm/ · services/tts/ · services/asr/.
7. Hybrid Memory — Memory v2 (ADR-005): memory/v2/context.py blends Chroma vector search, SQLite FTS5 keyword match, and a Markdown wiki knowledge base for long-term recall.
8. Product Tools — Minecraft — The Minecraft adapter is a Node.js project living inside the Python repo, exposing world actions to the orchestrator as callable tools: tools/minecraft/.
9. Live2D Avatar & Emotion — LLM emotion output is parsed and mapped to Live2D parameters in avatar/performance.py, driving expressions in real time alongside the conversation.
10. Configuration & Tool Registry — Declarative anchors: pyproject.toml pins the Python 3.13 backend; config/tools.yaml registers the product tools the orchestrator may call.

---

🚀 Quick Start

Prerequisites

• Python 3.13 (the toolchain targets 3.13 via ruff/mypy)
• Docker and the separately prepared Windows Qwen host service for the default voice runtime
• Node.js 22 and pnpm 11.7.0 only for local frontend development; Docker builds and verified image deployment do not need them on the host
• Optional inference capabilities use separate host environments; GPU packages do not enter the application image

Start with one command

py -3.13 scripts/bootstrap.py --open

Already configured? This is also available by double-clicking start-anima.cmd. The launcher prepares an isolated lightweight controller; the application environment and frontend build stay inside Docker. The default core keeps character dialogue, Live2D, memory v2 and the existing Qwen voice. First login uses admin / 123456 and requires a new password before accessing the product; subsequent launches preserve the password you choose. --extra bilibili search selects optional image dependencies; capability switches remain in config/animetta.yaml.

First-time configuration

Edit config/animetta.yaml to choose the persona and the complete provider map for the test, smoke, and production profiles. Provider selection lives only in this manifest; environment variables supply profile, endpoints, and secrets.

py -3.13 scripts/bootstrap.py --prepare-only
# Fill the selected LLM and existing Qwen credentials in the generated .env.
py -3.13 scripts/bootstrap.py --open

Use http://localhost/live.html for the character and http://localhost/dashboard for the dashboard. Lifecycle startup reuses a healthy Qwen host and prepares other services only when their capability is enabled.

Local backend development uses requirements.txt. For Web development, install with pnpm --dir frontend --filter animetta-desktop install --frozen-lockfile and start with pnpm --dir frontend dev. Installation scopes, optional capabilities and memory modes are documented in the core deployment guide.

Build from source or deploy a verified image

Build the current checkout when developing or testing local changes:

# Start or reuse the host runtimes, build animetta:local, and start Animetta.
py -3.13 scripts/runtime_lifecycle.py anima-up

Deploy an image that has already passed the main quality gate without rebuilding the application locally:

py -3.13 scripts/runtime_lifecycle.py anima-deploy --image ghcr.io/loiter74/animetta:sha-<40-character-commit>

Use ghcr.io/loiter74/animetta:main for the latest successful main build. Use the full sha-... tag, or the published @sha256:... digest, when the deployment must be reproducible or when rolling back. Private GHCR packages require a prior docker login ghcr.io; see the Docker deployment guide.

Routine py -3.13 scripts/runtime_lifecycle.py anima-down leaves the host Qwen and RVC processes and their loaded models running. Use host-tts-stop or host-rvc-stop only when GPU memory must be released. Neither host runtime is built or managed as a Docker container.

Once healthy, the frontend is served by nginx on port 80 and the backend health endpoint is at http://localhost:12394/health (also proxied at http://localhost/health).

Full deployment guides: Docker · Zeabur

---

🔧 Core Modules

Module	What it does	Docs
LangGraph engine	Directed-graph orchestration with conditional routing, tool loops, interrupt/resume	docs/architecture/overview.md
Provider plugins	Register LLM/ASR/TTS/VAD/Singing vendors via @ProviderRegistry	docs/reference/tools.md
Hybrid memory	Chroma (70%) + SQLite FTS5 (30%) + Markdown wiki + meme learner	ADR-002, ADR-005
Live2D emotion	LLM → emotion tag → Live2D param mapping (6 base emotions)	ADR-009
Minecraft bot	Mineflayer-based bot, decoupled external Voyager runtime	docs/development/minecraft-bot-architecture.md
Observability	OpenTelemetry traces + Prometheus metrics + Stats Dashboard	ADR-006

Supported providers:

Type	Providers
LLM	OpenAI · GLM (Zhipu) · Ollama · DeepSeek · Mock
ASR	OpenAI Whisper · GLM ASR · Mock
TTS (core)	Edge · MiMo · Qwen3 · GPT-SoVITS · Mock
TTS (contrib)	GLM · ChatTTS · Kokoro · VibeVoice
VAD	Silero VAD

Socket.IO event catalog and API reference live in docs/reference/.

---

📁 Project Structure

animetta/
├── src/animetta/          # Python backend (Starlette + LangGraph + Socket.IO)
│   ├── core/              # Entry point + service container
│   ├── orchestration/     # LangGraph state graph + WebSocket server
│   ├── services/          # LLM / ASR / TTS / VAD / Singing / Meme / Live2D
│   ├── memory/            # V2 atom-based memory (Chroma + SQLite FTS5)
│   ├── tools/             # Tool calling + MCP bridge + Minecraft bot
│   ├── avatar/            # Live2D emotion/expression analysis
│   ├── config/            # Pydantic configs + provider registry
│   ├── tracing/           # OpenTelemetry observability
│   ├── notifier/          # Alert channels (Discord, Feishu, Email)
│   ├── inspection/        # Health / telemetry background checks
│   └── acceptance/        # Golden soak state machine
├── frontend/              # Vue 3 + TypeScript + Vite (Electron desktop)
├── config/                # YAML config files (personas, services, tools)
├── design-system/         # Visual design spec (HTML spec sheets)
├── docs/                  # Architecture, ADRs, deployment, references
└── tests/                 # pytest (backend) + vitest (frontend)

---

🧩 Extending

Add a provider (LLM/ASR/TTS/VAD):

# 1. Create a config class
@ProviderRegistry.register("llm", "my_llm")
class MyLLMConfig(BaseLLMConfig):
    api_key: str

# 2. Register the service
@ProviderRegistry.register_service("llm", "my_llm")
class MyLLMAgent(AgentInterface):
    @classmethod
    def from_config(cls, config, **kwargs):
        return cls(api_key=config.api_key)

Add a graph node — follow the node pattern in src/animetta/orchestration/graph/.

Add a tool — use the @tool decorator in src/animetta/tools/.

Agent conventions (for Codex / ZCode): see AGENTS.md.

---

📚 Documentation

Topic	Location
Architecture overview	docs/architecture/overview.md
Architecture Decision Records (11)	docs/adrs/
Backend & Socket.IO API reference	docs/reference/
Testing guide	docs/development/testing.md
Deployment (Docker / Zeabur)	docs/deployment/
Design system	design-system/
Doc navigation index	docs/README.md

---

🤝 Contributing

See CONTRIBUTING.md for development setup, code standards, and test commands.

---

📊 Tech Stack

Layer	Technology
Orchestration	LangGraph · LangChain
Backend	Starlette · Socket.IO ASGI
Frontend	Vue 3 · Vite · TypeScript · Pinia · UnoCSS · pixi.js · Live2D Cubism SDK · Electron
Memory	ChromaDB · SQLite FTS5 · Markdown Wiki
Tracing	OpenTelemetry · Prometheus · Langfuse
AI	OpenAI · Zhipu GLM · DeepSeek · Ollama · Whisper · Qwen3-TTS · GPT-SoVITS
Audio	Demucs · GPT-SoVITS · RVC · yt-dlp
Game	Mineflayer (Node.js)

---

📄 License

MIT License — Copyright (c) 2026 Cowork