audio-mcp

---

A Model Context Protocol server that exposes the three.ws voice and animation pipeline over stdio. Make an avatar speak, let users talk back, turn that speech into a per-frame lipsync track, and browse the recorded motion-capture library — everything you need to drive a 3D agent's mouth and movement.

The speech, recognition, and lipsync lanes run on free NVIDIA NIM models (Magpie TTS, Riva ASR, Audio2Face-3D), with OpenAI as the paid TTS backstop — the same provider policy as the three.ws platform. No key required to start; point THREE_WS_BASE at a deployment and go.

Install

npm install @three-ws/audio-mcp

Or run with npx (no install):

npx @three-ws/audio-mcp

Quick start

Claude Code, one line:

claude mcp add audio -- npx -y @three-ws/audio-mcp

Claude Desktop / Cursor (claude_desktop_config.json or mcp.json):

{
	"mcpServers": {
		"audio": {
			"command": "npx",
			"args": ["-y", "@three-ws/audio-mcp"]
		}
	}
}

Inspect the surface with the MCP Inspector:

npx -y @modelcontextprotocol/inspector npx @three-ws/audio-mcp

Tools

Tool	Type	What it does
text_to_speech	read-only	Synthesize spoken audio from text; returns a base64 audio data URL plus the real voice/format/mime. Free Magpie, OpenAI backstop.
speech_to_text	read-only	Transcribe spoken audio (base64 wav/pcm/flac/ogg) with the free NVIDIA Riva recognizer; returns text, confidence, duration.
audio_to_face	read-only	Turn speech (text you pass, or audio you have) into a per-frame ARKit blendshape lipsync track via Audio2Face-3D.
motion_capture_clips	read-only	Browse the motion-capture clip library — recorded face/pose/hand/VMC tracks — with kind filter and cursor pagination.
motion_capture_clip	read-only	Fetch one clip by id including its full per-frame animation track to play back on an avatar.

All five tools are read-only: they transform input (synthesize/recognize/animate) or read live library data, never mutating platform state. None are idempotent — the audio bytes, transcript, and catalog move between calls.

Input parameters

text_to_speech — text (required, 1–4096 chars), voice (default nova), format (mp3|opus|aac|flac|wav|pcm, default mp3), model (OpenAI backstop only), language (BCP-47, default en-US), speed (0.5–2.0, OpenAI lane only).

speech_to_text — audio (required, base64 or data: URL), format (wav|pcm|flac|ogg, default wav), language (BCP-47, default en-US), sampleRate (Hz, PCM only), words (bool), model (advanced).

audio_to_face — text or audio (one required), format (wav|pcm, audio path), sampleRate (PCM only), voice / language (text path).

motion_capture_clips — kind (face|pose|hand|vmc), include_public (bool, authenticated only), limit (1–100, default 50), cursor.

motion_capture_clip — id (required, clip UUID).

Examples

// text_to_speech
> { "text": "Hello, I'm your three.ws agent.", "voice": "nova" }
{
  "ok": true,
  "voice": "Magpie-Multilingual.EN-US.Aria",
  "model": "magpie-tts-multilingual",
  "format": "wav",
  "mime": "audio/wav",
  "sizeBytes": 274476,
  "audio": "data:audio/wav;base64,UklGR…"
}

// audio_to_face (text path — returns audio + the lipsync track together)
> { "text": "Watch my lips move." }
{
  "ok": true,
  "audio": { "base64": "UklGR…", "contentType": "audio/wav", "format": "wav" },
  "animation": {
    "fps": 30,
    "blendShapeNames": ["jawOpen", "mouthClose", "…"],
    "durationSec": 1.9,
    "frames": [ { "t": 0.0, "w": [0.0, 0.1, /* … */] } ]
  }
}

A face frame is { t, w }: t = seconds from clip start, w = weights in the order of blendShapeNames (ARKit-52 naming). Play the audio and sample the track by the audio element's currentTime.

Notes on payloads

Audio crosses the wire as base64 (TTS returns it, ASR/A2F accept it). Both the synthesis input (text) and the recognition input (audio, ~8 MB cap) are bounded by the platform — keep clips to a spoken utterance rather than long recordings. For lipsync on audio you already have, pass wav/pcm directly; browser-recorded WebM/Opus must be decoded to WAV or PCM first (Riva and Audio2Face do not accept the WebM container).

Requirements

• Node.js >= 20.
• Network access to https://three.ws (or your own THREE_WS_BASE).

Environment variables

Variable	Required	Default	Purpose
THREE_WS_BASE	no	https://three.ws	Which deployment to talk to.
THREE_WS_API_KEY	no	—	Bearer token. Raises the metered TTS/ASR/A2F rate limits and unlocks your own private mocap clips.
THREE_WS_TIMEOUT_MS	no	60000	Per-request timeout for the live model calls.

Links

• Homepage: https://three.ws
• Changelog: https://three.ws/changelog
• Issues: https://github.com/nirholas/three.ws/issues
• License: Apache-2.0, see LICENSE

---

Support the project

If audio-mcp saves you time, star it on GitHub. Stars are how other developers and AI agents find the repositories worth trusting, and they cost you one click.

Know someone who would use it? Post on X · Share on Bluesky · Share on LinkedIn · Submit to Hacker News · Share on Reddit

Built for AI agents too

Coding agents and LLM tooling can read this repo directly: AGENTS.md, llms.txt, llms-full.txt. Point an agent at https://github.com/nirholas/audio-mcp and it has the context it needs.

More from the same author

• All repositories by nirholas: the full catalog, grouped by topic
• three.ws: the platform for 3D AI agents with Solana wallets, a skill marketplace and x402 payments
• Questions or ideas: open an issue or start a discussion

Contributors

Star history

Part of three.ws

three.ws is a platform for 3D AI agents with Solana wallets: avatars, a skill marketplace, x402 payments and more than seventy MCP servers. @three-ws/audio-mcp is one package from it.

• Documentation site: https://nirholas.github.io/audio-mcp/
• npm: https://www.npmjs.com/package/@three-ws/audio-mcp
• Canonical source: https://github.com/nirholas/three.ws (this repository is a generated mirror of packages/audio-mcp)
• Agent-readable summary: llms.txt and AGENTS.md
• Issues and ideas: https://github.com/nirholas/audio-mcp/issues