{"id":"7ce864d4-ddf5-4f8e-ba80-035ce7068a69","slug":"voicesmith","kind":"mcp","name":"voicesmith","summary":"Clone a voice on your own laptop, with the owner's permission, and let your AI agent talk in it. Checks every take before you hear it and watermarks everything. CLI + MCP server, no cloud.","owner":"fergo5002","repoUrl":"https://github.com/fergo5002/voicesmith","license":"MIT","licenseTier":"permissive","trust":"community","stars":0,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-10-06T13:22:29.000Z","addedAt":"2026-10-28T08:59:22.868Z","logoUrl":"https://avatars.githubusercontent.com/u/202719395?u=6a7a6bd0daf38fb63a540654d054ecf26bb301ff&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["design-media","cloud-infra"],"tags":["ai-agents","consent","local-first","python","speech-synthesis","text-to-speech","tts","voice-cloning"],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-28T08:59:58.555Z","sample":false,"url":"https://conduits.dev/tools/voicesmith","language":"Python","transports":["stdio"],"install":{"pypi":"voicesmith"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# voicesmith\n\n**Your AI agent, in a voice you chose. Cloned on your own laptop, with the owner's say-so, and watermarked on the way out.**\n\nHand it recordings you already have (podcasts, videos, voice notes, a YouTube link) and it builds a voice that Claude Code, Codex, Cursor or your own scripts can speak in. Nothing leaves your machine.\n\nIt's fussy on purpose. Every take is transcribed and voice-matched before you hear it, and nothing renders at all until the person whose voice it is has said yes.\n\nA real run, from a copy installed straight from this repo, on a laptop with no GPU (only the file paths are shortened):\n\n```console\n$ voicesmith say reader1089 \"This was rendered by the copy installed straight from GitHub.\" --engine sopro --quality fast -o live.ogg -o live.m4a\n14:48:22 loading Sopro V2 Turbo (Halo Research)\n14:48:39 rendering 1 part(s) with reference ref02\n14:48:55 part 1/1 take 1: q=0.93\n14:48:55 mastering, watermarking and verifying\nwrote live.ogg\nwrote live.m4a\n3.6s of audio in 43s with sopro (1 take(s)); similarity 0.874, word error 0.0%. Manifest: live.voicesmith.json\n```\n\n## What makes it different\n\n- **It does the tedious bit for you.** Give it a podcast. It finds the speech, works out who is talking, sets aside the co-host, drops clips with music, laughter, applause or background chatter, and keeps a varied handful of the cleanest clips of the right person. If two people talk about equally and you haven't said which one you mean, it stops and asks instead of guessing.\n- **It auditions itself.** `voicesmith tune` tries each installed engine that suits your machine and language against the best clips, and keeps the best-scoring engine and clip (mostly likeness, plus getting the words right). You do that once per voice; later renders use the winner unless you ask for speed with `--quality fast`.\n- **It is its own harshest critic.** Each take is transcribed and voice-matched. Dropped words, loops, trailing off, long dead air, or a voice drifting towards someone else's: binned and re-rendered before you hear anything.\n- **Consent or nothing.** A voice will not render until its owner has read a consent statement with a one-off code (checked by speech recognition and voice match), or you have recorded who authorised it and how. Through the MCP server, agents can use voices but cannot grant consent.\n- **It signs its work.** Every file carries an AudioSeal watermark and \"AI-generated\" tags, and is decoded and re-checked after encoding. If either did not survive, the file is not delivered. `voicesmith verify` checks any file.\n- **Your agent can use it.** One command adds it to Claude Code or Codex as an MCP server (Cursor takes a short config entry), so your assistant can make you voice notes in a voice you chose.\n- **Built for ordinary laptops.** The core needs no GPU or PyTorch. Each engine gets the right PyTorch build for your machine (CUDA, Apple Silicon or CPU). So far it has been measured on a CPU-only Windows laptop; see [the benchmarks](docs/benchmarks.md) for honest numbers.\n\n## Quick start\n\n```bash\n# 1. Install (needs Python 3.10 to 3.13; uv fetches one if you lack it. No uv? https://docs.astral.sh/uv/)\nuv tool install git+https://github.com/fergo5002/voicesmith\nvoicesmith doctor                       # checks ffmpeg, models, hardware, and tells you how to fix anything\nvoicesmith engines install recommended  # Chatterbox; adds Qwen3-TTS on NVIDIA machines\n\n# 2. Create a voice and get consent\nvoicesmith voice create sam --speaker \"Sam Example\"\nvoicesmith consent request sam          # prints a statement with a one-off code\n#    Sam reads it aloud. A voice note recorded on a phone is fine.\nvoicesmith consent verify sam sam-consent.m4a\n\n# 3. Give it audio of Sam speaking (5 to 30 minutes of solo speech is plenty)\nvoicesmith ingest sam interview.mp3 ./more-recordings https://www.youtube.com/watch?v=...\n\n# 4. Let it find the best engine and reference for Sam (run once)\nvoicesmith tune sam\n\n# 5. Speak\nvoicesmith say sam \"Thanks for coming in yesterday. Here's the plan.\" -o note.m4a\n\n# 6. Hand it to your agent\nclaude mcp add --scope user voicesmith -- voicesmith mcp    # Claude Code\ncodex mcp add voicesmith -- voicesmith mcp                  # Codex (Cursor and others: see below)\n```\n\nThen ask your agent for \"a voice note saying the tests passed\".\n\nEvery render writes `<name>.voicesmith.json` next to the audio with the scores, the engine, reference and seed, every take that was tried and why any were rejected, and the post-encode watermark check.\n\n**Several people talking?** If no one clearly dominates the recordings and there is no spoken consent to anchor on, ingest stops and lists the voices it found with example timestamps, rather than guessing. Re-run with `--pick 2`, or pass `--target clip.wav` with a few seconds of only the right person.\n\n**Want the closest match you can get?** `--quality best` tries more takes, and `--polish` also runs Chatterbox's voice conversion over each take, keeping it only when it scores higher. Both cost time; see the benchmarks for what they buy.\n\n## Use it from an agent\n\n**Claude Code** (`--scope user` makes it available in every project, not just the current folder)\n\n```bash\nclaude mcp add --scope user voicesmith -- voicesmith mcp\n```\n\n**Codex**: `codex mcp add voicesmith -- voicesmith mcp`, or add it to `~/.codex/config.toml`:\n\n```toml\n[mcp_servers.voicesmith]\ncommand = \"voicesmith\"\nargs = [\"mcp\"]\n```\n\n**Cursor, Windsurf and others:** add a stdio server whose command is `voicesmith mcp`.\n\nTools: `list_voices`, `voice_info`, `speak`, `job_status`, `verify_audio`, `consent_steps`, `doctor`. `speak` returns file paths and quality scores. Long renders come back as a job id to poll, so agents with short tool timeouts still work. Agents write only new files inside `~/.voicesmith/outputs`, never anywhere else, and cannot create or change consent; `consent_steps` tells the human what to do. Engines an agent has not used for 15 minutes are unloaded to free memory.\n\nThere is also a skill at [`skills/voicesmith/SKILL.md`](skills/voicesmith/SKILL.md) for agents that use skills. Copy it into your agent's skills folder (for Claude Code, `~/.claude/skills/voicesmith/`).\n\n## How it works\n\n```\nsources ──► decode (ffmpeg) ──► speech detection (Silero) ──► transcript + word timings (Parakeet)\n        ──► clips cut at pauses, 4 to 14 s ──► speaker embeddings (TitaNet)\n        ──► keep the consenting / dominant speaker ──► gates: clipping, bandwidth, noise, clarity\n        ──► score + diversity ──► sound-event screen (music, laughter...) ──► reference bank\n\ntune:   every installed engine × top references × probe sentences ──► best (engine, reference) per voice\n\nsay:    one continuous take where the engine allows it\n        ──► best-of-N with early stop: length, transcript, ending, loops, dead air, voice match, prosody\n        ──► join (if split) with punctuation-sized pauses and room tone\n        ──► trim, linear loudness towards -16 LUFS, capped at -1.5 dBTP ──► AudioSeal watermark\n        ──► encode with disclosure tags ──► decode and re-verify every file ──► deliver\n```\n\nThe core never imports PyTorch: all analysis runs on ONNX Runtime. Each synthesis engine lives in its own uv-managed environment, because engines pin conflicting versions of torch and transformers, and talks to the core over a small JSON protocol. Workers stay warm between renders inside the MCP server.\n\nDesign notes and sources: [`docs/research/`](docs/research/). Measured numbers: [`docs/benchmarks.md`](docs/benchmarks.md).\n\n## Engines\n\n| Engine | Weights licence | Languages | Why it is here |\n|---|---|---|---|\n| `chatterbox-turbo` | MIT | English | Best commercially usable cloner on the 2026 controlled-voice arena; built-in PerTh watermark |\n| `chatterbox-nano` | MIT | English | 110M-parameter Turbo for slow machines |\n| `chatterbox-multilingual` | MIT | 23 languages | Multilingual V3 |\n| `qwen3-tts-0.6b` / `qwen3-tts-1.7b` | Apache-2.0 | 10 languages | Transcript-conditioned cloning, strong identity, long text in one pass |\n| `sopro` | Apache-2.0 | EN, PT, FR, DE | 120M, int8 on CPU, the quick option |\n\nOnly engines whose weights allow commercial use ship by default. `voicesmith engines list` shows what is installed. The weights belong to their makers and carry their own licences.\n\n## Hardware\n\nvoicesmith detects what it is running on and installs the right PyTorch build for each engine: CUDA wheels when there is an NVIDIA GPU, the default wheels on macOS (MPS), and CPU wheels everywhere else. It deliberately avoids uv's automatic backend on Intel-graphics laptops, because that picks Intel XPU wheels for GPUs PyTorch XPU does not support.\n\nBe realistic about CPU-only machines: cloning models run between about 1.5x and 10x slower than real time on a laptop CPU (see the benchmarks). `--quality fast` uses the quickest engine whose tuned similarity is within 0.04 of the best. Expect a GPU to be much quicker; GPU speeds have not been measured for this release.\n\nLoudness is set with one linear gain, aiming for -16 LUFS but never pushing true peaks past -1.5 dBTP, so speech with sharp peaks can come out a little quieter than -16. That is deliberate: a limiter would get closer to the number by squashing the voice.\n\n## Consent and safety\n\nRead [`RESPONSIBLE_USE.md`](RESPONSIBLE_USE.md). In short: clone only people who agreed, never pass output off as a real recording, and know that this is MIT code, so the safeguards make the honest path easy rather than making abuse impossible.\n\n## Commands\n\n| Command | What it does |\n|---|---|\n| `voicesmith doctor` | Check the setup and print fixes |\n| `voicesmith engines list / install / remove` | Manage engine environments |\n| `voicesmith voice create / show / delete` | Manage voices |\n| `voicesmith voices` | List voices |\n| `voicesmith consent request / verify / attest / show / revoke` | Consent records (every change is kept in `consent/history.jsonl`) |\n| `voicesmith ingest <voice> <files, folders, URLs> [--record SECONDS] [--pick N] [--target clip]` | Add audio and rebuild references |\n| `voicesmith tune <voice>` | Pick the best engine and reference for a voice |\n| `voicesmith say <voice> \"text\" -o out.m4a [--quality fast/balanced/best] [--polish]` | Render speech |\n| `voicesmith verify <file>` | Check a file for the watermark and disclosure tags |\n| `voicesmith mcp` | Run the MCP server on stdio |\n\nEverything lives in `~/.voicesmith` (override with `VOICESMITH_HOME`).\n\n## Troubleshooting\n\n- **First run is slow.** Engines download several GB of weights on first use. On Windows, Defender scans each new model file; adding `~/.voicesmith` and `~/.cache/huggingface` to its exclusions speeds this up.\n- **YouTube downloads fail.** yt-dlp now needs a JavaScript runtime for YouTube. Install [deno](https://deno.com).\n- **\"no usable consent\".** Do the consent step. Agents cannot do it for you.\n- **A render keeps failing its checks.** The manifest lists every take and why it was rejected. Try `voicesmith tune` again with more references, or ingest cleaner audio.\n- **Logs** are in `~/.voicesmith/logs/`.\n\n## Development\n\n```bash\ngit clone https://github.com/fergo5002/voicesmith && cd voicesmith\nuv venv && uv pip install -e \".[dev]\"\nuv run pytest -q\n```\n\nSee [`AGENTS.md`](AGENTS.md) for the layout and the rules for new engines.\n\n## Licence\n\nMIT for this code. Models are downloaded from their makers and keep their own licences: Chatterbox (MIT), Qwen3-TTS (Apache-2.0), Sopro (Apache-2.0), Parakeet TDT and TitaNet (CC-BY-4.0, NVIDIA), Silero VAD (MIT), CED (Apache-2.0), AudioSeal (MIT).\n"}