ai-jam-sessions

---

Listen first: three hymns, sung on the beat

Amazing Grace, America the Beautiful and the Battle Hymn of the Republic, every verse, sung by a synthetic voice over a piano arrangement written for each hymn. The landing page plays them with a 3D score that follows the voice and the words lighting up as they are sung.

Each performance was assembled from sixteen takes of a score-conditioned singer:

• one take per phrase, chosen by a local listener and a pitch tracker;
• placed on the score by time-warping, so nothing inside a phrase is cut;
• checked by two timing instruments and two pitch trackers;
• passed by ear, with every problem marked on a listening review that traced it back to its cause.

A sound check before each song measures the voice's timing footprint, and says how many takes the song will need.

How it was made: handbook → Vocals.

	Amazing Grace	America the Beautiful	Battle Hymn
syllables within 40 ms of the score	107 of 112	217 of 224	285 of 414
notes within 50 cents	135 of 140	218 of 224	368 of 424

The Battle Hymn's quick dotted notes are left on the singer's own timing, which is where most of its timing misses come from.

The voice is synthetic: SoulX-Singer (Apache-2.0), with its own example timbre, imitating no real singer. The piano arrangements were written for this project by Kimi-K3 and are dedicated to the public domain (CC0). All three hymns are in the public domain.

What is this?

A piano and guitar that AI learns to play. Not a synthesizer, not a MIDI library — a teaching instrument.

An LLM can read and write text, but it can't experience music the way we do. No ears, no fingers, no muscle memory. AI Jam Sessions closes that gap by giving the model senses it can actually use:

• Reading — real MIDI sheet music with deep musical annotations. Not hand-written approximations — parsed, analyzed, and explained.
• Hearing — six audio engines (oscillator piano, sample piano, vocal samples, physical vocal tract, additive vocal synth, physically-modeled guitar) that play through your speakers, so the humans in the room become the AI's ears. And now the model has ears of its own, twice over: it can measure a recording after the fact (see Listening) and it can watch the band while the music is still going (see The Live Ensemble).
• Seeing — a piano roll that renders what was played as SVG the model can read back and verify. An interactive guitar tablature editor. A browser cockpit with a visual keyboard, dual-mode note editor, and tuning lab.
• Remembering — a practice journal that persists across sessions, so learning compounds over time.
• Singing — vocal tract synthesis with 20 voice presets, from operatic soprano to electronic choir. Sing-along mode with solfege, contour, and syllable narration. And whole songs sung on the piano's clock: a score-conditioned singer driven by the song's arrangement, assembled phrase by phrase from many takes, and gated on timing (40 ms) and pitch (50 cents) before you hear it — see Sing.

Every one of the 109 songs is now fully annotated — historical context, bar-by-bar structural analysis, key moments, teaching goals, and performance tips, in all 12 genres. An earlier version of this README said the raw songs were "waiting for the AI to absorb the patterns, play the music, and write its own annotations." That is exactly what happened: the annotations were written by AI against a deterministic per-song analysis (chords, repetition structure, section boundaries, content-verified keys), gated by a quality rubric, and adversarially fact-checked claim by claim — measure numbers, chord windows, and structural counts all verified against the actual MIDI before anything shipped.

Out of this same work, we also publish jam-actions-v0 — a public dataset of 57 multi-turn MCP tool-use traces over real classical piano. It teaches LLMs to do grounded tool-use over symbolic music, not just text generation, and ships with a 7-axis release gate that distinguishes "passing on evidence" from "passing because the task is trivial." See Training Dataset below for the full story.

Listening

For a long time this server could make sound but never examine it. The model played, a human listened, and the model took their word for it. That gap is now closed.

Point it at a WAV and it measures what is there. Not by looking at a picture and guessing — by running the signal through the same kind of tools it already uses on the score:

• analyze_audio — onsets, the pitch contour, and level. Pitch comes back as note names with cent deviations, never as raw frequencies. Level means the real numbers now: BS.1770-4 integrated loudness in LUFS, the sample peak in dBFS, and a clipped-sample count, plus an integrity pass that reports dropouts and noise bursts — the click-and-glitch half of render QA. A Balance section says where the energy sits, from sub-bass to air: each band's share against pink noise, the brightness, and the tilt.
• transcribe_audio — the recording as notes: pitch, start, duration, and how far each note sits from concert pitch.
• score_audio_take — grade a performance against a song in the library by ear. It transcribes the recording, matches it to the score, and reports which notes landed, which drifted, and which were missed. Then view_scored_piano_roll draws the result over the score, exactly as it does for a captured MIDI take. This is how you grade a real instrument, a sung take, or anything where there is no MIDI to capture.
• view_spectrogram — see the sound. A constant-Q spectrogram with a piano keyboard down the left edge, so pitch is readable at a glance, and the song's intended notes drawn over it on request.
• check_loop_seam — judge a looping render where it wraps: the waveform step after extrapolating the tail's own slope, burst-shaped energy at the join, and the level shift across the seam. A phase-perfect loop reads clean even when the raw boundary sample jumps.
• compare_balance — the tone of a render against a reference that is known to be right, band by band in dB. A cue that should sit like its siblings, a re-render that should match the take it replaces. Loudness cancels out, so only the tone is compared.

What it will not tell you. The picture is for finding where something is wrong; every number comes from signal processing, never from a model reading an image. The transcriber follows one line at a time, so a chord or a full mix will produce something confident and wrong, and it says so. Onset detection runs around F1 0.88 at the state of the art, so a "missed" note may be one the transcriber could not hear rather than one you did not play — the tools carry that caveat in their own output rather than burying it here.

The whole surface is dependency-free: the transform, the pitch tracker, the onset detector, the WAV decoder and the PNG encoder are all in this repo, and they produce identical numbers in Node and in the browser.

The Live Ensemble

Listening grades a recording once it has finished. This is the other half: asking what every instrument is doing right now, mid-performance.

ensemble_now()

It answers with each instrument's held notes, how long each has been held, and the combined chord across the whole ensemble. During a duet the two voices are reported separately, so you can see the piano holding a triad while the synth carries the melody over it.

Two channels, and the cheap one is the accurate one

This is the part worth understanding, because it decides which number to trust.

Intent — what each engine was told to play. When the model is the one performing, this is not an estimate. A piano chord is not something to transcribe; it is three note-ons that were sent. The notes are exact, free, and immediate.

Acoustic — what actually came out. Each engine can fan its output into a private analysis bus, so every instrument is measured at the source with no separation and no ambiguity. This channel is verification, not discovery: it is how you learn that a voice drifted off the clock, a take clipped, or an engine went silent while still being sent notes.

When the two disagree, that is a fact about the render, not a correction to the notes.

What it costs

Watching an instrument costs about 9 microseconds per audio callback against a 42.67 ms block, which is roughly 0.02% of the audio budget, measured with zero dropped samples. An instrument with no observer attached costs nothing at all.

What it will not tell you

The acoustic channel lags, and says by how much: about 23 ms for pitch, and 70 ms for a confirmed onset, because an onset cannot be confirmed until the audio after it has arrived. Onsets near that edge are withheld rather than reported and later retracted.

The acoustic tracker follows one line at a time, so it will not name the notes of a chord — and it does not pretend to. A chord it cannot resolve is its known limitation rather than a finding, and the ensemble stays quiet about it instead of crying wolf on every chord the piano plays.

The Piano Roll

The piano roll is how the AI sees music. It renders any song as SVG — blue for right hand, coral for left, with beat grids, dynamics, and measure boundaries:

Two color modes: hand (blue/coral) or pitch-class (chromatic rainbow — every C is red, every F# is cyan). The SVG format means the model can both see the image and read the markup to verify pitch, rhythm, and hand independence.

The Cockpit

A browser-based composition studio that lives in this repo at apps/cockpit — and runs live at mcp-tool-shop-org.github.io/ai-jam-sessions/cockpit. No plugins, no DAW, no install; everything stays in your browser (your work autosaves locally). Prefer to hack on it?

cd apps/cockpit && npm install && npm run dev   # Vite dev server, opens in your browser

• A sampled Concert Grand by default — the cockpit ships a pruned Salamander Grand pack (90 OGGs, 8 MB) that loads on your first interaction and plays through the same output chain as the synth voices; before it loads (or offline) the tuned oscillator piano covers seamlessly. Samples by Alexander Holm, CC-BY 3.0.
• Panel mode — the listening room — blind pairwise A/B auditions of the composition engine's voicings over real library melodies: loudness-matched clips offline-rendered through the real voice path, seeded shuffled trials with hidden floor catch-trials, Bradley-Terry rankings with bootstrap confidence intervals, and honest outcomes (PROVISIONAL until every pair reaches its vote budget; UNINTERPRETABLE when the discrimination floor fails). A second sub-mode runs the same ranking with local LLM judges, plus History across both run kinds and a Compare view (Kendall τ + engine-rank match) that asks whether the cheap proxy tracks the human truth.
• Beat-accurate transport — notes live in musical time, so the BPM control actually retimes playback; a click-to-seek time-ruler with drag-to-set loop regions; auto-scroll that follows the playhead
• Record-arm capture — play the QWERTY keys, on-screen keyboard, or a Web MIDI device and it lands in the score: 1-bar count-in, looper-style overdub across loop cycles (or replace mode), raw performance timing preserved under a quantized view, each pass one undoable unit
• Full undo/redo — every edit including Clear and Import is reversible (Ctrl+Z), with drag gestures coalescing the way real editors do
• Multi-select + clipboard — marquee selection under a Select/Draw tool toggle, platform-standard modifier clicks, copy/cut/paste-at-playhead, Duplicate
• Touch + accessibility — pointer events with capture on every surface, tap-to-relocate as a non-drag alternative, keyboard note editing, colorblind-safe scored overlays
• Dual-mode piano roll — switch between Instrument mode (chromatic pitch-class colors) and Vocal mode (notes colored by vowel shape: /a/ /e/ /i/ /o/ /u/)
• Visual keyboard — two octaves from C4, mapped to your QWERTY keyboard. Click or type.
• 20 voice presets — 15 Kokoro-mapped voices (Aoede, Heart, Jessica, Sky, Eric, Fenrir, Liam, Onyx, Alice, Emma, Isabella, George, Lewis, plus choir and synth-vox), 4 tract-mapped voices, and a synthetic choir section
• 10 instrument presets — the 6 server-side piano voices plus synth-pad, organ, bell, and strings
• Note inspector — click any note to edit velocity, vowel, and breathiness
• 7 tuning systems — Equal temperament, just intonation (major/minor), Pythagorean, quarter-comma meantone, Werckmeister III, or custom cent offsets. Adjustable A4 reference (392–494 Hz).
• Tuning audit — frequency table, interval tester with beat-frequency analysis, and tuning export/import
• Score import/export — serialize the entire score as JSON and load it back
• LLM-facing API — window.__cockpit exposes exportScore(), importScore(), addNote(), play(), stop(), panic(), setMode(), and getScore() so an LLM can compose, arrange, and play back programmatically

The Learning Loop

The Song Library

109 annotated songs across 12 genres, built from real MIDI files. Each genre has one deeply annotated exemplar — with historical context, bar-by-bar harmonic analysis, key moments, teaching goals, and performance tips (including vocal guidance). These exemplars serve as templates: the AI studies one, then annotates the rest. The folk exemplar is America the Beautiful (Samuel A. Ward, 1882, public domain): an F-major hymn arranged in this repository, melody F4–C5.

What ships, and what you fetch. The annotations are ours and ship with every song. The MIDI files were downloaded from public MIDI sites when the library was built, and a per-file provenance audit (docs/findings/library-provenance-audit.md) found that only 14 of those downloads carry a licence that permits redistribution — Bernd Krueger's piano-midi.de arrangements (CC-BY-SA-3.0-DE) and the Mutopia Project's public-domain typesettings. Those 14 are in the npm package. The other 94 downloads are not: their .json ships, with a provenance block naming the source, its terms and the file's SHA-256, and ai-jam-sessions library fetch --accept-source-terms downloads each one from the site that published it, under that site's terms, refusing any file whose hash no longer matches what the annotations were verified against. Twelve files that turned out to be a different piece than their name were quarantined, which is why that downloaded library is 108 songs and not the 120 earlier versions claimed. America the Beautiful is the 109th song and the fifteenth MIDI the package ships: the arrangement was made here and dedicated to the public domain. Versions before 2.6.0 shipped all 120 MIDI files; that was a mistake, and it is corrected here rather than papered over.

Genre	Exemplar	Key	What it teaches
Blues	The Thrill Is Gone (B.B. King)	B minor	Minor blues form, call-and-response, playing behind the beat
Classical	Für Elise (Beethoven)	A minor	Rondo form, touch differentiation, pedaling discipline
Film	Comptine d'un autre été (Tiersen)	E minor	Arpeggiated textures, dynamic architecture without harmonic change
Folk	America the Beautiful (Ward)	F major	Tonic-dominant hymn harmony, patriotic phrasing, melody in F4–C5
Jazz	Autumn Leaves (Kosma)	G minor	ii-V-I progressions, guide tones, swing eighths, rootless voicings
Latin	The Girl from Ipanema (Jobim)	F major	Bossa nova rhythm, chromatic modulation, vocal restraint
New-Age	River Flows in You (Yiruma)	A major	I-V-vi-IV recognition, flowing arpeggios, rubato
Pop	Imagine (Lennon)	C major	Arpeggiated accompaniment, restraint, vocal sincerity
Ragtime	The Entertainer (Joplin)	C major	Oom-pah bass, syncopation, multi-strain form, tempo discipline
R&B	Superstition (Stevie Wonder)	Eb minor	16th-note funk, percussive keyboard, ghost notes
Rock	Your Song (Elton John)	Eb major	Piano ballad voice-leading, inversions, conversational singing
Soul	Lean on Me (Bill Withers)	C major	Diatonic melody, gospel accompaniment, call-and-response

Songs progress from raw (MIDI only) → annotated → ready (fully playable with musical language). The AI promotes songs by studying them and writing annotations with annotate_song.

Sound Engines

Six engines, plus a layered combinator that runs any two simultaneously:

Engine	Type	What it sounds like
Oscillator Piano	Additive synthesis	Multi-harmonic piano with hammer noise, inharmonicity, velocity-shaped brightness, 48-voice polyphony, stereo imaging. Zero dependencies.
Sample Piano	Sample playback	Salamander Grand Piano — the real thing. The default engine whenever a pack is installed (samples/AccurateSalamander or AI_JAM_SAMPLES_DIR); the npm tarball stays sample-free, so you supply the Salamander download. The browser cockpit ships its own pruned 8 MB pack (90 OGGs, CC-BY 3.0 Alexander Holm) — no setup at all on the web.
Vocal (Sample)	Pitch-shifted samples	Sustained vowel tones with portamento and legato mode.
Vocal Tract	Physical model	Pink Trombone — LF glottal waveform through a 44-cell digital waveguide. Four presets: soprano, alto, tenor, bass.
Vocal Synth	Additive synthesis	15 Kokoro voice presets with formant shaping, breathiness, vibrato. Deterministic (seeded RNG).
Guitar	Additive synthesis	Physically-modeled plucked string — 4 presets (steel dreadnought, nylon classical, jazz archtop, twelve-string), 8 tunings, 17 tunable parameters.
Layered	Combinator	Wraps two engines and dispatches every MIDI event to both — piano+synth, vocal+synth, etc.

Keyboard Voices

Six tunable piano voices, each adjustable per-parameter (brightness, decay, hammer hardness, detune, stereo width, and more):

Voice	Character
Concert Grand	Rich, full, classical
Upright	Warm, intimate, folk
Electric Piano	Silky, jazzy, Fender Rhodes feel
Honky-Tonk	Detuned, ragtime, saloon
Music Box	Crystalline, ethereal
Bright Grand	Cutting, contemporary, pop

Guitar Voices

Four guitar voice presets with physically-modeled string synthesis, each with 17 tunable parameters (brightness, body resonance, pluck position, string damping, and more):

Voice	Character
Steel Dreadnought	Bright, balanced, classic acoustic
Nylon Classical	Warm, soft, rounded
Jazz Archtop	Mellow, woody, clean
Twelve-String	Shimmering, doubled, chorus-like

The Practice Journal

After every session, the server captures what happened — which song, what speed, how many measures, how long. The AI adds its own reflections: what it noticed, what patterns it recognized, what to try next.

---
### 14:32 — Autumn Leaves
**jazz** | intermediate | G minor | 69 BPM × 0.7 | 32/32 measures | 45s

The ii-V-I in bars 5-8 (Cm7-F7-BbMaj7) is the same gravity as the V-i
in The Thrill Is Gone, just in major. Blues and jazz share more than the
genre labels suggest.

Next: try at full speed. Compare the Ipanema bridge modulation with this.
---

One markdown file per day, stored in ~/.ai-jam-sessions/journal/. Human-readable, append-only. Next session, the AI reads its journal and picks up where it left off.

Training Dataset

jam-actions-v0 — a public dataset of multi-turn MCP tool-use traces grounded in classical-piano MIDI. Built from the same library this server teaches with, the dataset teaches LLMs to do grounded tool-use over symbolic music — not just text generation.

Each record pairs a 4-measure phrase window with an annotated teaching target and a target trace — a turn-by-turn session in which an assistant uses the MCP tools above (get_events_in_measure, get_events_in_hand, count_distinct_pitch_classes, and the rest of the 9-tool MIDI inspector surface) to read, analyze, and discuss the phrase.

Version	0.6.0 (2026-09-25) — correction release (see below)
Records	57 (public subset): train 45, held-out test 12 (clair-de-lune)
Compositions	4 classical piano works: Bach BWV 846, Mozart K. 545 I, Beethoven "Für Elise", Debussy "Clair de Lune"
Source MIDI	piano-midi.de — Bernd Krueger arrangements, each file carrying Krueger's own copyright credit
License	CC-BY-SA-3.0-DE (arrangements) over public-domain compositions
Where	mcp-tool-shop/jam-actions-v0 on Hugging Face and on Zenodo; DOIs and citation in the dataset card and CITATION.cff

The 0.6.0 correction. Versions 0.4.x and 0.5.x held 115 records across 8 works, all attributed to Bernd Krueger. That attribution was checked against piano-midi.de's composer pages, not against the files. When the song library was audited from the MIDI bytes, four of those songs — Chopin's Nocturne Op. 9 No. 2 and Prelude Op. 28 No. 4, Beethoven's "Pathétique" II and Schumann's "Träumerei" — turned out to have been built from files obtained from midiworld.com and bitmidi.com, with no established arrangement licence. Their 58 records carried those arrangements note for note, so 0.6.0 withdraws them. The 57 remaining records are byte-identical to 0.5.0. Please do not redistribute the withdrawn records from earlier versions. The full account is in docs/findings/published-dataset-licence-audit.md.

The evidence gate. The packager now re-reads each song's library provenance block — re-derived from the MIDI bytes — and refuses any record whose song has no redistributable arrangement licence, or whose source-file hash differs from the evidenced file. It fails closed. A test re-runs the same check on every committed published package, so a later provenance change turns the build red on the published sets themselves.

Quality story — the 7-axis release gate. The dataset's release gate distinguishes evidence-grounded passing from ceiling-saturated passing. Axes 1–6 are blocking (absolute floor, margin compound, tool-use rate, correct-after-tool, misinterpretation count, stratum floor); axis 7 is enriched-vs-non reporting. Its recorded verdicts were measured on the 115-record composition of 0.4.x and 0.5.x and remain reproducible from the tag jam-actions-v0-0.5.0-cut-2026-07-11; they have not been re-measured on 0.6.0.

Reproducibility. A fresh contributor on any platform (Windows native, macOS, Linux, WSL) can verify the package and rebuild it:

git clone https://github.com/mcp-tool-shop-org/ai-jam-sessions.git
cd ai-jam-sessions && pnpm install
pnpm exec tsx scripts/verify-public-package-checksums.ts                        # every file accounted for
pnpm exec tsx scripts/package-jam-actions-public.ts --today 2026-09-25 --dry-run # evidence gate runs first
pnpm build && pnpm exec tsx scripts/verify-public-package-execution.ts
# → "VERDICT: PASS" — every frozen tool call replays live (needs an audio device)

.gitattributes pins LF line endings for *.sha256 and the public-dataset tree so the checksum verifier works on every platform.

Fine-tuning history. The dataset's claims were tested with preregistered fine-tunes scored against sealed baselines. v0 (the 78 jam traces alone) returned an honest negative (report); v1 moved tool-grounded QA +0.202 but missed its preregistered bar by one paired win (report); B-1 re-tested the frozen v1 adapters on a 36-record cohort: 0.678 → 0.890, 29/36 paired wins against the ex-ante 24/34 bar (report). Those measurements stand as recorded. The adapters themselves are withdrawn: their training data included records from the four withdrawn songs, so they cannot be offered under this dataset's terms. Adapters trained only on licence-cleared material are published with the jam-actions-v1 datasets.

The MIDI arrangements are by Bernd Krueger (piano-midi.de), licensed CC-BY-SA-3.0-DE. The annotations, traces, and eval artifacts are by the AI Jam Sessions team, released under the same license so the share-alike chain is preserved end-to-end. License boundary: the repository's MIT license covers the code; everything under datasets/ is CC-BY-SA-3.0-DE. The working corpus at datasets/jam-actions-v0/ also contains records that are not published: two works whose provenance was never verified (Satie Gymnopédie No. 1, Debussy Arabesque No. 1) and the four withdrawn in 0.6.0 — see datasets/jam-actions-v0/PROVENANCE-NOTE.md.

The acoustic corpus

jam-actions-acoustic-v0 — the counterpart to the traces above, over audio rather than symbolic music. 72 records, each pairing a deliberately perturbed synthetic rendering of a public-domain phrase with the verdict the analysis tools actually return, so every label is checked against the instrument rather than only against itself.

Version	1.1.0 (2026-09-25) — withdrew the 36 "Träumerei" records of 1.0.x, whose source file has no established arrangement licence
Records	72 — 2 phrases (Bach, Für Elise) × 9 perturbation kinds × 4 target notes
Held out	by phrase (Für Elise), not by record, so a perturbed twin of the same melody cannot leak
Classes	match, pitch fail/warn, timing fail/pass, missed, extra, in-tune vibrato, nothing-to-grade silence
Audio	none distributed — each record carries a deterministic recipe and the SHA-256 of the waveform it produces
Schema	jam-actions-acoustic-v0/1.0.0

Two of the nine classes are there because a naive model answers them confidently and wrongly: a vibrato note whose correct verdict is in tune, and silence whose correct verdict is nothing to grade. Every threshold the verdict depends on is copied into the record, because both of them moved once during the build.

The corpus is reproducible from this repository. Regenerating it produces all 79 published files and a byte-identical checksums.sha256, and a test asserts exactly that without writing the published tree.

One caveat, measured rather than assumed. Each record carries wav_sha256, the hash of the waveform its recipe produces, and the renderer calls Math.pow and Math.sin once per sample. Neither is required to be correctly rounded, and V8's results changed between Node 22 and Node 24: of the 27,869 distinct Math.pow(2, x) arguments the original corpus evaluated, 253 return a different double. Almost all of that vanishes under 16-bit quantisation, but 2 of the 72 records — both the extra perturbation of Für Elise, whose motif sits on the one pitch where the semitone ratio itself differs — hash differently on Node 24. Every other field of every record reproduces on any engine, and the repository tests both claims separately. If you re-render and see those two mismatch, that is this, not a corrupt download. Making the waveform bit-portable means replacing the transcendentals, which changes every hash and therefore needs a new schema version.

Build your own

The scaffolding that corpus runs on is available for your own experiments. experiments/_template/ is a working example you can copy: declare a task, and you get SFT formatting, per-class scoring, trivial baselines over your declared verdict set, and a check that no holdout unit straddles the split.

The contract is the part worth reading. Ground truth is constructible rather than hand-written, labels are verified against what the tools measure, you split by the unit that leaks, and you report the baselines and the base model beside any result. Each of those rules cost something to learn.

Install

npm install -g @mcptoolshop/ai-jam-sessions

Requires Node.js 22+ (v2.0.0 raised the floor with node-web-audio-api 2.0). No MIDI drivers, no virtual ports, no external software.

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ai_jam_sessions": {
      "command": "npx",
      "args": ["-y", "-p", "@mcptoolshop/ai-jam-sessions", "ai-jam-sessions-mcp"]
    }
  }
}

Docker

Every release also publishes ghcr.io/mcp-tool-shop-org/ai-jam-sessions, a slim image that runs the MCP server on stdio. The one thing to know is that /data is the memory: the journal, server state, user songs and any fetched MIDI live there, so mount a volume or they die with the container.

{
  "mcpServers": {
    "ai_jam_sessions": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "-v", "ai-jam-data:/data", "ghcr.io/mcp-tool-shop-org/ai-jam-sessions"]
    }
  }
}

The image ships the 14 redistributable MIDI files; library fetch --accept-source-terms run once inside the container puts the other 94 in the volume. docker compose up does the same with a named volume, and --profile ollama adds an Ollama sidecar. Details, the CLI inside the image and what is deliberately not in it: docs/docker.md. Running the fine-tuned graders in Ollama, with the measured cost of a 4-bit base: docs/ollama-adapters.md.

MCP Tools

56 tools and 4 prompt templates across eight categories:

Learn

Tool	What it does
list_songs	Browse by genre, difficulty, or keyword
song_info	Full musical analysis — structure, key moments, teaching goals, style tips
registry_stats	Library-wide stats: total songs, genres, difficulties
list_measures	Every measure's notes, dynamics, and teaching notes
teaching_note	Deep dive into a single measure — fingering, dynamics, context
suggest_song	Recommendation based on genre, difficulty, and what you've played
practice_setup	Recommended speed, mode, voice settings, and CLI command for a song
compare_songs	Cross-genre pattern recognition — key relationships, pitch/interval similarity, shared forms, teaching connections
annotation_progress	Track annotation quality across the library — scores, grades, and improvement suggestions
server_info	Server version, library stats, engine list, active session

Play

Tool	What it does
play_song	Play through speakers — library songs or raw .mid files. Four engines (piano, vocal, tract, guitar), any speed, mode, measure range — plus a metronome with count-in and a record flag that captures the session for scoring. The synth and layered engines are CLI-only.
stop_playback	Stop
pause_playback	Pause or resume
set_speed	Change speed mid-playback (0.1×–4.0×)
playback_status	Real-time snapshot: current measure, tempo, speed, keyboard voice, state
view_piano_roll	Render as SVG (hand color or pitch-class chromatic rainbow)
score_performance	Score a MIDI play-along — pitch accuracy, timing, completeness, with graded feedback
mute_hand	Mute or unmute left/right hand during practice — isolate one hand at a time
detect_chord	Name the chord from a set of currently-sounding MIDI notes (e.g. [60,64,67] → C)
preview_teaching_cues	See all teaching notes and key moments before playing

Practice

Tool	What it does
practice_loop	The drill a real teacher assigns: loop measures 5–8 slower, and the tempo ramps up (+5%) only after a clean pass — each pass recorded, scored, and summarized
practice_status	Where the drill stands: current pass, speed, and a per-measure diagnostic of the last take
score_last_take	Score the most recent recorded take — pitch accuracy, timing, completeness, per-note verdicts
view_scored_piano_roll	The marked-up score every teacher uses: the piano roll overlaid with per-note verdicts in a colorblind-safe palette (solid = correct, dashed = timing, ✕ = missed)

Sing

Tool	What it does
sing_along	Singable text — note-names, solfege, contour, or syllables. With or without piano accompaniment.
ai_jam_sessions	Generate a jam brief — chord progression, melody outline, and style hints for reinterpretation
verify_harmony	The maker loop's verification gate: a proposed reharmonization is checked by the platform's own deterministic tools — chord fidelity (the chord engine must detect each intended chord), melody consonance (tone/tension/chromatic), bass voice-leading, key membership
auto_reharmonize	The maker loop in one call — a local model proposes a reharmonization, verify_harmony's deterministic gate checks every voicing, best-of-n until a verified interpretation comes back
compose_panel	Run the voice-leading composition panel on any songs: four systems realize accompaniments, blind cross-family LLM judges rank them, Bradley-Terry aggregates — with a discrimination-floor gate that voids uninterpretable runs (directional signal only, never a quality score). Runs for minutes and streams progress notifications while it works.

A whole song on the clock: the vocal route. Any song can carry a real sung vocal that lands on the piano.

• The clock. A score clock (scripts/build-score-clock.mjs) derives every syllable's pitch, onset and duration from the song's arrangement, on the player's own timeline.
• The sound check. Before a song is rendered, the same voice sings a sixteen-word calibration phrase at the song's tempo (scripts/soundcheck.py). Its timing footprint, per consonant group and for held against short notes, says how many takes the song needs and which words are risky, before any GPU time is spent on the song itself.
• The singer. A score-conditioned singer (SoulX-Singer, Apache-2.0) renders sixteen takes from that clock, locally or on a rented GPU through offrig.
• The pick. scripts/sing_clock.py --by-phrase --warp picks one take per phrase. A local listener's transcript and FCPE pitch decide, and the phrase is time-warped onto the score.
• The gates. Timing: an energy detector, cross-checked by a singing forced aligner, puts every vowel within 40 ms. Pitch: FCPE, with pYIN re-reading what it flags, puts every note within 50 cents.
• The ear. The listening review (scripts/review_marks.py) lets a person press M wherever something sounds wrong, with a category and a note. The report traces each mark to its take, its join and the gates, and weighs it by the reviewer's level: a listener's ear settles where it sounds off, while naming what it is weighs more from training. Today it is a local listening page; it is moving into the cockpit next. Route, levers and receipts: handbook → Vocals, docs/vocal-clock.md; the research behind the choices: docs/vocal-singing-study-2026-09.md.

Guitar

Tool	What it does
view_guitar_tab	Render interactive guitar tablature as HTML — click-to-edit, playback cursor, keyboard shortcuts
list_guitar_voices	Available guitar voice presets
list_guitar_tunings	Available guitar tuning systems (standard, drop-D, open G, DADGAD, etc.)
tune_guitar	Adjust any parameter of any guitar voice. Persists across sessions.
get_guitar_config	Current guitar voice config vs factory defaults
reset_guitar	Factory reset a guitar voice

Build

Tool	What it does
add_song	Add a new song as JSON
import_midi	Import a .mid file with metadata
annotate_song	Write musical language for a raw song and promote it to ready
save_practice_note	Journal entry with auto-captured session data
read_practice_journal	Load recent entries for context
list_keyboards	Available keyboard voices
tune_keyboard	Adjust any parameter of any keyboard voice. Persists across sessions.
get_keyboard_config	Current config vs factory defaults
reset_keyboard	Factory reset a keyboard voice
score_annotation	Score annotation quality across 5 dimensions — completeness, depth, specificity, teaching value, vocabulary
validate_song_entry	Validate a song JSON against the schema before adding
transpose_song	Transpose a song up or down by semitones — new key, new notes
list_sections	View structural sections of a song (Intro, Verse, Chorus, etc.)
add_section	Add a section marker to a song for structural navigation

Score

Tool	What it does
score_performance	Score a MIDI play-along against a library song — pitch, timing, completeness, with graded feedback
score_annotation	Score annotation quality across 5 dimensions

Listen

Measuring recorded audio. Monophonic: they follow one line at a time, so a chord or a full mix produces confident nonsense. Every number comes from signal processing, never from a model reading a picture.

Tool	What it does
analyze_audio	Measure a WAV — onset times, the pitch contour as note names with cents, level (BS.1770-4 LUFS, peak dBFS, clipped samples), spectral balance by band, and an integrity pass (dropouts, noise bursts, clicks)
transcribe_audio	Turn a monophonic recording into notes, with each note's deviation from concert pitch. Notes the tracker could not follow are omitted rather than guessed
score_audio_take	Grade a performance against a library song by ear, then hand the result to view_scored_piano_roll
view_spectrogram	See the sound — a constant-Q spectrogram on a piano-keyboard axis, optionally overlaid with the intended notes. Blind by default
check_loop_seam	Judge where a loop's end wraps back to its start — extrapolated waveform step, burst energy at the join, level shift across it
compare_balance	Compare a render's tonal balance against a reference, band by band in dB, with the change in brightness and tilt. Loudness cancels out
ensemble_now	What every instrument is playing right now, mid-performance. Notes come from what was sent, so they are exact rather than estimated

MCP Prompts

Four prompt templates for structured teaching workflows:

Prompt	What it does
annotate_song	Guided annotation workflow — study an exemplar, write musical language for a raw song
practice_plan	Build a structured practice plan based on genre, difficulty, and goals
performance_review	Review a completed session — what went well, what to focus on next
maker_loop	Walk the full maker loop — propose a reharmonization, verify it with the platform's deterministic tools, then add and play the verified result

CLI

ai-jam-sessions list [--genre <genre>] [--difficulty <level>]
ai-jam-sessions play <song-id> [--speed <mult>] [--mode <mode>] [--engine <piano|vocal|tract|synth|guitar|piano+synth|guitar+synth>] [--metronome] [--count-in <bars>] [--record]
ai-jam-sessions practice <song-id> --measures <start-end> [--start-speed <pct>] [--target <pct>] [--step <pct>]
ai-jam-sessions sing <song-id> [--with-piano] [--engine <engine>]
ai-jam-sessions view <song-id> [--measures <start-end>] [--out <file.svg>]
ai-jam-sessions view-guitar <song-id> [--measures <start-end>] [--tuning <tuning>]
ai-jam-sessions info <song-id>
ai-jam-sessions tune <keyboard-id> [--param value ...] [--reset] [--show]
ai-jam-sessions tune-guitar <voice-id> [--param value ...] [--reset] [--show]
ai-jam-sessions keyboards
ai-jam-sessions guitars
ai-jam-sessions stats
ai-jam-sessions library
ai-jam-sessions ports
ai-jam-sessions help
ai-jam-sessions --version

Status

v2.8.0: two hymns, sung. Amazing Grace and America the Beautiful are sung whole, every verse, on the piano's clock: sixteen takes, one take per phrase, warp placement, two-instrument timing and pitch gates, and a person listening last. Both passed by ear, and the landing page leads with them. The library's America the Beautiful is now Ward's own Materna melody. See CHANGELOG.

v2.7.0 — America the Beautiful. The library is 109 annotated songs. Fifteen MIDI files ship; 94 are still fetched from their sources. The new one is a public-domain arrangement made in this repository (melody by Samuel A. Ward, 1882), in F major, and it is the folk exemplar. A SoulX-Singer route sings it: timing and pitch gates pass, and the landing page keeps the three pitch warnings. See CHANGELOG.

v2.6.0 — the release that stops shipping what it had no licence to ship (see CHANGELOG). A per-file provenance audit of the song library found that of the 108 MIDI arrangements, 14 carry a licence that permits redistribution and 94 do not — and that twelve files were a different piece than their name. Every song now carries an evidence-backed provenance block (source URL, the site's terms, the arranger as the file's own copyright event names them, SHA-256, title verdict); the twelve are quarantined; the npm package ships the 14 and marks the rest unfetched, and ai-jam-sessions library fetch --accept-source-terms downloads each from the site that published it, under that site's terms, refusing any file whose hash no longer matches. Earlier versions shipped all 120 files and are deprecated on npm. The same audit reset the jam-actions-v1 dataset to the eleven songs whose arrangements are verified (three Krueger CC-BY-SA-3.0-DE, eight Mutopia Public Domain); the corpus, its near-gate probe and the training arc that found the shown-work target — a 3B rank-16 LoRA at one unchanged recipe going from a class prior to 54/54 held-out and 72/72 near the gate once the assistant turn wrote the digits of the comparison — are in the repo under datasets/ and experiments/coverage-v1-sft/, with publication to Hugging Face and Zenodo to follow from the verified corpus. Also in this release: scorePerformance caps the correct window at the caller's gate, so the 40 ms house rule is exactly the verdict window.

Previously in v2.5.0 — the release where the model can watch the band play. ensemble_now reports what every instrument is doing while the music is still going: held notes per instrument, how long each has been held, and the combined chord. It runs on two channels, and the cheap one is the accurate one — when this server performs it knows exactly what it sent, so a chord is three note-ons rather than a transcription problem, while a separate acoustic tap measures each engine at the source for verification. Measured cost is about 9 microseconds per audio callback; latency is stated rather than implied (~23 ms pitch, ~70 ms confirmed onset); and the limits are documented because they are actionable — the tracker is monophonic, layered children are tapped individually and never as a mix, and an instrument with no tap is not a silent instrument. The same release turns the dataset machinery into a contract anyone can declare against, with a worked template, so users can build their own corpora and train their own adapters against the same discipline. Along the way the acoustic corpus's reproducibility gate was found to cover 109 of its 115 published paths, and three of the six it missed were never emitted by the generator at all — regenerating deleted them. A full regeneration now reproduces every file and the checksum manifest byte for byte. The live surface is 54 tools and 4 prompt templates, with 3,389 tests passing across 165 files (1 skipped).

Previously in v2.4.0 — the release where the model got ears. Four tools closed the gap between rendering audio and examining it: analyze_audio for onsets, pitch contour and level; transcribe_audio for a monophonic recording as notes; score_audio_take to grade a performance by ear and hand the result to the existing scored piano roll unchanged; and view_spectrogram to see the sound on a constant-Q, piano-keyboard axis. All of it is dependency-free signal processing written in this repo — its own FFT, windows, mel and constant-Q transforms, onset detection and pitch tracking — because a model cannot reliably eyeball a picture and deterministic queries beat inference for questions with exact answers. That release also published jam-actions-acoustic-v0, 108 constructible-gold records of tool use over audio.

Previously in v2.3.0 — the release where the instrument learned to sing on the clock (see CHANGELOG). Any library song can now carry a real sung line that lands on the piano: a score clock derives every syllable's pitch, onset and duration from the song's MIDI on the player's own timeline; a local, Apache-2.0, score-conditioned singer (SoulX-Singer) sings from it; and two gates measure the artifact before anything is called a mix — timing (every vowel within 40 ms of the score) and pitch (every note within 50 cents). The shipped Amazing Grace run measures 6 ms worst timing and −2.7 cents global pitch, with receipts committed; the landing page carries it as an honest state, with the one remaining defect named (the opening splice, handed off). The route, its levers, and the research behind each choice (five study lanes, cited) are in the handbook and docs/. The live surface is unchanged at 49 tools and 4 prompt templates, with 3,080 tests passing (1 skipped) plus the vocal instrument's own pytest suite. Publication state: published — @mcptoolshop/ai-jam-sessions@2.3.0 on npm, provenance-attested.

Previously in v2.2.0 — the release where the instrument got real ears and a listening room. The cockpit's default piano is now a sampled Concert Grand — a pruned Salamander pack that loads on your first gesture and falls back to the tuned oscillator synth until it's ready — and the server picks the sample engine automatically whenever a full pack is installed. On top of it sits the Composition Panel: a blind, loudness-matched A/B listening room where a human ranks the composition engine's voicings against theory-valid and theory-invalid anchors (Bradley-Terry with bootstrap CIs, a MUSHRA-style discrimination floor, PROVISIONAL and UNINTERPRETABLE as first-class outcomes), beside a local-models panel that runs the same ranking with cross-family LLM judges and a Compare view (Kendall τ) that asks whether the cheap proxy tracks the human truth.

The same release carries the composition engine that feeds the panel (src/compose/: a deterministic voice-leading gate with named style presets, membership-by-construction voicing specs, a part-at-a-time refiner), a full health pass (45 findings fixed — security currency, humanized strings, a look-preserving visual amend), Satie and Debussy library entries re-sourced from receipted Mutopia public-domain bytes, and a stranger-test hardening pass: descriptive validation errors, structured {code, message, hint} error envelopes, a curated tarball, progress notifications on long tools, and CLI error grammar. That release shipped as @mcptoolshop/ai-jam-sessions@2.2.0 with 49 tools, 4 prompt templates, and 3,033 tests.

Previously in v2.1.0 — the release where the analyst became a maker. The maker loop ships as product: a model proposes a reharmonization of any library song, and the platform's own deterministic tools gate it — the chord engine must confirm every intended voicing (verify_harmony), every melody note is labeled against the new harmony, and only a verified interpretation goes on to add_song → play_song → view_piano_roll. Generation verified by construction — no rubric, no self-grading; the same inferChord that writes jam briefs is the judge. The maker_loop prompt template walks the whole loop.

Previously in v2.0.0 — the release where the dataset proved its discipline. Breaking: the Node.js floor is now 22 (node-web-audio-api 2.0); the tool surface itself is unchanged — six sound engines, 47 MCP tools, 3 prompt templates, and a fully annotated library: 120/120 songs across 12 genres (12 key fields corrected to content-detected keys this release). The teaching loop is closed end-to-end: metronome with count-in → live recording → per-note scoring → the marked-up scored piano roll → practice loops that ramp tempo only after clean passes. The browser cockpit is a real composition tool — beat-accurate transport with loop regions, record-arm capture, full undo/redo, multi-select and clipboard, touch support — live on the web.

Also publishes jam-actions-v0 — a 57-record training dataset of multi-turn MCP tool-use traces over classical piano, with a 7-axis release gate, cold-start reproducibility, and full Zenodo + CITATION.cff metadata (CC-BY-SA-3.0-DE) — mirrored on Hugging Face, and now carrying receipted fine-tuning results both ways: an honest negative (v0) and a preregistration-disciplined positive that stopped one paired win short of its own victory bar (v1) — see the fine-tuning receipts. This release also fixes the Bach records at the source (working-set revisions r001/r002 with errata) after the v1 pipeline's execution gate caught the published window overshooting BWV 846's actual 62 measures. 2506 tests passing across the MCP server + cockpit + dataset packagers + eval harnesses + release-gate validator. The MIDI is all there, every song can teach, and the corpus of that learning ships with it.

Security & Privacy

Data touched: song library (JSON + MIDI), user songs directory (~/.ai-jam-sessions/songs/), guitar tuning configs, practice journal entries, local audio output device.

Data NOT touched (default paths): the MCP server and CLI make no network calls, read no credentials, and touch no system files outside the user song directory. No telemetry is collected or sent. The opt-in dataset/eval tooling shipped in the same package (scripts/run-llm-eval.ts, provenance verifier) is the one exception: when you explicitly invoke it, it can call LLM APIs (reads ANTHROPIC_API_KEY from your environment, never stores it) and fetch provenance URLs. It never runs as part of the server, CLI, or install.

Permissions: MCP server uses stdio transport only (no HTTP). CLI accesses local filesystem and audio devices. See SECURITY.md for the full policy.

License

MIT