The Roll Is Not the Piano
A player piano roll fixes every note and tells you nothing about the instrument. It turns out that is exactly what a generated score does too.
TL;DR — I tried to build a beat and sing over it. The model handed back the same composition and a completely different performance, and the number that proves it is 0.133.
A player piano roll is a strip of paper with holes punched in it. Every hole is a note: which one, how long, when. Feed the same roll into two different pianos and you get the same music and two entirely different sounds — one bright and clattering, one soft, one badly out of tune.
That distinction turns out to be the whole story of what I spent this week finding out.
The thing every producer does
Make a beat. Get the drums and the bass sitting right, loop it until it feels good, and only then write the melody over the top. That is how records get made. I wanted to know if YuE2, which runs on a single graphics card in my room, could work the same way.
First I asked it for a backing track
The standard advice is simple: leave the lyrics blank and you get an instrumental. So I did.
It sang. Not words — it had none — but a voice, all the way through. When I split the file into voice and accompaniment and measured them, the voice in the supposedly-instrumental version was louder than the voice in the same song generated with a full set of lyrics: an RMS of 0.0934 against 0.0872.
An empty lyrics field does not mean play the backing. It means sing without words. The score still had 371 notes waiting in its vocal line, and the model sang every one of them on a vowel.
Then I tried to keep the backing across two takes
If I cannot extract a beat, perhaps I can repeat one. YuE2 writes out a readable score before it makes any sound — every note, every chord symbol — and you can hand that score back to it. Generate once, keep the score, generate again with words this time.
The score came through untouched.
| Render | BPM | Score length | Vocal notes | Chords | Score SHA-256 |
|---|---|---|---|---|---|
| A — lyrics empty | 116 | 175.9 s | 371 | 85 | 0f5984530f9d |
| B — A's score plus lyrics | 116 | 175.9 s | 371 | 85 | 0f5984530f9d |
| C — control, no score supplied | 116 | 136.6 s | 254 | 64 | 8b7d4a491867 |
The same hash. Not similar — identical, to the byte. The project's own verification tool confirms it with zero differences found.
Then I separated both recordings and asked how alike the backing tracks were.
| Backing stems compared | Waveform correlation |
|---|---|
| A vs B — identical score | +0.133 |
| A vs C — different scores | +0.027 |
| B vs C — different scores | +0.036 |
| reference: same latents, different decoder | +0.954 to +0.991 |
Nought point one three three.
The bottom row of that table is there so you can see what agreement actually looks like. Earlier I took one finished piece of audio and ran it through two different decoders — same performance, different rendering — and the waveforms agreed at 0.954 and above. Two renders of a byte-identical composition agree at 0.133, which is barely distinguishable from two unrelated songs at 0.027.
Even the frequency balance moves. Sub-bass drops from 23.9 to 18.4 percent of the energy while the band just above it climbs from 37.9 to 45.8. The kick and the bass are simply sharing the low end differently. It is a different band playing the same chart.
The sentence I had already read
"Editing style, lyrics or ABC requires new semantic generation and synthesis. Cached latents are reusable when changing only the decoder, not when changing the music."
That is from the project's own documentation and I read it before I started. I still had to measure it to believe it, which is probably the most honest thing in this piece.
Supplying a score fixes what is played. It does not fix how it sounds. The roll is not the piano.
What you can actually do
What it will not do. Take audio as an input, in any form. Render a backing track without a voice on it. Reproduce a backing track across two takes, even from a byte-identical score. Hand you the parts as separate files - one render, one mixed file, and nothing in the runtime that would emit anything else.
What it will do. Hold a melody and a harmony exactly, verified to the byte. Change one passage and leave every other one alone, measured to the hertz.
The bottom two rows are not nothing. In an earlier experiment I opened a finished song's score, raised thirty-two notes in the last chorus, and re-rendered — and that passage, and only that passage, came back higher, with the untouched sections measuring identical frequencies. The control is real and it is surgical. It just operates on the composition and stops there.
If you want the producer's workflow you are left with two options. You can render twice and glue one take's backing under another take's vocal, which works and is editing rather than making. Or you can use a different model — one that will take audio as an input or give you the parts as separate files. YuE2 does neither, and there is nothing in its code that suggests it was ever meant to.
It writes songs and performs them. It does not hand you the tracks.
If you want to try this yourself
The calls, the numbers, and exactly what supplying a score buys you.
Reproducing the test
# A - ask for an instrumental the way every guide says to
pipe(style=STYLE, lyrics="", cot="full", seed=871001).save_artifacts("out/A")
# B - inherit A's score exactly, add the words
abc = Path("out/A/score.abc").read_text(encoding="utf-8")
pipe(style=STYLE, lyrics=LYRICS, cot="full", seed=871001, abc=abc).save_artifacts("out/B")
# C - control: same style, same lyrics, same seed, no score supplied
pipe(style=STYLE, lyrics=LYRICS, cot="full", seed=871001).save_artifacts("out/C")
# confirm B really inherited A
python skills/yue2-music/scripts/abc_tools.py compare out/A/score.abc out/B/score.abc --voices Vocal
# -> {"match": true, "differences": []}
# split each render and correlate the backing stems
python -m demucs --two-stems=vocals -n htdemucs -o sep out/A/audio.flac out/B/audio.flac
The numbers
vocal-stem RMS A (no lyrics) 0.0934 C (lyrics) 0.0872
vocal share of energy A 38% C 41%
backing-stem correlation
A vs B (identical score) +0.133
A vs C (different scores) +0.027
B vs C (different scores) +0.036
same latents, different decoder +0.954 to +0.991 <- what 'same performance' looks like
backing band energy, 20-60 / 60-120 / 120-250 / 250-2k / 2k+ Hz
A 23.9 37.9 26.1 8.5 3.6
B 18.4 45.8 19.8 12.0 4.0
The kick-versus-bass balance moves by five to eight percentage points between two renders of the same composition. That is not a re-mix of one performance; it is a second performance.
What the score does and does not fix
FIXED by supplying abc= every pitch, every duration, every chord symbol, meter, tempo,
section boundaries - verified byte-identical by the project tool
NOT fixed instrument balance, groove feel, mix, the actual performance;
changing style, lyrics or abc re-runs semantic generation
and synthesis from scratch
The documentation says this outright: "Editing style, lyrics or ABC requires new semantic generation and synthesis. Cached latents are reusable when changing only the decoder, not when changing the music." I read that sentence before running the test and still had to measure it to believe it.
Further reading: the model is YuE2; the sentence this whole piece turns on lives in its generation and covers reference, and the pipeline it describes is documented here.
Image: William Braid White, sectional illustration of player piano interior action, 1909, Public domain, via Wikimedia Commons.
← Back to all posts