The Same Mistake, Twice in One Week

TL;DR — I published a finding about AI music generation, replicated it properly, and discovered I had been reporting the outcome of a coin flip. Here is what twenty runs say instead — and the one capability that behaved exactly as promised.

Three days ago I wrote that a long, detailed style prompt beat the short one the documentation recommends. I had the numbers right there: 47 percent against 25 percent. It felt like a finding. It was a coin flip.

I know that now because I ran it again — four prompt lengths, five seeds each, twenty generations with everything else nailed down. And the thing that makes this worth writing about is not that I was wrong. It is how I was wrong, and that it was the second time the same week.

The shape of the mistake

Here is the replication.

Style prompt	Lyric fidelity (5 seeds)	Mean	Song length
27 words	32 · 34 · 39 · 43 · 65	42.6%	128.0 s
65 words	20 · 33 · 42 · 47 · 65	41.4%	136.8 s
124 words	26 · 39 · 46 · 49 · 84	48.8%	145.0 s
233 words	26 · 39 · 49 · 61 · 78	50.6%	138.8 s

Look down the columns rather than across them. The four averages sit between 41 and 51 — a spread of nine points. The scatter inside each row is far wider: one group runs from 26 to 84. The average standard deviation within a group is 17.9 points, against a 9.2-point difference between groups.

The noise is twice the signal. Any single pair of runs will show you whatever you want to see, and three days ago a single pair showed me something I wrote down as true.

The same mistake, the first time

Earlier in the week I had watched eight generations of one song — identical lyrics, identical prompt, only the random seed different — score 0, 31, 44, 45, 50, 57, 66 and 70 percent. I even published that. I understood, in the abstract, that a single draw from this model tells you almost nothing.

Then I ran a two-sample comparison and believed it anyway.

A thing that worked exactly as advertised

Against that, here is a test that came out clean. YuE2 writes a readable score before it makes any sound, and you can edit that score and hand it back. I took a finished song, opened its score, and raised thirty-two notes in the final chorus.

I separated the vocal from both renders and measured the pitch window by window. Between 2:05 and 2:18 — the passage I edited — the original sits at 417 Hz and the edited version at 463, a rise of 1.8 semitones. Between 1:40 and 2:00: 415 Hz and 415 Hz. Between 0:00 and 1:00: 415 Hz and 415 Hz.

Identical to the hertz, everywhere I did not touch. Thirty-two numbers changed in a text file, and thirteen seconds of a two-and-a-half-minute song moved, and nothing else did.

I meant to raise it a clean two semitones and got 1.8, because my own transposition script misread a few pitches. The notation propagates an accidental across octaves by letter name — an unusual rule the documentation warns about in bold — and I had not implemented it. The feature was fine. My tool was not.

The cover that sang in a language I never gave it

I have been trying to cover released songs: take the original melody, put new lyrics on it, let the model rebuild the arrangement. It kept coming out wrong, and I had assumed the problem was the arrangement.

The lyrics I supplied are Korean. What the transcription returns is English-shaped nonsense — "Come on, wing go go."

The cause turned out to be arithmetic. That melody carries 643 notes. My lyric sheet carries 185 syllables. That is three and a half notes for every syllable I gave it, and the model fills the gap with whatever sounds like singing. A second cover, at 2.4 notes per syllable, kept 14 percent of the lyric instead of 5.

A real pop melody is much denser than a lyric sheet with section tags. Covering one means writing to its syllable count, not borrowing a lyric from somewhere else.

Two things that turned out not to matter

The model ships two audio decoders — one for listening, one that its published benchmark used. I had wondered whether we were hearing something different from what was scored. We are not: decoding the same cached audio through both, band energy differs by at most four tenths of a percentage point and the waveforms correlate above 0.95.

And I ran all eighteen of my scores through the project's own validator expecting to find the structural problem I had flagged earlier. All eighteen passed. That concern was mine, not the format's.

What I actually take from this

The useful result of the day is dull and practical: the score predicts the finished song's length to within two percent, across sixteen songs. Planning a score takes twenty seconds and rendering takes two minutes, so you can look before you pay.

But the thing I will remember is that I ran a two-sample test three days after measuring, and publishing, that this model's seed-to-seed variance swamps almost everything else. Knowing a fact and using it are different skills.

FAQ

Why did the first result turn out to be wrong?

It came from a single pair of generations. Replicating it with five seeds per condition showed a within-group standard deviation of 17.9 points against a between-group difference of 9.2 — the variation between identical runs was larger than the effect being claimed.

Can you edit part of an AI-generated song without regenerating the whole thing?

With YuE2, yes, because it exposes the score it wrote. Editing 32 of 276 notes moved a 13-second passage up 1.8 semitones and left two control windows measuring identical frequencies.

Why do AI song covers ignore the lyrics you give them?

Usually because the original melody has far more notes than the new lyric has syllables. At 3.5 notes per syllable the model discarded the supplied Korean lyric and sang invented English-sounding syllables instead.

How many times should you generate before judging an AI music model?

More than twice. Eight runs of one song, differing only by random seed, scored anywhere from 0 to 70 percent on how faithfully the lyrics were sung.

If you want to try this yourself

Every prompt and setting this piece describes, ready to copy.

The request

{
  "id": "kongkong",
  "style": "Korean, K-pop girl group dance, bright confident female vocals, 126 BPM, punchy kick, deep synth bass, crisp claps, glossy synth plucks, catchy repeated hook, no electric guitar",
  "lyrics": "[Intro]\n\n[Verse]\n...",
  "cot": "full",
  "seed": 831001
}

cot is full (melody and chords), melody (melody only) or off (no score). There is no bpm, negative_prompt or reference_audio field. Put the tempo in the style as a single number, never a range: a range breaks it, a number is followed to within about 3.5 BPM.

Lyric structure

[Intro]          <- tag only, no words: an instrumental opening
[Verse]
[Pre-Chorus]
[Chorus]         <- write the hook TWICE inside the block
[Interlude]      <- tag only
[Pre-Chorus]
[Chorus]
[Bridge]
[Chorus]
[Outro]

Aim for 1.0 to 1.3 notes per syllable. Below that the model raps the line; above it, the surplus notes come back as vocalisations. At 3.5 it abandons your lyric entirely. Keep the lyrics in one language - mixing two inside a line measured worst of every condition tested.

The four style prompts from the replication

27 words  (the compact form the documentation recommends)
Korean, K-pop girl group dance, bright confident female vocals, 126 BPM, punchy kick,
deep synth bass, crisp claps, glossy synth plucks, catchy repeated hook, no electric guitar

65 words  = the 27-word prompt, plus:
Wide modern club mix with a lot of low end and a clean bright top. Short supersaw stabs
answer each vocal line. Hand claps on the backbeat, shakers running underneath, and a
short filter build into every chorus.

124 words = the 65-word prompt, plus:
The verses sit half-time and double into the chorus. Pitched vocal chops work as a
percussion layer between phrases. The bass line glides between notes rather than stepping.
Reverb is short and dry on the lead so the consonants stay crisp, and the stacked group
vocals open wide only on the hook. Nothing is shouted and nothing is busy.

233 words = the 124-word prompt, plus a full arrangement script:
Arrangement events: the intro states the hook motif on a muted pluck alone; the first verse
strips to kick, sub and claps; the pre-chorus removes the kick for two bars and brings a riser;
the first chorus lands with the full stack and a sub drop on the downbeat; the post-chorus keeps
only the chant and the drums; the bridge drops to pad and voice alone before the final chorus
returns doubled with an octave-up vocal layer.

Check the length before you pay for a render

plan = pipe.plan(style=STYLE, lyrics=LYRICS, cot="full", seed=831001)
# the score nominal duration predicted the final audio within 2% on 16 of 16 songs.
# planning ~20 s, rendering ~120 s - so reject a short score here, before it costs anything.

Running a 24 GB pipeline on a 16 GB card

# 1. Windows torch wheels ship without a FlashAttention kernel, but the auto-detection only
#    checks that the operator exists - so it picks flash and dies at execution.
from yue2 import cuda_graph, nar
_g = cuda_graph.GraphAR.__init__
cuda_graph.GraphAR.__init__ = lambda s, *a, **k: _g(s, *a, **dict(k, attention_backend="cudnn"))

# 2. NAR attention hands the whole query sequence to SDPA at once (3.77 GiB in one allocation).
_n = nar.CachedNAR.__init__
nar.CachedNAR.__init__ = lambda s, m, c, a="sdpa", q=None: _n(s, m, c, a, q or 2048)

pipe = YuE2Pipeline.from_pretrained(MODEL, vae=VAE, device="cuda", local_files_only=True,
                                    memory_budget_gib=15, offload_ar=True)

Score tools that ship with the project

python skills/yue2-music/scripts/abc_tools.py inspect score.abc
python skills/yue2-music/scripts/abc_tools.py strip-chords source.abc cover.abc --keep-voice Vocal
python skills/yue2-music/scripts/abc_tools.py compare source.abc edited.abc --voices Vocal

inspect reports the nominal duration. strip-chords is what a melody-only cover needs, because cot="melody" does not remove chord symbols from a file you supply. compare names the exact note index at which an edit starts.

---

Further reading: the YuE2 project publishes its benchmark protocol, its generation guide, a reference on its ABC dialect, and the code itself.

Image: Klaus-Dieter Keller, Galton box, Public domain, via Wikimedia Commons.

---

← Back to all posts