Show HN: I trained a 125M model to autocomplete piano on-device
Show HN: I trained a 125M model to autocomplete piano on-device
TL;DR: I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The biggest improvements came from finding the right MIDI representation, cleaning the training data aggressively, and adding DPO post-training.
简而言之： 我训练了一个 125M 参数的 Transformer 模型，用于实时自动补全钢琴演奏（在 iPhone 15 上约为每秒 108 个音符）。最大的改进来自于找到合适的 MIDI 表示方式、对训练数据进行严格清洗，以及在训练后加入了 DPO（直接偏好优化）。
Almost a year ago, I started tinkering with an idea: connect my MIDI piano to my phone, play something, and have AI autocomplete the song for me. Think GitHub Copilot, but for piano. It turned out to be a deeper rabbit hole than I expected. Fourteen experiments later, it is finally at a point where I am happy enough with it to write about.
大约一年前，我开始琢磨一个想法：将我的 MIDI 钢琴连接到手机上，当我弹奏时，让 AI 为我自动补全乐曲。就像钢琴版的 GitHub Copilot。事实证明，这比我预想的要复杂得多。经过十四次实验，它终于达到了让我满意并可以分享的程度。
The app, RollTab, is available for free here if you have a MIDI keyboard and an iPhone/iPad.
如果你有 MIDI 键盘和 iPhone/iPad，可以在这里免费下载这款名为 RollTab 的应用。
What’s in a MIDI File?
MIDI 文件里有什么？
A MIDI file is quite different from an MP3 or other audio formats. Rather than storing recorded sound, it stores music as a sequence of events: a key is pressed at a certain pitch and velocity, a key is released, the sustain pedal changes state, and so on. Other events include switching instruments or changing volume. These events are often organised into multiple tracks. A pop or game MIDI might have melody, chords, bass, drums, strings, and several synth parts. This project is focused on piano continuation, so I mostly kept piano-like material and removed or reduced the rest.
MIDI 文件与 MP3 或其他音频格式截然不同。它存储的不是录制的音频，而是将音乐存储为一系列事件：按下某个音高和力度的键、松开键、延音踏板状态改变等。其他事件还包括切换乐器或改变音量。这些事件通常被组织成多个轨道。流行音乐或游戏 MIDI 可能包含旋律、和弦、贝斯、鼓、弦乐和多个合成器声部。本项目专注于钢琴续写，所以我主要保留了类似钢琴的素材，并移除或精简了其余部分。
How Do You Tokenize Music?
如何对音乐进行分词（Tokenization）？
To train a transformer on these performances, I first needed to turn the MIDI events into a discrete sequence the model could read and predict. The most obvious mapping is to make a token for every MIDI event:
NOTE_ON_60_80 # {pitch}_{velocity}
NOTE_OFF_60 # {pitch}
TIME_SHIFT_12 # {time step}
为了在这些演奏数据上训练 Transformer，我首先需要将 MIDI 事件转换为模型可以读取和预测的离散序列。最直观的映射方式是为每个 MIDI 事件创建一个 Token：
NOTE_ON_60_80 # {音高}_{力度}
NOTE_OFF_60 # {音高}
TIME_SHIFT_12 # {时间步长}
If you include pitch and velocity directly in a NOTE_ON token, the vocabulary can grow quickly. There are 128 MIDI pitches and 128 velocity values, so the naive combined note-on vocabulary has up to: 128 * 128 + 128 = 16,512 tokens just for note-on and note-off. In practice you would probably bucket velocity, but the basic issue remains: many combinations are rare, and the model has to learn a lot of structure from sparse tokens.
如果你直接将音高和力度包含在 NOTE_ON Token 中，词汇表会迅速膨胀。MIDI 有 128 个音高和 128 个力度值，因此简单的组合式 note-on 词汇表仅 note-on 和 note-off 就多达 128 * 128 + 128 = 16,512 个 Token。在实践中，你可能会对力度进行分桶，但基本问题依然存在：许多组合非常罕见，模型必须从稀疏的 Token 中学习大量的结构。
A common improvement is to factor the representation with a grammar:
[NOTE_ON, PITCH, VELOCITY] | [NOTE_OFF, PITCH] | [TIME_SHIFT, DURATION]
Now the output spaces are smaller:
NOTE_ON / NOTE_OFF / TIME_SHIFT
PITCH: 128 values
VELOCITY: ~16
DURATION: ~100
一种常见的改进方法是使用语法对表示进行分解：
[NOTE_ON, PITCH, VELOCITY] | [NOTE_OFF, PITCH] | [TIME_SHIFT, DURATION]
现在输出空间变小了：
NOTE_ON / NOTE_OFF / TIME_SHIFT
PITCH：128 个值
VELOCITY：约 16 个值
DURATION：约 100 个值
You can enforce the grammar during generation by masking invalid next tokens. After NOTE_ON, only pitch tokens are valid. After pitch, only velocity tokens are valid. This guarantees syntactically valid output.
你可以在生成过程中通过屏蔽无效的下一个 Token 来强制执行语法。在 NOTE_ON 之后，只有音高 Token 是有效的；在音高之后，只有力度 Token 是有效的。这保证了输出在语法上的正确性。
I tried note-on/note-off style representations, but my models tended to drift. They would forget to emit note-off, leave hanging notes, or lose track of active state. That was especially bad for my target: a small model running close to real time on a laptop or phone.
我尝试过 note-on/note-off 风格的表示，但我的模型往往会产生偏差。它们会忘记发出 note-off，导致音符持续不断，或者丢失对当前状态的追踪。对于我的目标——在笔记本电脑或手机上近乎实时运行的小型模型来说，这尤其糟糕。
The final representation
最终的表示方式
The representation I eventually settled on was: NOTE(pitch, delta_onset, duration, velocity)
There is no separate TIME_SHIFT event in the final version. Silence is represented by delta_onset on the next note: the time since the previous note onset.
我最终确定的表示方式是：NOTE(pitch, delta_onset, duration, velocity)
最终版本中没有单独的 TIME_SHIFT 事件。静音由下一个音符的 delta_onset 表示：即距离上一个音符开始的时间。
Instead of spending four transformer passes generating the attributes of a note, the transformer advances the music by one complete note at a time. In practice, this gets the large model to about 108 notes/second on an iPhone, well above anything a human would need for live playing.
Transformer 不再需要通过四次传递来生成一个音符的属性，而是每次推进一个完整的音符。在实践中，这使得该模型在 iPhone 上达到了每秒约 108 个音符的速度，远超人类现场演奏所需的速度。
Sustain Pedal
延音踏板
As you might know, pressing down the sustain pedal on a piano makes notes play even after you release them. I didn’t want to muddy the implementation with adding sustain pedal events. Instead, sustain is baked into note duration during preprocessing. If the key is released while the sustain pedal is down, the note is extended to the pedal-up time. If the same pitch is played again first, the earlier note is cut off at the retrigger. The result is a note duration that approximates the actual sounding duration. This loses the explicit pedal gesture, but it makes the modeling problem much simpler: the model only…
如你所知，按下钢琴的延音踏板会使音符在你松开键后继续发声。我不想通过添加延音踏板事件来使实现变得复杂。相反，我在预处理过程中将延音效果融入了音符持续时间中。如果键在踏板按下时被松开，音符会延长到踏板抬起的时间点。如果同一个音高先被再次弹奏，之前的音符会在重新触发时被切断。结果就是音符的持续时间近似于实际的发声时长。这虽然丢失了明确的踏板动作，但大大简化了建模问题：模型只需要……