Stapes - Lixen

In development

Synthesize and sequence music in a terminal, and rip recordings into patterns and voices. Android and sheet music are next

Go Python FreeBSD Linux TOML MIDI AI Genetic Algorithms Machine Learning Transformer Terminal TUI Audio

How a rip works Build and run GitHub

About

Stapes is the workbench of a pure-Go synthesis and music engine. The same engine that plays ViF’s sound and composes its music in gameplay. Everything the engine plays, from a one-shot effect to a drum kit, a voice or a pattern, is a TOML document Stapes opens, edits and plays. It runs as a command line, a full-screen terminal UI or a script. It is named for the ear bone that carries sound inward.

Stapes Rip reads an audio file or listens to sounds and music, and writes what Stapes needs to play it back, in the engine’s own vocabulary: the tempo and the bars, the key, the drums, the melody and the bass as patterns, the voices that play them and the song that plays the bars in order. The Rip tab draws how.

Documents: An edit changes the document in memory; apply hands it to the running engine, where the next note takes it, and revert brings the engine's copy back. undo steps back over the last hundred edits.
Paths: Any field is reached by its dotted path, read through the TOML tags the loader reads: set blip.layer.0.source.freq 880 changes exactly what the file would. A zero value means the engine's default, so a saved document holds only what was set.
Checks: The command line checks a value's type and width; the engine's own rules, a frequency under Nyquist or an envelope's length, run at validate and apply.
Slots: The sequencer plays four, rhythm, melody, fill and bass, as the game draws them. A pattern takes its slot on the next bar, and a slot or a track mutes or solos while its pattern plays on in time.
Voices: The tonal instruments are entries of the sound document, made, edited and applied like a sound; a new name is a new instrument any pattern's track can play.
Scripts: A .slab file is one command a line: # comments, - before a line tolerates its failure and ! expects one. A recipe reads its files beside itself, so it runs from anywhere.
MIDI: A pattern or a song is written as a MIDI file, each voice on its nearest General MIDI program, and a MIDI file opens as patterns and a song, each part on the built-in voice for its program.
Home: Stapes has a repository of its own, and its engine is a library it shares with ViF. Its root, ~/.config/stapes, holds its settings, the rip's tuned thresholds and RipModel's weights. It opens the sound and music documents in ViF's folder, so what it saves, the game plays next.
Next: An Android build, inside ViF's Android app; a view that shows patterns and songs as sheet music; and RipModel's training and tuning, on more real recordings.

How a rip works A rip, end to end

Rip

Stapes Rip turns a recording into the engine’s own vocabulary: the tempo and the bars, the key and each bar’s chord, the drum hits, the melody and bass lines as patterns on a sixteenth-note grid, the voices that play them, and a song that plays the bars in order. The result plays right away in Stapes and in the game, and Rip checks it against the recording, by ear and by score.

Rip has two parts. Ripping turns a recording into patterns. Training teaches RipModel, a transformer, to hear recordings: from music the engine renders itself, NSynth’s recorded instrument notes, E-GMD’s drum performances and recordings made on a phone. Rip hears with the built-in analysis or with RipModel, and both feed the same decoders.

What each stage runs on

TransformerRipModel, which hears the frames
GAa genetic search: a pool of candidates bred toward a score
DSPspectra and filters, and the effects added in training
Decodingdynamic programming, Viterbi paths, peak picking
Renderthe engine renders a candidate to compare

Pick a stage to read it below

Ripping

Recording A WAV, or a MIDI score

1HearTransformerDSP Frames of onsets, drums and 69 pitches

1Hearfrom sound to frames

The recording is read at 22.05 kHz and cut into frames: a 93 ms window every 11.6 ms, about 86 a second. Its tuning is measured first, in cents off A440, so a recording forty cents flat still reads as in tune. Then a front end turns each frame into the same set of fields, so the decoders work the same whichever front end heard it.

DSPThe analysis

Split each frame into its harmonic and percussive parts, by medians across time and across frequency
Drums each in its own band: a kick's rise under 100 Hz, a hihat's over 5 kHz, a snare between
Pitches by salience, the sum of a pitch's partials; the bass from a window four times longer, which tells semitones apart at 110 Hz

TransformerRipModel

Frames 229 log-mel bands from 30 Hz, each in dB against the frame's own level, and the level
Linear 230 wide to 256
4 layersAttention 8 heads, 64 frames either sideFeed-forward 1024 wide, GELU
Heads per frameonset 1drums 3pitch 69voicing 1strike 69release 69velocity 69wave 5genes 32

How far each head looks around its frame. Each of the eight weights nearer frames by its own slope (ALiBi): one stays on an attack, another spans the whole window. Nothing past 64 frames counts.

Fields: Per frame: the onset; the kick, snare and hihat; the salience of 69 pitches, E1 to C7; each register's chroma.
Model only: Which pitches are struck and released, how hard, and the wave and 32 genes of each note's voice.
Weights: 3.2 million, run in pure Go in float32. Rip uses RipModel when Stapes' root holds ripmodel.bin, and the analysis otherwise.

2ReadDecoding Tempo, bars, key, drums, two lines

2Readfrom frames to music

The decoders are shared, classical and cheap. Their thresholds live in RipParams: tune searches them, and Stapes' root keeps them in rip.toml.

One bar: the salience arrives late and leaks into the pitch above. The path stays on a pitch until moving pays off, and its notes land on sixteenths.

Tempo: The autocorrelation of the onsets, with a prior centred on 120 bpm, within the sequencer's 80 to 200.
Beats: Dynamic programming that keeps to that period.
Bars: The bar line falls where the downbeats carry a kick and the key's root in the bass, with the snare on two and four. A phrase that starts early gets a pickup bar.
Key: The root and scale whose tones hold the most chroma, with the bass voting for the tonic, and a chord root for each bar.
Drums: Peak picking: a kick by its rise under 100 Hz, a hihat by its rise over 5 kHz, a snare between; a hit can be two drums at once.
Notes: A Viterbi path through the salience, with a cost to move and to start or end a note. Each note is then set on a sixteenth and the key's nearest degree, a leftover semitone kept as an accidental. With the model, a melody note starts where its pitch is struck and lasts until it is released, any number at once.
Lines: The melody register, F3 and up, is read as two lines at once, so a slow line under a pluck is heard too. The bass goes up to A3.

3WriteGARender Patterns, their voices, a transcript

3Writefrom music to patterns

Each part becomes patterns of up to four bars, ready to play, and the transport takes the tempo, key and tuning. Each line gets a voice. With the model, its voice head predicts the genes of every note, averaged over the line. A line with no voice heard is fitted by ear: a genetic search judged by the engine's renders.

Renderthe line's six longest, loudest notes, at the recording's pitch, loudness and tuning
Compareeach render's mel frames with the recording around its note
Breedtournaments of three, uniform crossover, bounded mutation

16 voices a generation for 8 generations, starting from every registered voice. Mutation widens while the best improves and narrows when it stalls; no voice is rendered twice.

A voice is a wave and 32 genes, each from 0 to 1 on a perceptual scale: oscillators, string and partials, filter, envelope, and the output drive and level. Shown: the built-in piano.

Patterns: <name>_beat_1, _melody_1 with the second line as a track of its own, _bass_1, and on.
Voices: <name>_melody_voice, _under_voice and _bass_voice, each then set to the recording's loudness, which the mel frames do not capture. fit runs a longer search.
Transcript: Every note in seconds, before it is set on the grid, with how strongly the recording supports it. notes writes it as a MIDI file.

4CompressGA A dictionary of bars, and a song

4Compressbars into a song

Each part gets as few one-bar patterns as its bars allow. The distance between two bars is the share of their steps they do not have in common. Bars within 5% of each other form a group, and the group's middle bar, the medoid, stands for all of them.

Dictionaryabc

Songaabacaba

Eight bars of a beat: bars 2 and 7 each have one extra kick, within 5% of bars 1 and 3, so three patterns, <name>_beat_a, _b and _c, play all eight.

Group: Each bar starts as its own group; the two nearest groups merge while their mean distance stays within the threshold.
Search: A genetic search over which bars share a pattern: 32 assignments for 40 generations, scored by the number of patterns used, with a heavy penalty for any bar too far from its pattern.
Song: The pattern each bar plays, at the recording's tempo, key and tuning; song name play plays it live. Its render is ripped again and scored against the first rip, which measures what compressing lost.

5RefineRenderGA Render, compare, keep what sounds closer

5Refinerender, compare, keep

A rip is good when its render sounds like the recording. Refining works in rounds, and a change is kept only when the bars that play it sound closer to the recording.

Changea voice, by a genetic search, or one step of a line
Renderthe bars of the song that play it
Comparerender and recording, through one front end, bar by bar
Keepwhat sounds closer

Three rounds, or until a round gains too little.

Voices: Searched again by the whole song's score, starting from the current voice and every registered voice. No voice may sound more than a tenth further from the line's notes than its fit.
Steps: Each tonal step is tried a semitone and an octave up and down, at the recording's strongest pitch there, a step earlier or later, longer or shorter, and silent. An empty step tries the strongest pitch, which is how a missed note gets added.
Score: Pitch salience, mel bands and onsets, each compared by cosine similarity, with the frames paired bar by bar on each side's own beats.

6CheckRenderGA Scores, a second rip, the bench

6Checkhow good a rip is

truth: Scores a rip against known notes: a hand-corrected transcript, a tapped rhythm, or a MIDI file, whose drum channel scores the drums hit by hit. Per line, the notes matched, missed and extra, and how many are heard at each loudness, in 6 dB bands.
agree: Compares two rips of the same piece when no score exists.
bench: Rips new loops the engine renders, whose notes are known, clean and through a room and a mix, and scores the analysis and the model side by side.
tune: A genetic search over the decoders' thresholds, each candidate scored by how well it rips rendered loops.

Not told apart yet: a bass note's attack from a kick, a plucked chord from a snare, a key change from accidentals, and a third line or a pedalled chord, which two lines cannot hold. Soft notes are missed more often than loud ones.

Training

GatherRender Rendered and recorded loops, labelled

Gathertraining data with exact labels

Most of what the model learns from is music the engine renders itself. The engine knows exactly what it played, so the labels are exact and free. Real sound comes in too: NSynth's recorded instrument notes, E-GMD's drum performances and recordings made on a phone.

One rendered bar and the labels kept with its frames: every drum hit, every note's pitch and length, where each is struck and released, and the voice that plays it.

Loops: Four bars each, from the bank's rhythm, melody and bass patterns, across the sequencer's tempos and keys; twenty thousand by default.
New lines: For each part, three loops in five play a beat, melody or bass line newly drawn from the key, with a broken chord in the free slot.
Soft, bare: A third of the generated melodies and accompaniments play half their length 6 to 18 dB softer. Three generated loops in ten drop the drums, and one in seven drops the bass.
NSynth: Half the tonal lines can play on NSynth's recorded notes in place of the engine's voices, each note shifted from the nearest recorded pitch and cut at its release.
E-GMD: Drum performances recorded with the MIDI they were played from, each a loop labelled by its MIDI; a set share of the training windows comes from them.
Labels: Per frame: the onsets, the drums, the pitches sounding, where each is struck and released and how hard, and every note's wave and 32 genes.

RoughenDSP A room, a mix, new voices

Roughenso it learns recordings, not the synthesizer

Each loop goes through a room and a mix drawn for it alone, in this order, so the model learns what recordings sound like, not the engine's clean output. The labels stay the notes as played.

Gain and swell −6 to +3 dB, swelling by up to 6 dB over the loop
Low-pass one pole, 4 to 11 kHz, in half the loops
Equaliser two peaking bands, 80 Hz to 8 kHz, ±6 dB
Soft clip a tanh drive of 1.5 to 3.5, in a third
Compressor 2:1 to 6:1, in half
Reverb Schroeder's four combs into two allpasses: a booth, or in a third a larger hall
Noise floor 40 to 65 dB under the mix, in half

Voices: Redrawn for each loop near the registered voices: every gene nudged at random, about a third switched to another wave, some driven hard, as a guitar through an amp.
Pedal: In a third of the loops, every release except the bass's is three to ten times longer, as with a piano's sustain pedal.
Tuning: Off A440 by up to a quarter tone, as real recordings can be.
Codec: On request, a third of the loops go through Opus, MP3 or AAC and back, at 64 to 192 kbps, as recordings are stored.

TrainTransformer Trained in PyTorch, run in Go

Trainin PyTorch, for Go

tool/rip-train holds a PyTorch copy of the model, outside the Go module. It trains on frames that Go computes, so both sides hear the same input, and writes the weight file the Go model loads; the two agree to float32 rounding.

Windowssix seconds of a loop, 512 frames, eight per step
Losscross-entropy weighted toward rare strikes and onsets, squared error for genes and velocity
Held out200 loops scored every 1000 steps, the best step kept
Benchships only when it beats the analysis, on rendered loops and on real recordings

A round: pipeline.sh renders the corpus in parallel parts, trains, benches and installs: 40000 loops and as many steps in about four hours on a laptop's GTX 1060. Across machines, each trains a shard, and their weights are averaged into the next round's start.
Shape: The trainer can widen or deepen the model, add convolutions before its layers, or split them into towers for groups of heads. The weight file records the shape, and Go builds whatever it reads.

Weightsripmodel.binloaded by stage 1, Hear

Engine

The engine is Audio, a Go library ViF plays too. Stapes edits what it plays and listens to what it renders.

Synthesis

Pure Go, 44.1 kHz stereo, no CGO. A sound is layers of oscillators, sweeps, FM, noise and impulses, shaped by filters, envelopes, ring modulation and waveshaping and mixed on buses. Each sound is rendered ahead in several variants, so repeated hits differ slightly, and a drum renders a layer per velocity, so a harder hit is brighter as well as louder.

Voices

An oscillator stack through a resonant low-pass with its own envelope, then an ADSR; or a string, modelled as a delay line or as its partials. Any voice is 32 genes on perceptual scales: the vector RipModel predicts and the genetic search evolves.

Patterns and harmony

Drum hits and notes on a sixteenth-note grid. Each note is a degree of the key, with an accidental when it falls outside the key, resolved against a root, a scale and a chord progression as it plays. TOML macros define a drum lane or a line once, for reuse.

Sequencer

Four crossfading slots: rhythm, melody, fill and bass. Changes land on the next bar line, and tempo, swing and tuning follow the transport. A song plays bar by bar, each bar’s patterns scheduled while the previous bar plays.

Live registry

One command channel feeds one mixer goroutine, which keeps playback, transport and hot reloads in order. When a definition is replaced mid-note, the sounding note finishes with the old one and the next note uses the new one.

Offline and seeded

The same engine renders a note, a pattern or a song to a buffer. Rip uses this to fit voices, score songs and make its training data. Every random draw is seeded, so a render, a rip and the model’s output come out the same each time.

Workflow

A rip takes a few commands, typed at the command line or in the TUI. Each result is a document entry like any other, to play, edit and save.

Rip: rip song.wav name writes the patterns, voices and transcript, and sets the transport's tempo, key and tuning. rip song.mid score does the same from a MIDI score, without listening.
Listen: slot the patterns, or render them to a WAV. notes name shows the transcript and writes it as a MIDI file, to correct in a sequencer.
Check: truth name score.mid scores the rip against a score, a hand-corrected transcript or a tapped rhythm. agree compares two rips of the same piece.
Compress: compress name writes the dictionary of bars and the song. song name play plays it through the live sequencer.
Refine: refine name renders the rip, compares it with the recording and keeps what sounds closer. fit name refits only the voices.
Keep: Save the patterns, the song and the voices, as below, or write the song as a MIDI file with song name song.mid. Put the whole recipe in a .slab file and run it.

rip song.wav bite nodrums        # the patterns and fitted voices; nodrums: no kit
compress bite                    # the dictionary and the song bite
song bite play                   # plays it; music stop ends it
save pattern bite_patterns.toml  # the patterns and the song
write bite_melody_voice bite_melody_voice.toml
write bite_bass_voice bite_bass_voice.toml

In the TUI

The TUI runs the same commands as the command line: its keys build and run command strings, recorded takes included, so a script can repeat anything done in it.

Browse: Patterns, sounds and voices beside the mixer. space plays a sound or a voice's middle C, or puts a pattern in its slot; 0 to 3 pick the slot.
Mixer: Every slot on one timeline under a shared playhead. Slots and tracks mute, and tracks solo, while their patterns keep playing in time.
Editor: A keyboard that records takes onto the key's degrees, a step grid with velocities, chances and lengths, and a piano roll. A new pattern loops in its slot before it is saved.
Working: A command that takes minutes, such as a rip, a fit, a refine or a training corpus, runs under a box that counts the seconds and shows its latest output.
Files: ^O opens a directory: Enter loads a document, runs a script, or rips a .wav or a .mid.

Build

Runtime

Platforms: Linux and FreeBSD, in a terminal; Android next, inside ViF’s Android app
Audio: PulseAudio or PipeWire, ALSA, sox or ffplay as a player process, OSS on FreeBSD; a WAV file or none for scripts
Toolchain: Go 1.27, no CGO; RipModel runs in pure Go, in float32
Libraries: Terminal, Color, TOML, Audio and Genetic
Trainer: Python 3.11 or later with PyTorch, outside the Go module; a CPU works, a GPU is about ten times faster
Data: TOML sound and pattern documents, in ViF’s audio folder; in Stapes' root, ripmodel.bin, RipModel’s weights, and rip.toml, the tuned thresholds
License: BSD-3-Clause

Build

Clone: git clone https://github.com/lixenwraith/stapes --depth 1
Build: cd stapes && make release for bin/stapes, or go run ./cmd/stapes; make install puts it on the path
TUI: ./bin/stapes -tui
REPL: ./bin/stapes, then help
Script: ./bin/stapes -headless -s recipe.slab
Model: make venv, then tool/rip-train/pipeline.sh all renders a corpus, trains, benches and installs the weights; its README is the guide

-tui, -T: Opens the full-screen TUI; empty documents start from the built-ins.
-s file: Runs a .slab script line by line and exits; a failing line stops it and reports its line number.
-headless: Discards audio; -ab names a backend instead: pacat, aplay, null or wav:out.wav.
-config-dir: Stapes' root, ~/.config/stapes by default. The documents stay in ViF's audio folder, where save writes for the next game; -config-music and -config-sounds name a document's file outright.
-vol: The effects' volume, 0 to 1; music keeps its own level.