Soundlab - Lixen

In development

Synthesize and sequence music in a terminal, and rip recordings into its patterns and voices; built from ViF's repository for now

Go Python FreeBSD Linux TOML AI Genetic Algorithms Machine Learning Transformer Terminal TUI

How a rip works Build and run GitHub

About

Soundlab is the workbench of a pure-Go synthesis and music engine, the one that plays ViF’s sound and writes its music as the game is played. Everything the engine plays, from a one-shot effect to a drum kit, a voice or a pattern, is a TOML document Soundlab opens, edits and plays while you listen. It runs as a command line, a full-screen terminal UI or a script, and all three drive one table of verbs, so whatever is done by hand can be written down and run again.

It also listens. rip hears a recording and writes what Soundlab needs to play it back, in the engine’s own vocabulary: the tempo and the bars, the key, the drums, the melody and the bass as patterns, the voices that play them and the song that plays the bars in order. The RipNet tab draws how.

Documents: An edit changes the document in memory; apply hands it to the running engine, where the next note takes it, and revert brings the engine's copy back. undo steps back over the last hundred edits.
Paths: Any field is reached by its dotted path, read through the TOML tags the loader reads: set blip.layer.0.source.freq 880 changes exactly what the file would. A zero value means the engine's default, so a saved document holds only what was set.
Checks: The command line checks a value's type and width; the engine's own rules, a frequency under Nyquist or an envelope's length, run at validate and apply.
Slots: The sequencer plays four, rhythm, melody, fill and bass, as the game draws them. A pattern takes its slot on the next bar, and a slot or a track mutes or solos while its pattern plays on in time.
Voices: The tonal instruments are entries of the sound document, made, edited and applied like a sound; a new name is a new instrument any pattern's track can play.
Scripts: A .slab file is one command a line: # comments, - before a line tolerates its failure and ! expects one. A recipe reads its files beside itself, so it runs from anywhere.
Home: Soundlab and its engine live in ViF's repository today, the command beside the game, and will move together into a repository of their own.

How a rip works A rip, end to end

RipNet

A rip listens to a recording and writes what Soundlab needs to play it back: the tempo and where the bars fall, the key and the chord each bar leans on, the drum hits, the melody and bass lines as patterns on the sixteenth grid, the voices that play them, and a song that plays the bars in order. All of it is the engine’s own vocabulary, so the result plays at once, in Soundlab and in the game, and is checked by ear and by number against the recording.

Two halves meet in the middle. Ripping turns a recording into patterns; training teaches RipNet, a transformer, to hear the recordings ripping starts from, on music the engine renders itself. Either ear, the built-in analysis or the model, feeds the same decoders.

What each stage runs on

TransformerRipNet, a model that hears the frames
GAa genetic search: a pool of candidates bred toward a score
DSPspectra and filters, and the rooms of training
Decodingdynamic programming, Viterbi paths, peak picking
Renderthe engine plays a candidate to be heard

Pick a stage to read it below

Ripping

Recording A WAV, or a MIDI score

1HearTransformerDSP Frames of onsets, drums and 69 pitches

1Hearfrom sound to frames

The recording is read at 22.05 kHz and cut into frames, a 93 ms window every 11.6 ms, about 86 a second. Its tuning is read first, the cents its notes sit off A440, so a recording forty cents flat reads as in tune. A front end then hears each frame into the same fields, and the decoders after it never know which one heard.

DSPThe analysis

Split each frame into its harmonic and percussive parts, by medians across time and across frequency
Drums each in its own band: a kick's rise under 100 Hz, a hihat's over 5 kHz, a snare between
Pitches by salience, the sum of a pitch's partials; the bass from a window four times longer, which tells semitones apart at 110 Hz

TransformerRipNet

Frames 229 log-mel bands from 30 Hz, each in dB against the frame's own level, and the level
Linear 230 wide to 256
4 layersAttention 8 heads, 64 frames either sideFeed-forward 1024 wide, GELU
Heads per frameonset 1drums 3pitch 69voicing 1strike 69release 69velocity 69wave 5genes 32

How far a head hears around its frame: each of the eight leans on nearer frames by a slope of its own (ALiBi), one on an attack, another across the whole window; past 64 frames, nothing.

Fields: Per frame: the onset; the kick, snare and hihat; the salience of 69 pitches, E1 to C7; each register's chroma.
Only the model: Which pitches are struck and let go, and how hard, and the wave and 32 genes each note is played with.
Weights: 3.2 million, run in pure Go in float32, the same on every host; Soundlab hears with them when the configuration root holds audio/ripnet.bin, and with the analysis otherwise.

2ReadDecoding Tempo, bars, key, drums, two lines

2Readfrom frames to music

The decoders are shared, classical and cheap. Every threshold they hold lives in RipParams, which tune searches and a configuration root's audio/rip.toml carries.

One bar: the salience arrives late and leaks into the pitch above; the path holds to one pitch until moving pays, and its notes land on sixteenths.

Tempo: The autocorrelation of the onsets under a prior centred on 120 bpm, within the sequencer's 80 to 200.
Beats: Dynamic programming that holds to the period.
Bars: The bar line at the phase whose downbeats carry a kick and the key's root in the bass, the snare on two and four; a phrase that leads in gets a pickup bar.
Key: The root and scale whose tones hold the most chroma, the bass voting for the tonic, and a chord root per bar.
Drums: Peak picking: a kick by its rise under 100 Hz, a hihat by its rise over 5 kHz, a snare between; a hit can be two drums at once.
Notes: A Viterbi path through the salience that pays to move and to start or end a note, each note then set on a sixteenth and the key's nearest degree, a semitone left over kept as an accidental. With the model, a melody note starts where its pitch is struck and lasts until it is let go, any number at once.
Lines: The melody register, F3 up, as two lines at once, so a slow line under a pluck is heard too; the bass up to A3.

3WriteGARender Patterns, their voices, a transcript

3Writefrom music to Soundlab

Patterns of up to four bars a part go into the document, playable at once, and the transport takes the tempo, key and tuning. Each line gets a voice. Through the model, its voice head names the genes of every note, averaged over the line; a line no voice was heard for is fitted by ear, by a genetic search the engine's renders judge.

Renderthe line's six longest, loudest notes, at the recording's pitch, loudness and tuning
Compareeach render's mel frames with the recording around its note
Breedtournaments of three, uniform crossover, bounded mutation

16 voices a generation for 8 generations, seeded with every registered voice; the mutation widens while the best improves and narrows when it stalls, and no voice is rendered twice.

A voice is a wave and 32 genes, each 0 to 1 on the scale the ear hears: oscillators, string and partials, filter, envelope, and the drive and level out. Here, the shipped piano.

Patterns: <name>_beat_1, _melody_1 with the second line as a track of its own, _bass_1, and on.
Voices: <name>_melody_voice, _under_voice and _bass_voice, each level then set to the recording's loudness, which the mel frames do not hear; fit searches longer.
Transcript: Every note in seconds, before the grid, with how much of the recording's salience supports it; notes writes it as a MIDI file.

4CompressGA A dictionary of bars, and a song

4Compressbars into a song

As few one-bar patterns as the bars allow, part by part. Two bars differ by the steps they do not share over the steps either has; bars within 5% of each other form a group, and its middle bar, the medoid, stands for it.

Dictionaryabc

Songaabacaba

Eight bars of a beat: bars 2 and 7 carry one kick more, within 5% of bars 1 and 3, so three patterns, <name>_beat_a, _b and _c, play all eight.

Group: Each bar its own group at first, the two nearest merged while their mean distance holds within the threshold.
Search: A genetic search over which bar joins which, 32 assignments for 40 generations, scored by the patterns in use, and heavily by any bar too far from its own.
Song: The patterns each bar plays, at the recording's tempo, key and tuning, which song name play conducts live. Its render is ripped back and scored against the rip, so what compressing lost is a number.

5RefineRenderGA Render, compare, keep what sounds closer

5Refineclosing the loop

The engine is the ground truth of its own vocabulary, so a rip is good when its render sounds like the recording. Refining works in rounds, and a change stays only when the bars that play it sound closer.

Changea voice, by a genetic search, or one step of a line
Renderthe bars of the song that play it
Comparerender and recording, through one front end, bar by bar
Keepwhat sounds closer

Three rounds, or until a round gains too little.

Voices: Searched again by the whole song's score, seeded with the one in hand and every registered voice; none may sound more than a tenth further from the line's notes than its fit.
Steps: Each tonal step tried a semitone and an octave either way, at the recording's strongest pitch there, a step earlier or later, longer or shorter, and silent; an empty step tries the strongest pitch, the one way in for a note the rip missed.
Score: Pitch salience, mel bands and onsets, each compared as a cosine, the frames paired bar by bar on each one's own beats.

6CheckRenderGA Scores, a second rip, the bench

6Checkhow good a rip is

truth: Scores a rip against notes given: a transcript corrected by hand, a rhythm tapped out, or a MIDI file, whose drum channel scores the drums hit by hit. Per line, the notes heard as given, missed and extra, and how many are heard by how loud the recording plays them, in 6 dB bands.
agree: Holds two rips of one piece to each other where no score exists.
bench: Rips fresh loops the engine renders, whose answer is known, clean and through a room and a mix, and scores the analysis and the model side by side.
tune: A genetic search over the decoders' thresholds, each candidate scored by its rips of rendered loops against what they hold.

Not told apart yet: a bass note's attack from a kick, a plucked chord's from a snare, a change of key from accidentals, and a third line or the chord a pedal rings, which two lines cannot hold. Soft notes are heard less often than loud ones.

Training

GenerateRender Loops whose every note is known

Generatemusic that knows its notes

The model learns from music the engine renders itself, since the engine knows exactly what it played: the labels cost nothing and are exact, and nothing is labelled by hand.

One bar as the engine rendered it, and what it keeps beside its frames: every drum hit, every note's pitch and length, where each is struck and let go, and the voice that plays it.

Loops: Four bars each, from the bank's rhythm, melody and bass patterns, across the sequencer's tempos and keys, twenty thousand by default.
New lines: Three loops in five per part play a beat, a melody or a bass line drawn from the key instead, with a broken chord in the free slot.
Soft, bare: A third of the generated melodies and accompaniments play half their length 20 to 30 dB softer; three generated loops in ten leave out their drums, and one in seven its bass.
Labels: Per frame: the onsets, the drums, the pitches sounding, where each is struck and let go and how hard, and every note's wave and 32 genes.

RoughenDSP A room, a mix, new voices

Roughenso it never learns the synthesizer

Each loop plays through a room and a mix drawn for it alone, in this order, so the model hears what recordings sound like rather than the engine's clean output. The labels stay the notes as played.

Gain and swell −6 to +3 dB, swelling by up to 6 dB over the loop
Low-pass one pole, 4 to 11 kHz, in half the loops
Equaliser two peaking bands, 80 Hz to 8 kHz, ±6 dB
Soft clip a tanh drive of 1.5 to 3.5, in a third
Compressor 2:1 to 6:1, in half
Reverb Schroeder's four combs into two allpasses: a booth, or in a third a hall further back
Noise floor 40 to 65 dB under the mix, in half

Voices: Redrawn for each loop near the registered ones, every gene nudged at random, about a third on another wave.
Pedal: In a third of the loops, every release but the bass's three to ten times as long, as under a piano's pedal.
Tuning: Off A440 by up to a quarter tone, as the recordings it will hear can be.

TrainTransformer A PyTorch twin, weights for Go

Traina twin in PyTorch

tool/rip-train holds the model's PyTorch twin, outside the Go module. It learns from the loops' frames, which Go makes so both sides hear alike, and writes the weight file the Go model loads, the two agreeing to float32 rounding.

Windowssix seconds of a loop, 512 frames, eight a step
Losscross-entropy leaning on rare strikes and onsets, squared error for genes and velocity
Held out200 loops scored every 1000 steps, the best step kept
Benchships only when it beats the analysis, on rendered loops and on real recordings

A round: pipeline.sh renders the corpus in parts at once, trains, benches and installs: 40000 loops and as many steps in about four hours on a laptop's GTX 1060, or across machines in shards whose weights are averaged into the next round's start.
Shape: The trainer may widen or deepen the model, put convolutions before its layers, or split them into towers for groups of heads; the weight file carries its shape, and the Go side builds what it reads.

Weightsripnet.binwhat stage 1, Hear, listens with

Engine

Synthesis

Pure Go, 44.1 kHz stereo, no CGO. A sound is layers of oscillators, sweeps, FM, noise and impulses through filters, envelopes, ring modulation and waveshaping, summed on buses and rendered ahead into variants, so no two hits are quite alike.

Voices

An oscillator stack through a resonant low-pass and its envelope into an ADSR, or a string, as a delay line or as its partials. Thirty-two genes on the scales the ear hears describe any of them, the one vector a model predicts and a search evolves.

Patterns and harmony

Drum hits and notes on a sixteenth grid, each note a degree of the key with an accidental for one off it, resolved against a root, a scale and a chord progression as it plays. TOML macros write a drum lane or a line once.

Sequencer

Four crossfading slots: rhythm, melody, a fill and bass. Changes land on the bar line, tempo, swing and tuning follow the transport, and a song is conducted bar by bar, each bar’s patterns scheduled while the one before plays.

Live registry

One command channel into one mixer goroutine keeps playback, transport and hot reloads in order. A definition replaced mid-note lets the sounding note finish on what it began with; the next takes the new one.

Offline and seeded

The same engine renders a note, a pattern or a song to a buffer, which is how a rip fits voices, judges a song and makes its training data. Every draw is seeded, so a render, a rip and a model’s answer repeat on every host.

Workflow

A rip is a few verbs, at the command line or the TUI’s own; each one’s result is a document entry like any other, to play, edit and save.

Rip: rip song.wav name writes the patterns, the voices and the transcript, and sets the transport's tempo, key and tuning; rip song.mid score does the same from a score, hearing nothing.
Listen: slot the patterns, or render them to a WAV, and read the transcript with notes name, which also writes it as a MIDI file to correct in a sequencer.
Check: truth name score.mid against a score, a transcript corrected by hand or a rhythm tapped out; agree holds two rips of one piece to each other.
Compress: compress name writes the dictionary of bars and the song; song name play conducts it through the live sequencer.
Refine: refine name closes the loop on the recording; fit name refits the voices alone.
Keep: Save the patterns, the song and the voices, below. In a .slab file the whole recipe is one word, run.

rip song.wav bite nodrums        # the patterns and fitted voices; nodrums: no kit
compress bite                    # the dictionary and the song bite
song bite play                   # conducts it; music stop ends it
save pattern bite_patterns.toml  # the patterns and the song
write bite_melody_voice bite_melody_voice.toml
write bite_bass_voice bite_bass_voice.toml

In the TUI

The TUI is a shell over the same verbs: its keys build and run command strings, a recorded take included, so a script can reproduce anything it does.

Browse: Patterns, sounds and voices beside the mixer; space plays a sound or a voice's middle C, or puts a pattern in its slot, and 0–3 pick the slot.
Mixer: Every slot on one timeline under a shared playhead; a slot or a track mutes, or a track solos, while its pattern plays on in time.
Editor: A drawn keyboard that records takes onto the key's degrees, a step grid with velocities, chances and lengths, and a piano roll; a new pattern loops in its slot before it is ever saved.
Working: A verb that takes minutes, a rip, a fit, a refine or a training corpus, runs behind the screen under a box that counts the seconds and shows its last lines.
Files: ^O opens a directory: Enter loads a document, runs a script, or rips a .wav or a .mid.

Build

Runtime

Platforms: Linux and FreeBSD, in a terminal
Audio: PulseAudio or PipeWire, ALSA, sox or ffplay as a player process, OSS on FreeBSD; a WAV file or none for scripts
Toolchain: Go 1.27, no CGO; the model runs in pure Go, float32, the same on every host
Dependencies: own Go modules for terminal, TOML and colour
Trainer: Python 3.11 or later with PyTorch, outside the Go module; a CPU will do, a GPU is an order of magnitude faster
Data: TOML sound and pattern documents; audio/ripnet.bin, the model’s weights, and audio/rip.toml, tuned thresholds, in the configuration root
License: BSD-3-Clause

Build

Clone: git clone https://github.com/lixenwraith/vif --depth 1
Build: cd vif && make tools for bin/soundlab, or go run ./cmd/soundlab
TUI: ./bin/soundlab -tui
REPL: ./bin/soundlab, then help
Script: ./bin/soundlab -headless -s recipe.slab
Model: tool/rip-train/pipeline.sh all renders a corpus, trains, benches and installs the weights; its README is the guide

-tui, -T: The full-screen TUI; empty documents are seeded from the built-ins.
-s file: Runs a .slab script line by line and exits; a failing line stops it with its number.
-headless: Discards audio; -ab names a backend instead: pacat, aplay, null or wav:out.wav.
-config-dir: The configuration root ViF reads, resolved as ViF resolves it, with -config-music and -config-sounds for single documents; save writes where the next game reads first.
-vol: The effects' volume, 0 to 1; music keeps its own level.