A pure-Go synthesis, sound effect and music engine. It renders sounds, voices and patterns from TOML documents, arranges music over a harmony, streams 16-bit stereo to the player the host has, and rips a recording back into patterns.
- Sounds as layered sources through processors and buses, rendered ahead into variants
- Voices from oscillator stacks and modelled strings; patterns on a sixteenth-note grid, in scale degrees
- A sequencer that arranges patterns over a root, scale and chord progression
- Plays through PulseAudio, PipeWire, ALSA, SoX, FFmpeg or OSS, or writes a WAV file
- Rips a recording into tempo, key, drums and notes, and reads and writes MIDI
The engine
Everything the engine plays is data it is handed: TOML documents of sounds,
voices and patterns, and the embedder’s commands. It ships none of its own;
bank embeds the set ViF and Stapes play, which anyone may use or ignore.
In
- Sounds and voices
[[sound]]and[[voice]]tables, files beside them byinclude - Patterns
[[pattern]]tables, placed by role, group, tier and motif - Commands
Play,SetHarmony,SetIntensity,DefineVoice…
Start
- Register the base banks, then the documents over them, macros expanded
- Pre-render every sound’s variants, each seeded by its name and index
- Arrange the kit of drums and voices, and the patterns by group and tier
Mixer
- Sequencer four crossfading slots: rhythm, melody, a free layer or the fill, bass
- Voices and sounds oscillator stacks, plucked and modal strings, and the sfx playing
- Mix music and sfx muted apart, slots and tracks masked, pause ramped on the master
Out
- A player pacat, pw-cat, aplay, sox, ffplay, or OSS on FreeBSD; failing over
- A file or a writer
wav:<path>, or the host’s ownSink
- Sounds
- Layers of sources, oscillators, sweeps, FM, noise, impulses and bursts, through chains of filters, envelopes, modulation and shaping, into optional buses and a master. Each is rendered ahead into variants, checked for duration, frequencies and size before a sample is made.
- Voices
- Oscillator stacks through a resonant low-pass and an ADSR, or strings: a plucked delay line after Karplus and Strong, in tune at every pitch, and modal strings built from their partials, the bass and the piano.
- Patterns
- Steps of hits and notes on a sixteenth-note grid, the notes scale degrees that the harmony, a root, a scale and a chord progression, resolves at play time. The sequencer arranges them by group, tier and motif across four crossfading slots.
- Live editing
DefineSound,DefineVoiceandDefinePatternswap a definition while the mixer runs: a sounding note finishes on what it began with, the next takes the new one. This is how Stapes edits while it plays.- Determinism
- Every render and draw is seeded: a sound's noise by its name and variant, the sequencer's choices on streams of one seed. The same seed and commands play the same music.
Ripping a recording
RipWAV listens to a recording and estimates what the sequencer would need to
play it: the tempo and the bars, the key and each bar’s chord root, the kick,
snare and hihat hits, a melody of up to two lines and the bass, as steps of the
grid; Patterns writes them out. A rip also keeps its transcript, every note
in seconds, which MIDI files carry both ways.
What hears the recording is a seam. A front end fills the same frames, the onsets, each pitch’s salience and the chroma, and the same decoders read them: the package’s own signal analysis is one front end, and RipModel, below, the other. The searches that fit voices and compress bars into a song live in Stapes; the primitives they search with are here, rendering a voice at a note offline and measuring the distance between mel frames.
RipModel
RipModel is a transformer that hears a recording frame by frame: what struck, which drum, which of 69 pitches sound, which are struck or let go and how hard, and the wave and 32 genes of the voice playing them. This package decides what it hears and what it must say, and turns its answers into a rip.
Data
RenderMusicloops the engine plays offline, every note knownExpectedRipthe rip a perfect listener would make of them: the labels- Recordings drums played from MIDI, instrument notes, a phone’s recordings
Features
MelFrames229 log-mel bands from 30 Hz, every 11.6 ms- Level-free bands each band in dB less the frame’s mean, and the mean
- PCENplane 2 each band over its own last 0.4 s: a hum stays flat, a strike stands out
Train
Hear
ripmodel.Loadinputs, planes and heads checked againstSpec; the last three heads optional- Forward the transformer, float32, in Go
- Nine heads onset, drums, pitch, voicing, wave, genes, strike, release, velocity
Rip
- Notes from strike to release, any number at once
- Voices each line’s wave and 32 genes
- Drums and beats through the same decoders the analysis feeds
- Shape
Specfixes the inputs and the nine heads in order; width, depth and window are the trainer's to change. A model trained before the strike, release and velocity heads still loads and rips from what it has, and one whose inputs or heads differ is refused.- Features
- Each mel band in dB less the frame's mean, so a soft phrase reaches the model as clearly as a loud one, and the mean as one input of its own. A second plane, PCEN, divides each band by its own average of the last 0.4 s: a fan or a room's hum holds near 0.3, and a note struck over it still stands out. It only looks back, so a live rip can compute it.
- Labels
- The engine renders loops whose every note is known, and
ExpectedRipsays what a perfect listener would rip from them: exact labels, as many as there is time to render, through a room and a mix when the trainer adds them. - Grading
CompareRipsscores a rip against the expected one, the tempo within two BPM, the bars, and each line by recall and precision.Fidelitygrades a render against the recording itself, bar by bar on each one's own beats, which a score against its own rip cannot.- Tuning
- Every threshold of the decoders sits in
RipParamswith a range and a place in a gene vector, so a genetic search can tune them; a[rip]table carries a set.