Models and algorithms
Where we can, we use classic signal-processing algorithms: they're more accurate and lighter than AI. For recognition we use small existing models with no custom training. A result only appears above a tested threshold.
Tuner: McLeod Pitch Method
Pitch detection with pitchy (McLeod Pitch Method), followed by a multi-period refinement. Runs in a Web Worker.
How
- The window adapts to the instrument: 2048 samples for violin, 4096 for guitar, 8192 for bass (at 48 kHz).
- A reading appears only at clarity ≥ 0.9, with a 5-reading median and octave-error protection (an octave jump must repeat 4 times).
- “In tune” = within ±3 cents for 0.5 s.
What we tested
- Synthetic sine and sawtooth tones, 40 Hz – 2 kHz, at 44.1 and 48 kHz: all within ±1 cent.
- With background noise (SNR ≈ 20 dB): within ±1 cent.
- On the synthesized instrument sounds (guitar, bass, ukulele, violin, with known detuning): every smoothed reading within ±1 cent.
Limitations
- Tested on synthetic signals, not on real recordings with measured tuning.
- Low piano strings (inharmonic) and noisy rooms reduce accuracy.
- One sound at a time: the tuner isn't for chords.
Licence: pitchy: MIT. The refinement: Sonoro code.
Chords: chroma and templates
Energy across the 12 notes (chroma) from spectral peaks, averaged over ~0.35 s, matched by cosine similarity against 108 chords: major, minor, 7, maj7, m7, sus2, sus4, dim, aug on all 12 roots.
How
- Templates include each note's harmonics (what a real instrument produces).
- The bass is voted over time for slash chords (e.g. D/F♯). Chords with the same notes (Csus2 = Gsus4) are named by the bass.
- A chord is shown only if it beats the runner-up by ≥ 0.03 and holds for 300 ms.
What we tested
- On 108 synthesized chords (piano and guitar), at the chosen threshold: ~95% correct when a chord is shown, shown on ~75% of frames.
- The piano demo progression: every chord recognised correctly.
Limitations
- On the synthetic guitar: C after E minor isn't shown, and D/F♯ reads as Dmaj7/F♯; the bass can alternate on weak low strings.
- Typical confusions: sus2 / m7, dim / minor.
- Accuracy on real, annotated recordings hasn't been measured yet.
Licence: Sonoro code; fft.js (MIT); chord spelling with tonal (MIT). Meyda (MIT) is used for the measurements on Visualize.
What's playing: YAMNet
A Google model trained on AudioSet (521 classes). Runs in the browser with TensorFlow.js, in a Web Worker, on audio resampled to 16 kHz.
How
- 0.975 s windows every 0.48 s (~50% overlap); averaged over the last ~3 s.
- 17 chosen classes (instruments, voice, whistling, clapping) + “other / not sure”.
- Each class has its own threshold, chosen on an evaluation set. Below it: “not sure yet”.
Limitations
- The evaluation set is small (~6 clips per class). Thresholds are a starting point, not a guarantee.
- It confuses keyboard with piano and singing with speech. Acoustic vs electric guitar is weak.
- On the synthesized demo sounds it's often unsure.
YAMNet: results on the evaluation set
114 openly licensed clips (CC0, public domain, CC BY, CC BY-SA) from Wikimedia Commons. When a label appears, it's right 92 % of the time; 27 % of the time it says “not sure”; on non-music sounds (rain, traffic, birds) it's wrong 3 % of the time. Mean float16 vs original difference: 0.0007.
| Class | Threshold | Precision | Recall | Clips detected | 90% target |
|---|---|---|---|---|---|
| Acoustic guitar | 0.10 | 100 % | 85 % | 100 % / 6 | yes |
| Electric guitar | 0.10 | 100 % | 74 % | 83 % / 6 | yes |
| Bass guitar | 0.10 | 100 % | 77 % | 83 % / 6 | yes |
| Violin / fiddle | 0.17 | 97 % | 67 % | 83 % / 6 | yes |
| Cello | 0.10 | 100 % | 83 % | 83 % / 6 | yes |
| Piano | 0.10 | 71 % | 83 % | 83 % / 6 | no |
| Keyboard / organ / synth | 0.49 | 90 % | 38 % | 50 % / 6 | yes |
| Flute | 0.14 | 91 % | 100 % | 100 % / 6 | yes |
| Trumpet | 0.19 | 91 % | 100 % | 100 % / 6 | yes |
| Saxophone | 0.24 | 100 % | 40 % | 50 % / 6 | yes |
| Drum kit | 0.23 | 94 % | 67 % | 83 % / 6 | yes |
| Singing | 0.10 | 100 % | 33 % | 33 % / 6 | yes |
| Speech | 0.79 | 76 % | 100 % | 100 % / 6 | no |
| Whistling | 0.11 | 91 % | 65 % | 67 % / 6 | yes |
| Clapping | 0.10 | 100 % | 71 % | 83 % / 6 | yes |
| Harmonica | 0.11 | 100 % | 21 % | 33 % / 6 | yes |
| Accordion | 0.19 | 91 % | 67 % | 67 % / 6 | yes |
Licence: Apache-2.0 (Google). Weights stored as float16: 8.0 MB (from 16 MB).
Transcribe: Basic Pitch
A Spotify model that finds the notes (including several at once) in a recording. Opt-in, runs on your device, at 22.05 kHz.
How
- Onset threshold 0.5, frame threshold 0.3, minimum length 11 frames (~128 ms).
- Chords over time come from the same chroma algorithm as above.
- MIDI export with @tonejs/midi.
What we tested
- A synthesized piano scale (8 notes): at least 7 notes found at the right pitch and time, with at most 3 extra notes.
Limitations
- Works best on a single instrument. On dense mixes or drums it misses notes.
- 30 s maximum per recording.
Licence: Apache-2.0 (Spotify). 0.9 MB.
Libraries
| Library | Used for | Licence |
|---|---|---|
| Next.js 16, React 19 | The site | MIT |
| TensorFlow.js 4.22 | Runs YAMNet and Basic Pitch | Apache-2.0 |
| pitchy 4.1 | Pitch detection | MIT |
| Meyda 5.6 | Level, brightness, spectral flatness | MIT |
| tonal 6.5 | Chord tones | MIT |
| fft.js 4.0 | Spectrum for chroma | MIT |
| three.js 0.187, @react-three/fiber 9.8 | Chladni plate, 3D spectrum | MIT |
| @tonejs/midi 2.0 | MIDI export | MIT |
| zustand, zod, idb-keyval | Settings, validation, local storage | MIT / MIT / Apache-2.0 |
| framer-motion, lucide-react | Animation, icons | MIT / ISC |
Fonts: Inter, Manrope, JetBrains Mono (SIL Open Font License 1.1), self-hosted.
Demo sounds
“Guitar: the 6 strings”, “Piano: chords” and the like are synthesized by Sonoro (we know exactly which notes and how detuned they are). The real recordings are 7 s excerpts from Wikimedia Commons:
| Sound | Original | Author | Licence |
|---|---|---|---|
| Piano (recording) | C major, Middle C, I-IV-V-I-I-V-I chord progression.ogg (source) | CuriousC3301 | CC0 |
| Guitar (recording) | Audio files of spanish guitar chords 01.wav (source) | Wilfredor | CC0 |
| Violin (recording) | Minstrel Boy 2.ogg (source) | Endersslay | CC0 |
| Flute (recording) | Joachim Andersen - Etude 3 for Solo Flute.oga (source) | Scott Goff (flute) | CC BY-SA 2.0 |
| Singing (recording) | Twinkle Twinkle Little Star - sung with full lyrics.ogg (source) | Dcoetzee | CC0 |
| Whistling (recording) | Whistling la donna a mobile.ogg (source) | Stanislav Kozlovskiy | CC BY-SA 4.0 |
| Clapping (recording) | Palmas sevillanas (flamenco clapping), 160 BPM.ogg (source) | Santiago Sanchez Cifuentes | CC0 |
| Drums (recording) | Rock beat.ogg (source) | Krol111 | CC BY-SA 3.0 |
| Speech (recording) | IMF Spoken Word.ogg (source) | John Bilkey | CC0 |