Skip to content

Models and algorithms

Where we can, we use classic signal-processing algorithms: they're more accurate and lighter than AI. For recognition we use small existing models with no custom training. A result only appears above a tested threshold.

Tuner: McLeod Pitch Method

Pitch detection with pitchy (McLeod Pitch Method), followed by a multi-period refinement. Runs in a Web Worker.

How

  • The window adapts to the instrument: 2048 samples for violin, 4096 for guitar, 8192 for bass (at 48 kHz).
  • A reading appears only at clarity ≥ 0.9, with a 5-reading median and octave-error protection (an octave jump must repeat 4 times).
  • “In tune” = within ±3 cents for 0.5 s.

What we tested

  • Synthetic sine and sawtooth tones, 40 Hz – 2 kHz, at 44.1 and 48 kHz: all within ±1 cent.
  • With background noise (SNR ≈ 20 dB): within ±1 cent.
  • On the synthesized instrument sounds (guitar, bass, ukulele, violin, with known detuning): every smoothed reading within ±1 cent.

Limitations

  • Tested on synthetic signals, not on real recordings with measured tuning.
  • Low piano strings (inharmonic) and noisy rooms reduce accuracy.
  • One sound at a time: the tuner isn't for chords.

Licence: pitchy: MIT. The refinement: Sonoro code.

Chords: chroma and templates

Energy across the 12 notes (chroma) from spectral peaks, averaged over ~0.35 s, matched by cosine similarity against 108 chords: major, minor, 7, maj7, m7, sus2, sus4, dim, aug on all 12 roots.

How

  • Templates include each note's harmonics (what a real instrument produces).
  • The bass is voted over time for slash chords (e.g. D/F♯). Chords with the same notes (Csus2 = Gsus4) are named by the bass.
  • A chord is shown only if it beats the runner-up by ≥ 0.03 and holds for 300 ms.

What we tested

  • On 108 synthesized chords (piano and guitar), at the chosen threshold: ~95% correct when a chord is shown, shown on ~75% of frames.
  • The piano demo progression: every chord recognised correctly.

Limitations

  • On the synthetic guitar: C after E minor isn't shown, and D/F♯ reads as Dmaj7/F♯; the bass can alternate on weak low strings.
  • Typical confusions: sus2 / m7, dim / minor.
  • Accuracy on real, annotated recordings hasn't been measured yet.

Licence: Sonoro code; fft.js (MIT); chord spelling with tonal (MIT). Meyda (MIT) is used for the measurements on Visualize.

What's playing: YAMNet

A Google model trained on AudioSet (521 classes). Runs in the browser with TensorFlow.js, in a Web Worker, on audio resampled to 16 kHz.

How

  • 0.975 s windows every 0.48 s (~50% overlap); averaged over the last ~3 s.
  • 17 chosen classes (instruments, voice, whistling, clapping) + “other / not sure”.
  • Each class has its own threshold, chosen on an evaluation set. Below it: “not sure yet”.

Limitations

  • The evaluation set is small (~6 clips per class). Thresholds are a starting point, not a guarantee.
  • It confuses keyboard with piano and singing with speech. Acoustic vs electric guitar is weak.
  • On the synthesized demo sounds it's often unsure.

YAMNet: results on the evaluation set

114 openly licensed clips (CC0, public domain, CC BY, CC BY-SA) from Wikimedia Commons. When a label appears, it's right 92 % of the time; 27 % of the time it says “not sure”; on non-music sounds (rain, traffic, birds) it's wrong 3 % of the time. Mean float16 vs original difference: 0.0007.

ClassThresholdPrecisionRecallClips detected90% target
Acoustic guitar0.10100 %85 %100 % / 6yes
Electric guitar0.10100 %74 %83 % / 6yes
Bass guitar0.10100 %77 %83 % / 6yes
Violin / fiddle0.1797 %67 %83 % / 6yes
Cello0.10100 %83 %83 % / 6yes
Piano0.1071 %83 %83 % / 6no
Keyboard / organ / synth0.4990 %38 %50 % / 6yes
Flute0.1491 %100 %100 % / 6yes
Trumpet0.1991 %100 %100 % / 6yes
Saxophone0.24100 %40 %50 % / 6yes
Drum kit0.2394 %67 %83 % / 6yes
Singing0.10100 %33 %33 % / 6yes
Speech0.7976 %100 %100 % / 6no
Whistling0.1191 %65 %67 % / 6yes
Clapping0.10100 %71 %83 % / 6yes
Harmonica0.11100 %21 %33 % / 6yes
Accordion0.1991 %67 %67 % / 6yes

Licence: Apache-2.0 (Google). Weights stored as float16: 8.0 MB (from 16 MB).

Transcribe: Basic Pitch

A Spotify model that finds the notes (including several at once) in a recording. Opt-in, runs on your device, at 22.05 kHz.

How

  • Onset threshold 0.5, frame threshold 0.3, minimum length 11 frames (~128 ms).
  • Chords over time come from the same chroma algorithm as above.
  • MIDI export with @tonejs/midi.

What we tested

  • A synthesized piano scale (8 notes): at least 7 notes found at the right pitch and time, with at most 3 extra notes.

Limitations

  • Works best on a single instrument. On dense mixes or drums it misses notes.
  • 30 s maximum per recording.

Licence: Apache-2.0 (Spotify). 0.9 MB.

Libraries

LibraryUsed forLicence
Next.js 16, React 19The siteMIT
TensorFlow.js 4.22Runs YAMNet and Basic PitchApache-2.0
pitchy 4.1Pitch detectionMIT
Meyda 5.6Level, brightness, spectral flatnessMIT
tonal 6.5Chord tonesMIT
fft.js 4.0Spectrum for chromaMIT
three.js 0.187, @react-three/fiber 9.8Chladni plate, 3D spectrumMIT
@tonejs/midi 2.0MIDI exportMIT
zustand, zod, idb-keyvalSettings, validation, local storageMIT / MIT / Apache-2.0
framer-motion, lucide-reactAnimation, iconsMIT / ISC

Fonts: Inter, Manrope, JetBrains Mono (SIL Open Font License 1.1), self-hosted.

Demo sounds

“Guitar: the 6 strings”, “Piano: chords” and the like are synthesized by Sonoro (we know exactly which notes and how detuned they are). The real recordings are 7 s excerpts from Wikimedia Commons:

SoundOriginalAuthorLicence
Piano (recording)C major, Middle C, I-IV-V-I-I-V-I chord progression.ogg (source)CuriousC3301CC0
Guitar (recording)Audio files of spanish guitar chords 01.wav (source)WilfredorCC0
Violin (recording)Minstrel Boy 2.ogg (source)EndersslayCC0
Flute (recording)Joachim Andersen - Etude 3 for Solo Flute.oga (source)Scott Goff (flute)CC BY-SA 2.0
Singing (recording)Twinkle Twinkle Little Star - sung with full lyrics.ogg (source)DcoetzeeCC0
Whistling (recording)Whistling la donna a mobile.ogg (source)Stanislav KozlovskiyCC BY-SA 4.0
Clapping (recording)Palmas sevillanas (flamenco clapping), 160 BPM.ogg (source)Santiago Sanchez CifuentesCC0
Drums (recording)Rock beat.ogg (source)Krol111CC BY-SA 3.0
Speech (recording)IMF Spoken Word.ogg (source)John BilkeyCC0