From 0.52 to 0.90: Five Steps to a Better Chord Analyzer

chordotomy listens to a recording and writes out its chords, beat by beat, with Roman numerals. This is the story of how its accuracy went up, one measured step at a time.

How it’s measured

Two public datasets, scored with mir_eval:

Tiny AAM: 20 mixed tracks, annotated in major and minor only.
GuitarSet: 180 solo-guitar accompaniment takes, full of sevenths and jazz voicings.

The main number is majmin: the share of time where the analyzer names the right major or minor triad. Below, a “point” is 0.01 of a score. Here’s where it went:

	Tiny AAM	GuitarSet
Start: DSP only	0.738	0.519
1. Whitening	0.788	0.619
2. A bigger vocabulary	0.793	0.661
3. A pretrained model	0.913	0.787
4. The bass	0.934	0.872
5. The beats	0.950	0.900

One footnote. After step 3, I found a bug in my own scoring: it stretched Tiny AAM’s last chord over each file’s silent tail. Fixing it lifted every Tiny AAM chord score by about a point, with no change to the analyzer. From step 4 on, Tiny AAM is scored the fixed way.

Where it started: “no chord” won too often

The first version was textbook signal processing. Fold the spectrum into a 12-note chroma, compare it with a template for each chord, and add a flat template for “no chord”.

On real mixes, “no chord” won far too often. It covered 12% of Tiny AAM, where the annotations have 1.5%, and 77% of one track. A full mix’s chroma is never sparse enough for a triad template to beat a flat line.

1. Whitening: 0.519 → 0.619

The fix came from Mauch and Dixon’s 2010 paper: whiten each bin of the spectrum against a running one-octave background, so the notes stand out from the mush around them. Their plugin is GPL, so I implemented it from the paper. “No chord” also stopped being a template. A chord now has to clear a bar, and beats far quieter than the loud ones go to “no chord”. The bass register got its own chroma, which votes for the chords whose tones it holds.

This step also set a rule for the rest: improvements have a price, and the price gets written down. A suite of synthesized test clips has to stay green, so two-beat chord changes and first-inversion chords must survive. Without that constraint, Tiny AAM could have reached about 0.823. With it, 0.788. I took the 0.788.

2. A bigger vocabulary: 0.619 → 0.661

Up to here the analyzer knew three chord types: major, minor and dominant seventh. Anything else had to be named by the nearest of those, often the wrong one. I added major and minor sevenths, minor sixth, half-diminished seventh, diminished seventh and sus4. The templates also learned each note’s overtones, its octave, twelfth and double octave, so a triad’s own overtones stop looking like a seventh. On their own they were worth 2 to 3 points on GuitarSet. GuitarSet’s majmin reached 0.661, and its sevenths score went from 0.492 to 0.531.

Tiny AAM paid for it. Annotated in major and minor only, it scores a sus4 as a wrong triad and a seventh as a wrong seventh chord, and it gave up 0.9 points of root accuracy and 1.2 of its sevenths score. From here on the two datasets had different jobs. GuitarSet’s sevenths was the score to push, and Tiny AAM’s floors capped how much of the new vocabulary the decoder could use.

3. A pretrained model: 0.661 → 0.787

Next I compared the DSP head to head with lv-chordia, a pretrained chord recognizer (Jiang, Chen, Li and Xia, ISMIR 2019). It scored above the DSP on every metric, on both datasets, and the biggest jump in this table is its.

Two things made it acceptable:

License. Its weights are MIT, committed to the same repository as the code. A model whose weights’ terms I can’t state doesn’t go in.
Size. torch alone is over 500 MB. So the model lives in an optional extra, and the default install and the DSP path never import it. pip install chordotomy still works without a model.

The README states what the model was trained on, so anyone who would rather not use it can leave it out.

4. The bass: 0.787 → 0.872

chordotomy also reads the bass under each chord, so slash chords like C/E come out as such. That bass came from a simple DSP rule, and in crowded passages it grabbed the wrong low note. A bass outside the chord is scored as an extra note in it, so false slash chords were quietly costing chord accuracy.

Two changes fixed it. A tied vote over a chord’s beats now prefers a chord tone. And lv-chordia turned out to have a separate output just for the bass, which my code computed and never read. Now that output judges any bass outside the chord.

GuitarSet’s majmin rose to 0.872 and its sevenths to 0.819, but none of that is better chord hearing. The chord names didn’t change; the false extra notes went away. The bass itself did get better: the share of beats with the right bass rose from 0.49 to 0.61. The cost was some real inversions. I let inversion recall fall by at most a tenth, and the bass output took it from 0.227 to 0.208. I found this bug with a single song, which is its own post: The Model Already Knew.

5. The beats: 0.872 → 0.900

lv-chordia labels short frames, and chordotomy snaps the labels to beats. That snap scored below the model’s own frame output, by 1.3 points on Tiny AAM and 2.1 on GuitarSet. Was snapping the problem?

No. The same snap onto the datasets’ annotated beats scored above the raw model. The loss was librosa’s beat tracker, which got only 72 of GuitarSet’s 180 takes right.

Beat This! (Foscarin, Schlüter and Widmer, ISMIR 2024) tracks beats much better, but the beats it picks itself aren’t evenly spaced. Over chords struck once a bar it leaves gaps of up to 1.84 s. So I kept librosa’s steady-tempo tracker and fed it Beat This!’s beat probability instead of librosa’s onset strength:

envelope = beats.activation(y)  # Beat This!'s beat probability per frame
tempo, frames = librosa.beat.beat_track(
    onset_envelope=envelope, sr=SR, hop_length=HOP, trim=False
)

GuitarSet’s beat accuracy (beat F) went from 0.517 to 0.919. The snapped chords now beat the raw model on both datasets: 0.950 against 0.947 on Tiny AAM, and 0.900 against 0.893 on GuitarSet.

It wasn’t free either. 21 of the 180 takes lost majmin, the worst of them fast jazz takes that Beat This! tracks at half tempo, and inversion recall slipped to 0.191, under the floor from step 4. I traced that to the beats moving a few tens of milliseconds, closer to the annotated ones, and accepted it.

One honesty check. Beat This! was trained on GuitarSet, so GuitarSet’s numbers here use checkpoints that never trained on the take being scored, though they had heard other takes of the same tune. The model that ships trained on all 180 takes and scores 0.902.

What made it work

Measure before fixing. The beat step started with “how much does snapping cost?” The answer pointed somewhere else entirely.
Write down the price. Every step traded something: 0.823 for 0.788, Tiny AAM’s root for GuitarSet’s sevenths, inversions for fewer false slashes. Each trade has a floor and a reason in the docs.
Hard constraints that aren’t scores. The synthesized test clips encode what must never break, like a real slash chord held for four beats.
Datasets can’t show what they barely contain. GuitarSet has a few 7sus4 chords, 0.07% of its time, and Tiny AAM none, so the model never output one there, and a bug that turned every 7sus4 into sus4 never moved a score. A real song found it in minutes.

The DSP engine still exists for anyone who’d rather skip the model, at 0.803 on Tiny AAM and 0.696 on GuitarSet. Both engines ship in one package on PyPI (uv tool install "chordotomy[model]"), and the viewer plays a recording with its chords.