Korva is a scroll of the internet’s best ideas.
Music AI listens to frequencies, not music
About 14 years ago, Bob Sturm showed that music classifiers were behaving like Clever Hans horses. He applied equalization filters and delays to songs, and the classification changed. Rock with a delay added could suddenly come back as reggae.
Why the label should change is a causal question: which part of the audio caused the model to say rock or reggae? Asking a person why a piece is the blues has no simple answer either. There is rhythm, instrumentation, harmonic progression, and phrasing. The models, though, are probably doing shortcut learning on frequencies. If they were using musical information, it would not be possible to subtract a few frequencies and have a listener believe they were hearing Bach.
One more test makes it plain. A hundred separate minimal blues signals were blended together, and the classifier called the result blues. To a listener it sounded like something heard from inside the womb, or outside a festival tent. These models do work, and their accuracy is very good, but they don't work in a way a person would recognize or understand. Fourteen years on, the problem has not improved, and may be worse.
Why would a genre model learn shortcuts?
A model is rewarded only for matching labels, so it takes whichever signal predicts them most cheaply. In music datasets, recording conditions, production style, and loudness often travel with genre, and those show up as frequency patterns. Bob Sturm's analysis of the widely used GTZAN dataset, which holds 1,000 thirty-second clips across ten genres, documented repeated tracks, mislabeled clips, and distorted audio. A model can score well on such data without learning anything about harmony or rhythm.
Why does adding a delay change the genre?
A delay adds a copy of the audio shifted slightly in time, which changes the pattern of frequencies across the clip. A human hears the same song with an echo. A model that relies on frequency patterns sees a different input and may move to a different label. Equalization works similarly by boosting or cutting parts of the spectrum, which changes the very features the model leans on.
How do researchers test what a model listens to?
One approach is to change an input in a controlled way and see whether the output moves. Researchers remove parts of the audio, filter it, or blend signals together and compare the predictions. If a label survives changes a listener would find destructive, or flips under changes a listener would not notice, the model is using something other than what people hear. Sturm's tests with filters and delays were an early example of this approach for music.