10

Hopefully this is not too subjective...

I'm looking for some direction in efforts to detect the different "parts" of a song, regardless of musical style. I have no idea where to look, but trusting in the power of the other StackOverflow sites, I figured someone here could help point the direction.

In most basic terms, one could detect different parts of a song by just grouping consecutive repeating patterns and calling those a "part". That's maybe not so hard -- computers are pretty good at detecting repetition in a signal, even when there's some small variation.

But it's hard when the "parts" overlap, as they do in most music.

It's hard to say what kinds of music would be most well-suited to this kind of system. I would guess that most classical-style symphonic music would be easiest to process.

Any ideas of where to look for research in this area?

themirror
  • 201
  • 1
  • 2
  • I think there is an iPhone app for recognizing the song from the snippet recording. And I think there was an article describing this app. I am sorry I do not have links, but I would start with that. – mpiktas Jul 18 '11 at 07:55
  • 1
    @mpiktas: you're probably referring to apps like [SoundHound](http://www.soundhound.com/) or [Shazam](http://www.shazam.com/). There is [a whitepaper](http://www.ee.columbia.edu/%7Edpwe/papers/Wang03-shazam.pdf) on how Shazam works, although it does not go too much into details. Although I'm not sure that's what the OP needs, it may be a good starting point. – nico Jul 23 '11 at 08:39
  • For a good, short blog post on Shazam (based, I believe, on the white paper), you can also try http://laplacian.wordpress.com/2009/01/10/how-shazam-works/ – raegtin Jul 24 '11 at 19:56

5 Answers5

6

I'm no expert on signal processing, but I know a fair bit about music theory. I'd say that, on the contrary, classical music would probably be some of the hardest music to analyze by simple mathematical methods. You'd best start with something simpler and more repetitive, such as pop or techno music. Pop often has a verse-chorus-verse...etc format that might be conducive to a simplistic version of your goals.

Try using a Fourier Transform on your data to break it into its most prominent constituent frequencies, maybe hierarchically among different subsections. In particular you can look for different things based on how you want to group the "parts" of your data.

  1. The slowest oscillations in your pop music will probably be the shifts between verse and chorus and back to verse (maybe 0.75 oscillations per minute?).

  2. Next you might find higher frequency oscillations among your chord progressions, that is, among each full measure of your song (maybe around 6 oscillations per minute?).

  3. Next highest frequency I'd think would be a bar within a measure (maybe about 24 oscillations per minute?) within which the strumming pattern and syncopation of lyrics often repeat in pop/folk music.

  4. Getting down into the gory details, next you'll find the beats and rhythms that repeat within each bar of your music. Picking out and isolating one of these (at maybe 148 oscillations/beats per minute?) would likely yield a bass drum kick, or a cowbell hit, or something along a similar order.

  5. Somewhere in between beats and tones you might to find rapid stylistic elements of your music such as speed/sweep picking on an electric guitar, or fast vocal rapping rhythm. (I have no idea how fast these might be, but I would guess somewhere on the order of 1000 beats per minute or more).

  6. Lastly, fastly, and probably most complexly, are the elements of tone and timbre. I know that the "middle A" note is standardized to be 440 Hz, that is, 440 oscillations per SECOND. I'm sure there are techniques for differentiating based on tonal quality and timbre what kinds of instruments are being used; there are even fairly good algorithms for detecting human vocals. However like I said, I'm no signal processing expert.

machine yearning
  • 291
  • 1
  • 2
  • 8
6

The music is usually described using MPEG7 descriptors with some additional stuff like MFCCs calculated on the chunks of piece made by some moving window approach (i.e. you have some window size and hop, start with the window placed on the beginning of the sound, calculate the descriptors on the window, then move it by hop and repeat until the end is reached).
This way a piece is transformed into a table; in your case it can be used to apply some clustering on the chunks and so detect those "parts".

5

There are a lot of different methods and a plethora of literature on this topic from a wide variety of perspectives. Here are a few highlights that might be good starting points for your search.

If your background is more musical than mathematical or computational you might be interested in the works of David Cope most of his published works focus on the analysis of classical music pieces, but he has a private venture called recombinant that seems more general. A lot of his work used music as a language type models, but I believe at least some of his most recent work has shifted more toward the whole musical genome like approach. He has a lot of software available online, but it is generally written in Lisp and some can only run in various versions of Apple's OS though some should work in Linux or anywhere you can get common lisp to run.

Analysis of signals and music in general has been a very popular problem in machine learning. There is good starting coverage in the Christopher Bishop texts Neural Networks for Pattern Recognition and Pattern Recognition and Machine Learning. Here is an example of a MSc paper that has the music classification part, but has good coverage on feature extraction, that author cites at least one of the Bishop texts and several other sources. He also recommends several sources for more current papers on the topics.

Books that are more mathematical or statistical (at least by their authorship if not by their content):

Since I mentioned Bishop and the computational perspective of machine learning I'd only be telling half the story if I didn't also suggest you take a glance at the more recent Elements of Statistical Learning (which is available for free legal download) by Hastie, Tibshirani, and Friedman. I don't remember there specifically being an audio processing example in this text, but a number of the methods covered could be adapted to this problem.

One more text worth considering is Jan Beran's Statistics in Musicology. This provides a number of statistical tools specifically for the analysis of musical works and also has numerous references.

Again there are many many other sources out there. A lot of this depends on what your background is and which approach to the problem you're most comfortable with. Hopefully at least some of this guides you a bit in your search for an answer. If you tell us more about your background, additional details about the problem, or ask a question in response to this post I'm sure I or many of the others here would be happy to direct you to more specific information. Best of luck!

Chris Simokat
  • 862
  • 1
  • 6
  • 11
4

Not a great answer but two places to look for research are:

International Society for Music Information Retrieval has tons of published papers about just this topic, amazing how much info there is www.ismir.net

& Echo Nest (A Startup with an API to do similar stuff) echonest.com

UPDATE: they also released some open source fingerprinting code. http://echoprint.me/

Community
  • 1
David Pitkin
  • 161
  • 2
0

I was interested in the similar problem. Here is the solution. It is not so old scientific proposal that is called scape plot. See this article for details (it looks nice).

In addition, I would recommend you to also visit author's website since there is a lot of similar applications of statistics in music. When searching for other similar sources, I recommend to use the term Music Information Retrieval that includes similar areas.

sitems
  • 3,649
  • 1
  • 25
  • 52