The goal of guitar stylometric analysis is to understand what a user might be playing on a guitar and characterize how they might be playing it. Standard sheet music is a very limited way to convey musical ideas. If you give two people the same sheet of music, chances are they will play it slightly differently. Everyone has their own artistic flair.
Everything that an artist is doing on an instrument (in terms of end-effectuation) can likely be sensed on the guitar and modeled by the principles of computer science and engineering. Whether we have the appropriate sensing capabilities to build appropriate classification is debatable. Can such artistic fluctuations can be captured and appropriately modeled?
The goal of my latest hack project is to try to build a Robotic sensing architecture and system for guitar that could be used to capture stylistic playing effects and understand them using scientific and engineering lenses. To make the system widely usable, it is built with only widely available consumer electronic devices without requiring special hardware. Thus far, I have experimented with two sensing modalities:
- Use of smartphone (with accelerometer) mounted on guitar body to detect vibration patterns near guitar strings
- Use of smart watch on user’s wrist to capture guitar strumming patterns.
Note I haven’t done machine learning yet on this data, and have mostly just applied standard signal processing techniques to it. I have just started building the platform and have yet to collect that much data. Once I get more data, I think I will be able to build more intricate computational models and do some cooler stuff applying large scale ML personalization work to this. My goal is to the apply the Lunar Tabs Machine Learning engine to such a collected data set. See this working paper for building high-dimensional, intricate (but computationally efficient) music personalization models:
Though the paper is written for guitar learning and difficulty analysis, the underlying Active Learning methods are powerful and expected to scale to the high-dimensional feature space of artistic perception and find the major features explaining artistic variation. Building these very intricate models of artistic variation is the eventual goal of the project.
However, I am first starting by building a good sensing platform to collect the right data for use in such models. It is neat what a couple of well-placed sensors on the guitar and user can do to get data set of explanatory guitar playing data.
Smart Phone mounted on Guitar
A smartphone (with accelerometer) mounted on the guitar can be used to capture vibration patterns on the guitar surface. Such a sensing modality can readily capture playing effects such as vibrato, percussive effects, and more. See figure below for setup.
My guitar has both a mounted mobile phone and a hexaphonic pickup. The mobile phone provides vibration pattern data while the hexaphonic pickup provides midi. Both can be used to characterize what the user is playing in real-time. Both are also held together shamelessly by tape which is an engineer’s best friend (don’t try this at home just yet).
Here are some of the effects we can detect with the mounted hardware.
Detecting Vibrato
Vibrato is a musical effect consisting of regular, pulsating change of pitch. Vibrato is characterized by periodicity in the accelerometer signal that can be identified using frequency domain analysis.
This is a counter-intuitive statement but…
It can actually be easier to detect vibrato from motion than audio. Since audio is often distorted by guitar electronic noise and other junk, it might take a bit more work to extract the raw signal that you’re looking for that you could get straight from the vibration source.
Using a Fast Fourier Transform (FFT) package, it was fairly straight-forward to (1) Run FFT on the accelerometer axes and (2) Threshold on magnitude of power of the real component to detect the presence of vibrato in real-time.
There is the the problem of false positives — a person might shake the guitar and it would still produce a periodic signal on the accelerometer. To counter this, a set of empirical thresholds for the algorithm can be chosen using Receiver Operating Characteristic (ROC) curve analysis to find the right set of thresholds to maximize the probability of dismissing the false positives. Vibrato has a particular frequency band that we can leverage.
Detecting Percussive Effects
Percussive guitar is a genre gaining in popularity. Pioneered by artists like Tommy Emmanuel (and some fairly cool you tube videos), percussive guitar players take advantage of tapping and other effects on the body of the guitar to produce percussion-like sounds.
Percussive effects can be readily detected by accelerometer vibration patterns on the guitar surface. The simplest ones are spikes that can processed using a band-pass filter. However, detection of signal can likely be boosted with Machine Learning techniques like PCA that can build a good shape model for different types of spikes. If you really want to get intricate in building signal models, see my thesis work (which I am still working on).
Aligning with Midi
Note that my guitar set-up screen shot showed a mounted Fishman Triple Play (http://www.fishman.com/tripleplay/details). Hexaphonic pickups such as these provide the capability to convert a standard analog guitar into midi. These products allow detection in real-time of what notes/chords the user is playing, conversion to midi, and then streaming over bluetooth to a usb receiver.
Midi is very very useful because it tells you very precisely what the user played in terms of a stream of time-stamped notes. Notes close together in a midi stream are clumped into chords. You could try to figure out what the user is playing in standard audio signal processing, but it’s messy and less accurate. Hexaphonic pickups greatly simplify the signal-processing problem in my experience. I highly recommend them since they’ve made my life so much easier.
Time-aligning the midi and detected vibration effects can help build a pretty intricate profile of what notes the user played and how they were played. Imagine using this system to capture artistic style from your favorite artist so you can have a make a better cover. Perhaps you could develop your own too and share your “style” with the world more quantitively in some type of multi-modal tab. Possibilities yet to be explored.
Detecting Strum Patterns using Smart Watch
The latest smart watches come with accelerometers, and the engineering challenge is to figure out what to do with them.
One use case I am working with is to use a smart watch to track a person as they are strumming the guitar. There is certainly application to online guitar learning. If a person’s strumming can detected, an application could help instruct whether an appropriate up/down stroke was made at the right time in playing a particular piece. I tend to mess up strum patterns but an intelligent app could tell me when I’m flubbing things up.
I have data collected from a Pebble smartwatch while I wait for other smart watches to come out. Here is my pebble (complete with a flappy bird app!)
Why did I choose to get a Pebble with the upcoming Android Wear and IOS smartwatches? Well, for Robotic enthusiasts like me, it’s really important to get actual representative sensor data to play with. Actual hardware and actual sensor always beats building stuff in idealistic simulation. If you want to get stuff to work in the real world it better well match statistical distributions of actual sensor data.
Standard chord guitar strumming produces a periodic vibration pattern that one should readily be able to apply FFT approaches to.
The data is very clean and very periodic. Up/down strokes on the guitar are very well tracked by the smart watches accelerometer. Each of the ups and downs of the signal corresponds to an up or down strum with my pick on the guitar.
I believe it is possible to capture plenty of diverse strum patterns in computational models of smartwatch accelerometer data. Different strum patterns and patterns at different tempos will have a unique time series profile that can be fingerprinted. Imagine a guitar app that actually helps you train your strumming.
An even more interesting thing that can be seen in the data is with alternate picking.
Interestingly, the accelerometer signal is still roughly periodic but has a linear component that corresponds with the vertical displacement of the user’s hand. This information could be used by an intelligent guitar teacher (or app) to help a student improve picking technique (since that can be a challenging thing to learn).
Next Steps
I would like to use this platform to collect data, and further improve the detection techniques for stylistic effects. I think there’s also neat things that could be done with fusing the different sensed modalities that haven’t been explored yet.
Also, I believe there is a lot that can be done with Machine Learning on this data to understand individual artistic nuances in playing guitar and for effective learning of different musical styles. Modeling stylistic effects actuation on instruments can help us better understand our musical capabilities.
And of course, the real (implicit) goal of this is to understand better how humans actuate music to allow Strong AI robotic agents to transcend human capability at creating music and art…
Dr. Prateek Tandon 