The Berkeley Science Review: Articles

The Sound of New Music
Expanding technology's horizons at CNMAT (view PDF)
by Meredith Carpenter

At first glance, the device looks like a futuristic TV tray. The top is covered with a grid of black squares glowing with red and green LED lights, and a patchwork of electronics and wires connects the machine to a nearby laptop computer. Professor David Wessel sits on a stool positioned behind the setup. Hunching down slightly, he begins rhythmically tapping the squares as though typing on a keyboard. While this may seem like the prelude to some complicated experiment—or perhaps a spaceship launch—Wessel is, in fact, playing music. This improvised composition of layered electronic sounds, so unlike a Beethoven sonata or Mozart fugue, epitomizes the genre of "new music."

Wessel, a music professor at UC Berkeley, is Co-Director of the Center for New Music and Audio Technologies, or CNMAT. Founded in 1987 by composer and Professor Emeritus Richard Felciano, CNMAT seeks to foster productive interactions between music and technology. With an interdisciplinary research program that includes music, physics, computer science, psychology, mathematics, engineering, and even architecture, CNMAT's ultimate goal is, according to Co-Director Edmund Campion, to conduct "research that has a musical application at the end of the day." These applications range from new sensor- based technologies for producing music (such as the device described above), to hearing aids with improved capabilities for music listening, to a better understanding of how the brain acquires musical knowledge.

Beyond the blue door
Headquartered in a distinctive blue-doored house on Arch Street in North Berkeley, CNMAT was modeled after the famous Institut de Recherche et Coordination Acoustique/Musique (the Institute for Music/Acoustic Research and Coordination, or IRCAM) in Paris. According to the IRCAM website, the goal of founder Pierre Boulez was "to bring science and art together in order to [develop new instruments] and rejuvenate musical language." In the same way, says Campion, "the notion of bridging the humanities and the sciences is one of the reasons we exist. The goal is to constantly renew, revise, and revisit the materials of music in the hopes of inspiring new music."

According to CNMAT's website, "new music exhibits novelty in its situation, presentation, compositional process, performance practice, or outcome. In some pieces, that novelty is dependent on or derived from the use of technology." Thus, the CNMAT facility contains computer labs as well as classrooms, performance space, recording studios, and offices, and the Department of Music offers several undergraduate classes that teach students to use software developed at CNMAT. In addition, CNMAT offers summer workshops for musicians and artists. Workshops offered this past summer included a handson course for creating sensor-based instruments, in which students learned to use conductive and piezoelectric fabrics to create instruments that can be used in musical performance, dance, and art installations. Piezo- electric materials generate an electrical signal in response to stress or force, making them ideal for creating new instruments that are activated by bodily movements or gestures.

Musical gestures
These so-called "gestural controllers" are a main focus of research at CNMAT. "I think we're in a kind of renaissance now in engaging the body in interaction with computing devices," says Wessel. "Witness the Nintendo Wii's success." In fact, Wii controllers are used in the workshop as starting materials for new gesturally controlled instruments. According to Wessel, "One of the really critical things we like to do here is to work on these interfaces for control. Laptops and mobile computing devices are pretty powerful, but it doesn't seem quite so interesting to see someone behind a screen on a concert stage—you wouldn't know if he was reading his email or doing office work or what—so we want to have gestural controllers that function more like musical instruments by engaging the body."

Gestural controllers also open up a whole new realm of flexibility that acoustic instruments cannot offer. For example, the keys of Wessel's multi-touch keyboard can communicate both spatial information (where the key is touched in the left-right and up-down dimensions) and pressure information. A computer program then generates sounds based on pre-programmed settings for each key, with variation depending on how exactly the key has been touched. Because the sounds produced and their qualities like pitch, volume, and timbre depend on computer-programmed settings for the controller, the controller itself does not have a characteristic sound. However, its mode of use can impart certain properties; for example, depending on how it's programmed, a Wii controller might allow for more sliding sounds than a controller with keys alone.

Adrian Freed, Research Director at CNMAT and the instructor of the sensor workshop, is CNMAT's resident sensor guru. One of his most recent prototypes is a drum-like controller called "the tablo." To create the device, Freed stretched an electrically conductive silver-plated fabric over an embroidery hoop. He then pulled the taut fabric down over an inverted bowl that he had covered with strips of conductive plastic. Pressing down on the fabric distorts it so that it touches the plastic sensors on the bowl in different places, shorting them out at the point of contact. A microprocessor measures the resistance of the sensors along their entire length and passes the information to a computer that actually produces the sound. The signal changes based on the location of the contact between the fabric and the sensor, allowing for flexibility in how the tablo can be played.

Conductive fabrics have become one of Freed's favorite materials for building new controllers. "I'm using fabrics partly because I'm in a hurry, because I have to try all kinds of things and work quickly with lots of prototypes," says Freed. "The conventional way of making these kinds of sensors is to make circuit-boards, which take days to come back from the manufacturer. With fabric, I can just take scissors and cut what I need." So why not just clothe yourself in conductive fabric and dance around? In fact, students at this year's sensor workshop did just that. "We had people wanting to build wearable light shows and clothing that makes sound—clothing that can be musical instrument interfaces," says Freed. However, he emphasizes that the source of inspiration for his new instruments is not the technology or the materials alone. "The way we run things around here is that music drives the technology. The technology in a certain way is sort of neutral and empty, waiting for things to need it."

Timing is everything
So why aren't these electronic instruments more popular? "Part of the reason that a lot of people still prefer acoustic instruments to electronic instruments is because the electronics are too slow," says Freed. "They're not accurate enough in communicating gestures. The sensors are pretty good these days, but there's been a long history of not taking time into account." To address this problem, Andy Schmeder, CNMAT's Research Programmer, recently investigated the timing requirements for transmitting gestural information from an electronic instrument's microprocessor to the computer, usually through a USB connection.

"Musical gestures are characterized by the need to transmit information that changes over a very fine time structure," explains Schmeder. "We're looking at the limits of expressive gestures that you can make with your finger, or some other device like a stick or a musical controller, and still transmit the information faithfully." It turns out that the time information has to be highly precise in order to communicate the gestures to a computer without losing any signal quality. "Acoustic instruments already have this, because they don't need to transmit the signal somewhere else. But this is something we need to be aware of if we're going to build a new instrument that's based on these technologies," says Schmeder.

These limitations, of course, are only part of the reason that electronic instruments haven't been more widely adopted. Many musicians are understandably resistant to the idea of abandoning the instruments they've been playing for years. "We are very interested in the skill set of a musician who spent thousands of hours learning an instrument. We don't want to ignore that, so augmenting tra- ditional instruments is also very important," says Wessel. "For example, we worked with Gibson Guitar on various kinds of processing to make the guitar more of a digital device." While some of CNMAT's funding comes from the university and individual donors, it has also participated in numerous collaborations with industry partners like Gibson, Yamaha, and Meyer Sound over the years. These partnerships are encouraged by the availability of UC Discovery Grants, which provide matching funds from the state for money invested by industry in research.

Music to my hearing aid
One current industrial partner is Starkey Laboratories, a hearing aid company with a Berkeley-based research center. CNMAT researchers are working with Starkey to optimize hearing aids for music listening. "People who wear hearing aids complain bitterly that their hearing aids don't sound right, and often they'll just turn them off when they're listening to music," says Wessel, who works on the project. The goal of the current research is to allow hearing aid users to better adjust them for music.

"Hearing aids are really tuned for speech intelligibility, and music is more complicated in many ways," Wessel explains. "First of all, there are a lot of varieties—everything from Gregorian chants to quick-tempoed Balinese music—so the settings that you would use for your hearing aid to best enjoy these different types of music would probably need to be varied." Because hearing aids are generally fitted only by an audiologist, who measures a person's hearing loss and sets the parameters accordingly, additional daily adjustment is not currently an option.

Eric Battenberg, a graduate student in the Department of Electrical Engineering and Computer Sciences (EECS), hopes to change that with a user-friendly computer program. "The idea is to create an interactive program that allows the user to locate hearing aid settings best tailored to that person's type of hearing loss," he explains. However, hearing aids can have over 70 different settings, which would be a challenge for a user to adjust manually. "You don't want to give people too many parameters to adjust, so it's important to keep the dimensionality of the task very low," says Wessel.

Instead, Battenberg's software starts with just a few combinations of parameters that correspond to common types of hearing loss. The user listens to music with each parameter set engaged and then lays them out on the screen based on their apparent similarity, placing similar sounding sets near to each other and distant ones further apart. These pre-established sets provide landmarks on an auditory "map" that the user can navigate between to find his or her preferred settings. As the user mouses across the screen, the software calculates a set of parameters that is the middle ground between the surrounding sets. When the user settles on a particular location that sounds best, the set of parameters at that location is recorded for subsequent use in a hearing aid.

The researchers wanted to test this technology using subjects with normal hearing, who are accustomed to listening to music on a daily basis, before taking it into tests with the hearing-impaired. To do so, Andy Schmeder created an audio processor that works like a hearing aid. "It basically simulates how a hearing aid operates by amplifying quiet sounds, but without making loud sounds even louder," he explains. "This can be combined with a computerized model of hearing loss in order to give a normal person the feeling of what it's like for the person wearing the hearing aid to be listening to music." Ultimately, the researchers hope to use their technology to augment the fitting of hearing aids. "We don't want to put audiologists out of business, but it'd be nice if people could adjust things on their own," says Wessel. "I could see this happening with an iPhone-like handset with a multi-touch interface, where you would adjust your parameters from day to day—even while you're sitting in a concert."

Attack of the Martian tritaves
Although some CNMAT thesis projects involve creating a new technique or technology, few PhD recipients can claim to have created an entirely new system of music. But that's exactly what CNMAT researcher Psyche Loui did for her psychology PhD, which she received in 2007. "For my thesis, I asked the question of how humans know what they know about music," says Loui, now a research instructor at Harvard Medical School. Along with Wessel, her thesis advisor, Loui sought to determine how the human brain acquires musical knowledge by studying how people learn an entirely new musical language. "Most of the music we hear in our everyday lives is composed according to certain rules and principles—for instance, songs belong to a certain key," explains Loui. "And evidence shows that people, regardless of age and musical training, know these rules implicitly." However, is this knowledge a reflection of how the auditory system works, or simply a result of the culture in which you were raised? To address this question, "one could either compare music from other cultures, or look at how old an infant has to be before they demonstrate sensitivity to these musical rules, or you could look at how humans learn a completely new musical system," says Loui.

She chose the last option because it allows for tighter control of the experiment—she was able to control the amount of exposure her subjects had to the musical language she created. "My idea was to invent a set of completely new musical rules—'Martian' music, if you will—and get people to listen to it, and then test them for what they know." To create her Martian music, Loui turned to physics. Most musical systems around the world use the octave scale, which is based on a two-to-one frequency ratio between notes that are one octave apart. Thus, if one note has a frequency of 440 Hertz, the note one octave above is at 880 Hertz, and the note an octave below is at 220 Hertz. Additionally, within the range of an octave, the common Western system has 12 divisions—the notes A through G plus a total of five sharp and flat notes. Loui's system, however, used a novel scale called the Bohlen-Pierce scale, which is based on a three-to-one frequency ratio between so-called "tritaves." One tritave above 440 Hertz is three times 440, or 1320 Hertz, and there are 13 divisions within each tritave.

Armed with this new musical system, Loui then had to compose songs to use in her experiments. Fortunately, says Loui, "it turns out that you can compose chords in the new scale and then compose chord progressions by putting these chords together. It sounds unlike any music that anyone is used to, but it still makes sense." Loui played this music to her experimental subjects (Cal undergrads receiving class credit) for half an hour. She then tested them on what they had heard by playing them the same melodies again, different melodies from the same system, and melodies from the traditional octave system. She was excited to find that, given exposure to a sufficiently large set of melodies, people could not only recognize tritave melodies they had just heard but also identify new melodies that belonged to the tritave system. "Essentially, people demonstrated generalized knowledge of the musical rule, not just rote learning what you played them," she says. "This suggests that the human brain can rapidly and flexibly pick up on sounds in the environment and learn rules and structures from these sounds."

Next, Loui examined whether listening to the new musical system elicited the same brain responses as Western music. "Usually, when music doesn't end the way you expect it to, the brain emits certain electrical signals," she explains. Loui collected electroencephalograms (EEGs) while subjects were listening to chord progressions in the new musical system. Her goal was to determine whether unexpected chord progressions would elicit the same brain signatures as those from Western music. "We did, in fact, observe the same brain response for infrequent chords in the new musical system, and these responses increased over time," says Loui. "As subjects sat there listening to the new music, the brain signals became increasingly sensitive to the structure of the sounds, again demonstrating that the brain can absorb the statistics of sounds in the environment and calculate their frequencies."

For her current work, Loui has turned her attention to the neurobiology of "congenital amusia"—otherwise known as tone-deafness. In a recent paper in the journal Current Biology, Loui and her collaborators investigated why tone-deaf people, while unable to consciously perceive pitch differences, can nevertheless speak fluently. Her results suggest that the brain has evolved multiple pathways for sound perception and production, such that the pitch information necessary for fluent speech can be acquired separately from the pathways needed to consciously perceive pitch differences.

A few final notes
What lies ahead for CNMAT? While most "new music" has yet to hit the Billboard charts, some is now being made available on the CNMAT website, which also holds a library of software for music composition and production developed by CNMAT researchers. "The website is a way of allowing the fruits of what's generated at CNMAT to be shared with the rest of the world," says Campion. "It's there as a trace of our history and demonstrates what we do in both research and creative production." Because all of the software was created using a common programming environment, called Max/MSP, musicians around the world can use the software and even design their own applications to contribute to the website.

Indeed, while CNMAT researchers remain on the cutting-edge of research in a wide array of disciplines, serving the musician remains their primary goal. "In some ways, I'd rather have called it the Center for New Music and Musical Science," says Wessel. "We like to develop technology, but it's re- Feature New music ally the music that is driving everything. We want to serve composers, serve the musical imagination." And that imagination, it seems, is limitless.

Meredith Carpenter is a graduate student in molecular and cell biology.

Want to know more? Check out:
cnmat.berkeley.edu

The Evolution of Music
Walking around the UC Berkeley campus between classes, it seems nearly everyone is wearing headphones. Although tastes may vary widely, we see music everywhere, in all cultures. How did this fundamental part of the human experience evolve?

"There's always been a debate about whether or not music serves any adaptive purpose," says Professor David Wessel, whose background is in mathematical psychology and who is interested in these basic questions about the evolution of music. Wessel disagrees with Harvard psychologist Steven Pinker, who, in his famous book How the Mind Works, said music is a side effect of evolution—"auditory cheesecake, an exquisite confection crafted to tickle the sensitive spots of...our mental faculties," but not adaptive in itself.

"In my view, music certainly played a role in how language evolved, and particularly the way emotional states are communicated one to another," says Wessel. "I would also say that music functions in bringing people together and promoting social cohesion." Steven Mithen, professor at the University of Reading and author of the influential 2006 book The Singing Neanderthals, also disagrees with Pinker's hypothesis, as do many other researchers in the field. "Mithen pronounced the auditory cheesecake idea as Ôdaft' at the neuroscience and music meeting in Montreal this summer," says Psyche Loui, a former CNMAT graduate student who studies how the brain processes music. "The audience applauded after the word 'daft.'"

Mithen hypothesizes that music and language both evolved from a common proto-language used by Neanderthals and the ancestors of modern humans. He calls this communication system "Hmmmmm," which is actually an acronym that stands for Holistic, manipulative, multi-modal, musical, and mimetic. Essentially, this form of communication (not yet a "language") was composed of messages rather than words, often accompanied by gestures, and had characteristics of music like pitch and rhythm. Unlike traditional depictions of early human language that consist of words without grammar ("me make fire"), Hmmmmm consisted of phrases with more complete meanings that were later segmented into discrete words. Birds use a similar system of communication, using different calls for different situationsÑthe calls each have meanings meant to manipulate the behavior of other birds, though you wouldn't call them a language.

But that may not be the whole story. As Darwin first suggested, sexual selection may have also been involved (data from Mick Jagger certainly support this hypothesis). "From birds to mammals, song seems to play a role in the whole sexual game and mating," says Wessel, and Mithen agrees that sexual selection could have spurred the evolution of the musicality of Hmmmmm. Other factors, such as the ability of music to pacify infants, may have also played a role. Though many of these hypotheses are difficult to test empirically, people will surely continue to theorize about music's origins for years to come. "As neither the enjoyment nor the capacity of producing musical notes are faculties of the least use to man," wrote Darwin, "...they must be ranked amongst the most mysterious with which he is endowed."

Comments on this article? Drop us a line at with 'letter to the editor' in the subject!