Interfaces for Musical Expression Based on Simulated Physical Models

This thesis is concerned with the design of interactive virtual musical instruments intended to augment acoustic instruments in live performance. For the purposes of this work, a virtual musical instrument is defined as a computer system designed to facilitate musical expression and/or exploration. The aim of the research is to develop understanding of the nature of virtual instruments and how musicians interact with them. The approach has been to use participatory design techniques to develop a series of virtual instruments for use in live performance and then to examine closely the experiences of musicians who use them.

An interaction design strategy which uses simulated physical models to mediate between the sounds produced by acoustic instruments and computer generated sounds and visuals has been developed. In this approach, a simple physical system is modelled in software and characteristics of acoustic sounds are mapped to forces and other parameters which affect the model. In response the model moves in ways that are physically realistic. These movements are then used as parameters to control video and audio synthesis.

Using a research approach which draws on action research, design science and participatory design, a series of virtual instruments which use this interaction technique were developed and used in live performances. A set of initial design criteria which guided development were identified. In order to refine these criteria and better understand the impact that using these virtual instruments has on musicians' music-making, a series of user studies were conducted. A number of expert musicians used the virtual instruments and discussed their experiences. These sessions were video-recorded, transcribed and analysed using grounded theory techniques.

The results of the study identified three modes of interaction with the virtual instruments: instrumental, conversational and ornamental. Musicians interacting with the virtual instruments in instrumental mode emphasise the importance of being in control and being able to trust that the instrument will respond consistently. When musicians use a virtual instrument ornamentally, they surrender detailed control of the generated sound and visuals to the computer, allowing it to create audio-visual layers that are added to the musicians' sound. The more complex, (and difficult to design for) conversational interaction involves the sharing of control between the musician and the virtual instrument. The balance of power is in flux, allowing the virtual instrument to 'talk back' to the musician, reflecting and transforming the sonic input in ways that move the performance in new musical directions.

The contributions of this thesis are therefore:-

  • a set of virtual musical instruments which use a unique interaction paradigm in which simulated physical models mediate between live sounds produced on acoustic instruments and computer generated sounds and visuals;
  • a theory of musician-virtual instrument interaction; and
  • a set of design criteria informed by practice and user studies.

Download the full thesis.