- Split View
-
Views
-
Cite
Cite
Asha Ward, Tom Davis, Ann Bevan, Music Technology and Alternate Controllers for Clients with Complex Needs, Music Therapy Perspectives, Volume 37, Issue 2, Fall 2019, Pages 151–168, https://doi.org/10.1093/mtp/miz006
- Share Icon Share
Abstract
Music technology can provide unique opportunities to allow access to music-making for clients with complex needs. While there is a growing trend of research in this area, technology has been shown to face a variety of issues leading to underuse in this context. This literature review is a collation of information from peer-reviewed publications, gray literature, and practice. Focusing on active music-making using new types of alternate controllers, this review aims to bring together information regarding the types of technology available, categorizes music technology and its use within the music therapy setting for clients with complex needs, catalogues work occurring within the field, and explores the issues and potentials surrounding music technology and its use in practice.
Introduction
This article provides a review of music technology used in music therapy, examining the types of technology currently used for sound exploration and music-making by both music therapy practitioners and researchers. The review highlights key developments within the field of music technology, with a focus on applications in music therapy for those with complex needs. The data gathered are a collation of peer reviewed and grey literature (institutional reports), alongside firsthand research carried out by the first author as part of an engineering doctorate.
The primary focus of this review is technology for active music-making, with a focus on alternate controllers that provide control and potential for expression through sound and music. For this review, active music-making is defined as playing instruments or actively exploring sound through interaction with technology. The term complex needs refers to a spectrum of cognitive, physical, and/or sensory impairments or disabilities that can lead to individuals experiencing minimal movement, disordered movement, altered states on consciousness, and/or no verbal communication (Magee, 2012).
Literature Review Strategy
Keyword searches of Google Scholar, Google, and the Bournemouth University Library Catalogue were used for article selection. The following keywords were used: music technology for music therapy, new interfaces for musical expression, music technology and special education needs, music technology SEN, and music technology complex needs. The Nordoff Robbins Evidence Bank (2014) (specifically account no. 16) was also consulted as well as Research and Resources for Music Therapy 2016 (Cripps, Tsiris, & Spiro, 2016). This selection of papers expanded as literature was reviewed. Papers were scanned for their significance as they pertained to the use of technology, both novel or off-the-shelf, with users with complex needs for active music-making or sonic exploration, or that they featured details of such technologies in use, or that they explored issues around and/or reviewed usage of such technology in use. Some grey literature was also consulted (Department for Education, 2011; Farrimond, Gillard, Bott, & Lonie, 2011; Ofsted, 2012; O’Malley & Stanton Fraser, 2004) as this provided a different perspective on technology usage in practice.
Context
Music technology reviews have been undertaken to address the use of music technology by music therapists (Cevasco & Hong, 2011; Clements-Cortes, 2013,Crowe & Rio, 2004; Hahna, Hadley, Miller, & Bonaventura, 2012; Knight & Krout, 2017; Knight & Lagasse, 2012; Magee, 2006; Magee & Burland, 2008; Streeter, 2007; Whitehead-Pleaux, Clark, & Spall, 2011), and to outline the aims of national music education plans within government policy (Department for Education, 2011; Ofsted, 2012), and to guide government policy (Farrimond et al., 2011). Magee (2014) edited a volume of articles drawing together uses of music technology in therapeutic and health settings. These authors highlighted the importance of music technology, the types of music technology used, where technology is useful, and how technology could be improved to break down barriers and allow access to music-making for those with complex needs. This literature review aims to take another step in this discussion, by further organizing this information and providing a timeline of development to the current state of the art.
The use of music technology for clients with complex needs in music therapy settings is broad, drawing from a variety of fields. Technology usage combines elements of human computer interaction (HCI), music therapy, music psychology, music education, and music technology. The scope of literature featured in this review reflects this, with a focus on the ways technology can be used to increase access to active music-making opportunities for clients who are unable to use traditional musical instruments.
The review begins with an overview of the evolution of technology, electronic music technology (EMT), and EMT for those with multifaceted needs. Following this are sections on the computer as the bridge, new developments, available technology, and music technology used in music therapy practice, including issues related to these practices.
Evolution of Technology
Within the last two decades, technology usage in general has become increasingly accessible (Cevasco & Hong, 2011). This has proliferated into all areas of everyday life as devices such as smartphones, smart watches, tablets, and portable computers become more ubiquitous both inside and outside of the home (Nagler, 2011). In the broader view, this explosion of technology has led to changes in how we interact with music (Misje, 2013). Software has become more accessible, with programs for music-making such as Garageband coming preinstalled on every Apple Macintosh computer (Cevasco & Hong, 2011). Handheld devices like the iPad offer free apps for portable music access and making, both increasing the opportunity for the everyday user to create and share content, and the opportunity to have a “band-in-a-box” with a variety of functions being achievable with one device. The uses and accessibility of technology has expanded the possibilities for users with complex needs to participate in active music-making.
Electronic Music Technology
Electronic music technology is a wide-reaching branch of technology that has progressed over the last 30 years. Developments in hardware and software, and creation of new instruments that utilize technology, have pushed boundaries forward both in terms of the creation and production of music. While the history of the development of EMT, specifically electronic instruments, is beyond the scope of this review, overviews can be found in literature (Bongers, 2000; Challis, 2009; Paradiso, 1997) along with proceedings from dedicated conferences like new interfaces for musical expression (NIME). Comprehensive introductions to the world of NIMEs can be found (Lyons & Fels, 2015) and books such as those by Miranda and Wanderley (2006), offering a reference point for the control of sound using technology and issues surrounding the creation of new instruments (Ward, Woodbury, & Davis, 2017).
EMT for Clients With Complex Needs
EMT that increases accessibility for clients with complex needs has been defined as a range of tools and devices which are able to generate musical sounds through electronic, digital, or mechanical means (Magee, 2012). Definitions include: “any equipment, device, or method that systematically fosters independent functioning, including the production of or response to music” (Crowe & Rio, 2004, p. 283); “the activation, playing, creation, amplification, and/or transcription of music through electronic and/or digital means” (Hahna et al., 2012, p. 456); and “a wide range of devices, equipment and software, spanning amplification devices, MIDI (musical instrument digital interface) devices and instruments, computer software, assistive devices, brain computer interfaces, as well as electronic musical instruments and specialist interfaces such as switches and sensors” (Burland & Magee, 2013, p. 179). These types of technology, and their relationship to music therapy clinical practice, began being discussed in the late 1980s (Krout, 1987) and early 1990s (Krout, 1992), with the use of music technology for those with complex needs also being covered in popular music magazines (Thomas, 2012).
While the term EMT covers a wide range of technology to facilitate musical interaction within the field of music therapy (Magee & Burland, 2008), instruments created with technology are often called digital musical instruments (DMIs) in the field of new interfaces for musical expression (NIME) (Poupyrev, Lyons, Fels, & Blain, 2001). Since the 1980s there has been a rapid expansion of EMT use with the field of music therapy (Whitehead-Pleaux et al., 2011) and many DMIs have been developed both commercially and for research purposes. DMIs can be aimed at a typical population or can be bespoke. Bespoke instruments use technology or combinations of technology to allow an individual access to active music-making. These technologies can include hardware and/or software.
DMI Development and Musical Genres
The expansion of DMIs can be in part attributed to the strong links between the type of music that has historically been created using music technology—that of hip-hop/rap and later, electronic dance music (EDM) (Crooke, 2018)—and client preference—particularly of children and adolescents—of those receiving music therapy (Viega, 2015). Music therapists striving to explore genres that heavily resonate with their clients have used hip-hop and rap and its strong links to the transformational power of music to “facilitate group experience and catalyse personal and social transformation” (Lightstone, 2012, p. 41). Themes explored within hip-hop culture and EDM, which took root from the oppressed origins that forged the development of genres, can be likened to a “universal language” (Lightstone, 2012, p. 40), particularly when working with children and adolescents (Hadley & Yancy, 2012).
Types of Technology
Categories of Technology
Crowe and Rio (2004) completed a comprehensive historical literature review of technology and its implication in music therapy practice and research for music therapy education. From this, they organized the types of technology into taxonomical structures. They concluded that there are seven types of technologies: “(a) adapted musical instruments, (b) recording technology (c) electric/electronic musical instruments, (d) computer applications, (e) medical technology, (f) assistive technology for the disabled and (g) technology-based music/sound healing practices” (Crowe & Rio, 2004, p. 291). These categories are exhaustive in terms of covering all types of technology used in the music therapy environment but do not focus on those used primarily for active music-making. The categories also include technology that is used for 1) analysis and logging of data about client progress, 2) creating and hearing listening material, and 3) medical technology that involves sound waves.
The rate of change within the technological environment of electronic music has meant that there have been several developments since the creation of these categories that are difficult to place within them, and there is technology that crosses between them. Magee’s classifications (2006, 2012) reflect more up-to-date inclusions of self-contained music creating devices (such as synthesizers), music listening devices (such as mp3 players like the iPod), digital handheld music devices or DHHMDs (Nagler, 2011) (such as the iTouch app and iPads), and music games (such as Guitar Hero).
Krout (2015) subsequently provided four categories of electronic music resources based upon those that have been reported as being useful in music therapy clinical practice, and are also affordable and available. These included general or stand-alone products, computer software, electronic keyboards, and tablet computers (e.g., iPads).
Digital Handheld Music Devices
The category of DHHMDs have become ubiquitous aids for aiding in music-making. DHHMDs have become part of everyday life in an unanticipated convergence of technologies that has altered the practice of music therapy in a profound manner (Nagler, 2011). DHHMDs offer a new class of music listening experiences, predictive selections, and active music-making without need for therapeutic interventions. These devices have become multitasking musical companions allowing complex musical ideas to be created and shared without technical training (Nagler, 2011). New technologies such as tablets featuring touch screens, particularly the iPad, have created a shift toward screen-based mobile music-making. The touch screen allows direct interaction to music apps using intuitive motion (Krout, 2015). Comprehensive reviews of iPad resources are available to help clinical practice (Knight, 2013). With each of the four methods of music therapy (recreating, improvising, listening, and composing) being able to be accentuated by apps (Knight, 2013).
iPads and Apps
iPads have become prolific in school settings, offering multifunctionality, the ability to tailor to individual styles of use, ease of use, portability, and high quality of graphics and sound (Krout, 2015). iPads have been used to create powerful and expressive controllers for digital music (Favilla & Pedell, 2014) with many music-based applications developed to meet different needs. Some apps tie into existing software to provide a new facet of access while others offer experiences unique to the device. Krout (2014b) provides an exploration of a number of apps for engaging young people with Autism Spectrum Disorders, the needs they address, and their efficacy in music therapy. He suggests that the therapist must balance the advantages and disadvantages of using such technology against each client’s needs, abilities, and goals. Apps such as Beatsurfing (Lobby & De Ridder, 2018) allow the creation of custom graphical user interfaces (GUIs). These can be designed by the user through building with lines, polygons, circles, and faders. Parameters such as size, color, orientation, 3D position, and value that can also be customized (Lobby & De Ridder, 2018). These GUIs can then be connected to MIDI-compatible software, hardware, or other MIDI-enabled apps to provide bespoke interaction and allow configurable sonic output. One such app is ThumbJam (Sonosaurus, 2018) which provides a vast array of features. Included in the app are over 40 sampled instruments, hundreds of scales, and an array of customization of how it can be played, and what is displayed on screen (including user uploaded backgrounds) (Matthews, 2018). ThumbJam also offers arpeggiating, looping, recording, effects selection and manipulation, instrument creation, and the ability to import and export data. The ever-growing app market means it is easier than ever to find screen-based applications that fit the needs of the user and also offers access to the developers in terms of suggesting updates and tailoring for specific needs.
The Computer as the Bridge
Traditional acoustic instruments are “stand-alone” in the fact that they are composed of an excitation mechanism (string, reed, skin, etc.), a resonant capacity (the body of the instrument), and the specific timbre they produce. If, however, we add a computer as a bridge in this system, we arrive at DMIs. A DMI “implies a musical instrument with a sound generator that is separable (but not necessarily separate) from its control interface” (Malloch, Birnbaum, Sinyor, & Wanderley, 2006, p. 49). DMIs break the coupling between the action used and the sound produced. This can be thought of as a three-layer system (Figure 1) consisting of the control interface, the processing (which can be achieved via a separate computer or an on-board system), and the effort mechanism or output (audio/visual/haptic feedback) (Hunt, Kirk, & Neighbour, 2004).
Useful methods of classification can be adopted from the fields of HCI, music technology, and new interfaces for musical expression (NIME), when categorizing new technology which uses the computer as the bridge. Wanderley (2001) suggested the term gestural controller to describe interfaces that consist of two elements. The first element is an interface that features one or more sensors to detect the physical interaction of the performer (these can be in the form of body movement, empty-handed gestures, or object manipulation). The second element is the auditory, tactile-kinesthetic, and/or visual feedback given to indicate the instrument’s status the performer.
Wanderley (2001) proposed a three-tier classification of such controllers as:
Instrument-like controllers—where the input device design tends to reproduce each feature of an existing (acoustic) instrument in detail (e.g., an electric keyboard)
Augmented Instruments (also called Hybrid Controllers)—instruments augmented by addition of sensors (e.g., the Yamaha Disklavier)
Alternate controllers—whose design does not follow one of an established instrument—an example would be the Hands (Waiswisz, 1985; Wanderley, 2001, p. 6)
Alternate controllers offer unique opportunities to create interactive musical systems from the ground up to specifically suit client need. Using new or bespoke modes of interaction and processing these interactions into meaningful content provide unique potential to increase accessibility to active music-making. Alternate controllers can be designed with client capabilities at the center of the design process, can be built to assist both in terms of physical access and learning needs, and can be tailored to provide feedback to suit the client or context they are being used in.
Alternate Controllers
Alternative controllers take two forms: 1) those that require physical touch to control, which are referred to as touch-based, and 2) those that do not, which are referred to as empty-handed.
Touch-Based Alternate Controllers
Touch-based controllers use direct physical interaction with a control interface to acquire control data for musical systems. Notable developments in this area are discussed below.
MidiGrid
One of the first examples of using a touch-based alternate controller was explored by Hunt and Kirk (2003). In their long-term project (beginning in 1987) titled MidiGrid, they utilized the mouse and keyboard to control sound in software used by children and young people in a music therapy setting. Hunt and Kirk (2003) used the advent of MIDI (musical instrumental digital interface) within their project. MIDI is a communication protocol that was developed in 1983 to allow various pieces of technology to use a connective language. The MidiGrid project was furthered by the development of MidiCreator (Kirk, Abbotson, Abbotson, Hunt, & Cleaton, 1994), which converted signals from electronic sensors into MIDI. MidiCreator could then be connected to the MidiGrid software. A computer could be equipped with MidiGrid allowing users to explore the creation and composition of musical work without the need to learn a traditional instrument. MidiGrid has been used by a wide range of people, such as composers, schoolchildren, special needs teachers, and their clients (Hunt & Kirk, 2003).
Skoog 2
A more recent development is the Skoog 2 (Skoogmusic, 2018), a wireless Bluetooth-enabled tactile foam cube with companion app and software. Manipulation of the Skoog surface can be mapped to proprietary sounds within the software or can connect to external MIDI-compatible software. The system provides a wealth of resources “out-of-the-box,” allowing for user customizable sounds and notes as well as controllable sensitivity settings for note triggering. This provides a hands-on musical experience for those with no previous musical knowledge affording individual exploration in a solo setting (Nath & Young, 2015).
Music production controllers
Music production controllers (MPCs) are generic devices developed for electronic music-makers that feature triggering pads often used with MIDI-compatible software. They provide another modality of interaction that can be used as a tool to increase accessibility; however, these devices require a person familiar with music technology to set them up. The configurability of these devices allows adjustment to fit specific client requirements; additionally, functionality allows user profiles to be stored and recalled as needed. In a setting where resources have to be shared, this is an important feature as it provides the flexibility to allow users with different abilities to dictate the media content being triggered by the pads. This also allows for different levels of support (from simple note triggering to timing support) depending once again on client needs and preferences. This type of music technology is often very attractive to children and young people providing a motivator for engagement.
Switches
Another touch-based alternate controller used extensively, particularly for clients with severe disabilities, is the switch (Bache, Derwent, & Magee, 2014; Crowe & Rio, 2004). Switches are electronic or mechanical devices which, via a control unit or cordless receiver, provide a simple mechanism for choosing and communicating (Magee, 2012). Switches use physical action or gesture to give direct access to a variety of electronic music devices. There are a large range of switches that offer many forms of control. Bache and colleagues (2014) provide a comprehensive overview of switches and their use with those with complex needs. Switches are a commonplace assistive technology that can be used in combination with specialist or commercial software. Custom built switches based on motor, cognitive, or sensory needs facilitate interaction based on clinical need. Sounds triggered by a switch can give a sense of control to clients, reinforcing a sense of self and allowing for expression (Swingler, 1998). Communication by using switches is often a starting point for nonspeaking clients (Hunt et al., 2004; Magee et al., 2011)
Mogees
An alternate controller providing an out-of-the-box package is the Mogees (Mogees, 2018). Mogees is a contact microphone which when placed on any surface detects when the surface is “played.” Mogees has the potential to be used in a variety of settings and with objects that users are familiar with or motivated to interact with. It is highly portable and affordable.
Musii
Finally, another interesting alternate controller is the multisensory interactive inflatable Musii (Musii, 2017). Musii is a soft inflatable object that emits sound and illuminates with color when touched. It enables any nonmusician to experience the act of creating music by translating physical interaction with the device into stimulating audio, visual, and tactile sensation (Musii, 2017).
Empty-Handed Controllers
Empty-handed controllers do not require physical touch and use mechanisms such as infrared light, ultrasonic sensors, electromagnetic fields, radar, cameras, or microphones to detect sound or physical movement. Sonic parameters can be mapped and controlled from this information. This can be particularly useful in facilitating clients with complex needs by providing high levels of control, especially for those with physical disabilities or impairments.
The earliest empty-handed controller is considered to be the Theremin, patented in 1928 by Leon Theremin, in which the player uses the proximity of their hands to two metal aerials to control frequency and amplitude of a sound. The earliest documented use of alternate controllers for music-making in music therapy can be traced back to 1987. Nagler and Lee (1987) used microcomputers in music therapy sessions to “investigate the possibility of enabling a severely physically handicapped person to create music with minimal assistance” (p. 72). Using an Apple II microcomputer, Mountain Computer Music System, Express 3 infrared tracking device, and the Viewpoint optical indicator (an infrared light beam) clients could control the music based on their head movements, allowing them to achieve independent music-making.
Soundbeam
One of the first empty-handed commercially available alternate controllers for music therapy was the Soundbeam system (Soundbeam, 2018). Soundbeam is a tool that converts movement within an ultrasonic beam into MIDI information. Although it can be found in the equipment stores of many special educational needs schools in the United Kingdom, it has been described as poorly used (Magee, 2012). Factors that contribute to this may be that due to its complexity, specialist training is required to use the device and there is an inherent difficulty in placing the beams optimally to suit the movement of some users. The beams travel out linearly which can be unsuitable for users who cannot follow that axis of movement (Ellis & Van Leeuwen, 2000). The lack of tactile feedback can also mean a disconnect between cause and effect for some users. Despite these drawbacks, Soundbeam has been extensively used in practice, possibly due to the unique mode of interaction it affords and the fact that there is a wealth of material and resources to enable people to use the system (Soundbeam, 2018).
Music Maker
Other motion capture systems use cameras to capture movement data. A notable development is Music Maker, which turns body movements into sound using a nonobtrusive camera. Music Maker uses displays of cartoon drawings or pictures of musical instruments to give an element of fun and can be adjusted according to patients’ levels of support needed, therapeutic goals, and type of equipment available in hospitals or patients’ homes (Gorman, Lahav, Saltzman, & Betke, 2007).
Eye gaze systems
Additionally, some control mechanisms include eye gaze systems. These detect the user’s direction of gaze as control information, often utilizing a “dwell” type eye event to elicit a mouse click. Eye gaze systems are often the only access method available to those with diagnosis of “locked-in syndrome” (Vamvakousis & Ramirez, 2016), they are used due to the efficient and less effortful way they can be used to provide access to the computer (Bache et al., 2014). Hardware and software developments by commercial companies such as Tobii, Sensory Guru, and Smartbox (Bache et al., 2014) have pushed forward the development of the musical applications of eye gaze. One such example is EyeMusic, which provides a “system that transforms eye movement data into musical compositions and data sonifications” (Hornof & Sato, 2004, p. 185). However, use of such systems do require skills developed over time by the client.
Clarion
A notable recent development in this area is the Clarion. The Clarion is a highly configurable software instrument developed as part of the Open Orchestras project (Open Orchestras, 2018). The Clarion allows the client to specify “the sound the instrument makes; the number of notes that are available to play; the shape, position and colour of the notes; and crucially the way in which [you] play them” (Farrimond, 2016). It integrates with eye gaze systems, SmartNav and the iPad, allowing use with existing hardware resources. Clarion comes as part of a package offered by Open Orchestras which includes the Clarion software, repertoire, training resources and support, and an evaluation framework.
New Developments
A number of recent related developments have impacted the world of DMIs. Microcontroller boards like Arduino (Arduino, 2015), affordable computers such as the Raspberry Pi (Raspberry Pi, 2015), and software such as Max/MSP (Cycling’74, 2018) allow for bespoke systems to be created. Sensors can directly capture a person’s input and then be integrated as a control device for software, or stand-alone bespoke devices can be created at a low cost. The development of the Internet of Things (Internet of Things Council, 2018), and Web portals and Webpages with tutorials such as Instructables (Autodesk, 2018) have provided a community of DIY developments and assistance (in the form of forums) for those wishing to create bespoke instruments. Hacker communities are also providing space and tools, along with “hackathon” style competitions (often 24-hr themed competitions which are supplied and sponsored by companies), allowing for rapid prototyping of accessible instruments and new tools while also bringing together people with a range of skill sets to create and share information online. There are now also many intermediary applications that allow for the quick creation of enticing interfaces to trigger music and sound.
Makey Makey
Packages such as the Makey Makey (Makey Makey, 2018) allow conductive objects (e.g., fruit, putty, metal) to be connected to a microcomputer to emulate keyboard presses, which can then be used to trigger sound. For example, users could create a piano from bananas by using Makey Makey and connecting it to software such as Garageband or SoundPlant (Blum, 2018). Both these softwares allow sounds to be assigned to keyboard presses.
Bare Conductive
Bare Conductive Touch Board (Bare Conductive, 2018) is another microcomputer featuring 12 touchpads that allow conductive materials to be connected via crocodile clips. The out-of-the-box setup allowed 12 sound samples to be triggered monophonically from a memory card placed in a slot embedded on the board which then play via an on-board headphone jack, or connected to a speaker. The board is well documented and designed to be used with minimal technical knowledge. The board also offers expansion for those with more technical knowledge as it contains a built-in general MIDI chip for those wishing to reprogram the board to allow polyphonic notes, or to allow creation of bespoke MIDI-enabled instruments. The Bare Conductive website (https://www.bareconductive.com/) features very comprehensive step-by-step guides for setting the board up and provides ideas for utilizing the board practically. These new developments expand possible modes of interaction by providing off-the-shelf software and hardware that may be commonplace in music therapy settings, or simply using everyday items that the client may find enticing and motivating to engage with.
Leap Motion
Other new technologies such as the hand gesture tracker Leap Motion (Leap Motion, 2017) offer toolkits to build custom systems. The Leap Motion system converts hand movements to data, thus providing a flexible tool for mapping client-specific movements to sound (Uwyn.com, 2018).
Microsoft Kinect
The Kinect (KinectSEN, 2018) is a camera-based movement tracker made by Microsoft that allows body movement by skeletal tracking to be used to control data, thereby producing sound through movement.
GestureSEN
An excellent resource for gesture-based systems used in special education is the gestureSEN website (https://web.archive.org/web/20180723042755/https://kinectsen.wikispaces.com/home). The site, run by teachers in special schools, aims to explore how established and emerging gesture-based technology could help people with severe learning difficulties with their engagement, creativity, and independence skills (Gesturesen.wikispaces.com, 2018). The site featured information on using eye gaze, Kinect, Leap Motion, iPad, and Virtual Reality in special education settings. Unfortunately, due to the closure of Wikispaces website, the content from the gestureSEN website is only viewable through internet archive websites such as https://web.archive.org/.
Games Controllers
Finally, game controllers such as the WiiMote and the Xbox controller alongside music-themed games can also provide unique mechanisms through which to access musical interaction, with schools typically having these resources available for general use.
All of the above offer new methods of access to music-making with the computer that move away from the keyboard and mouse paradigm. These tools provide the flexibility to create systems that tailor to client capability, motivation, and curiosity.
Available Technology
As evidenced by the literature presented, there are many technologies available for aiding accessibility to music-making. As a growing field that crosses many disciplinary areas, challenges are created for music therapists. The primary challenges are: knowing where to find this technology, examples of its use in similar contexts, and guidelines for integrating it into clinical practice.
Table 1 provides a summary of developments, including off-the-shelf DMIs, that have been used with clients who have complex needs. The DMIs included in the table were selected because there is evidence that they been used with people with complex needs, through either peer-reviewed published literature, anecdotally, or observed firsthand by the first author. The DMIs reviewed are further organized into two categories: 1) commercially/freely available, and 2) research only. This decision was based on the fact that while some of the research and technology developed may show great promise for clients with complex needs, they have not subsequently been made available for wider use. The two categories are then further divided into three subcategories: touch-based, software-based, and empty-handed. A final section provides some context to how the technology may be relevant to music-making in a music therapy setting.
Interaction modality . | . | Commercially/ freely available . | Description . | Details . |
---|---|---|---|---|
Touch . | . | General technology . | . | . |
Switches | Broad range of electronic or mechanical tools to allow on/off selections | Tailorable to user, ease of integration to current resources, can be wired or wireless, trigger or start/stop sound effects, musical notes/phrases, recorded samples, or sequences of samples. “For example, the client may activate a CD player or pre-recorded music track” (Knight & Lagasse, 2012, p. 192). Bache et al. (2014) provide an in-depth review of practical switch use. | ||
iPad | Touch screen handheld device speaker, microphone, and motion detection | Multiuse device, tailorable to user, familiar technology, enticing, direct interaction with apps, self-contained with speaker on-board, commonly available. Useful for quickly recording multitrack music arrangements from live or plugged-in sources. Ability to capture sound using built-in microphone. Apps can emulate instruments enabling the client to play a stringed instrument by touching the screen, or a wind instrument by blowing into the microphone (Knight & Lagasse, 2012). “Apps can be used to record, synthesize, manipulate, or provide feedback to client actions and sounds” (p. 194). Knight (2013) provides an in-depth review of iPad applications in music therapy. | ||
iPod Touch | As iPad | Similar to iPad but smaller in form factor and with enhanced security (not connected to a network) | ||
Apps | Pieces of software for use on tablet or smartphone devices | Wide variety, some free. Apps for Children with Special Needs (2018) is a website for finding specific apps aimed at children with special needs featuring music as well as other apps with reviews and videos. Offering the ability to tailor content and interaction to client requirements. | ||
Generic music production controllers (MPCs) | Trigger pads with velocity sensing | MIDI-compatible. Some come with own software instruments, requires technical knowledge to set up. Provides ability to preselect sounds representative of a wide variety of genres enables clients an accessible way to perform their cultural or musical identities. Allows therapists to offer diverse sound palates valuable for playing differing feelings or emotions during improvisation (Crooke & McFerran, 2019). Can also be used in composition to perform and record drum beats and melodic patterns (Crooke, 2018), or to trigger or launch a range of loops or samples, allowing for the playing of precomposed pieces and live remixing. Music Radar (2018) provide an overview of available MPCs. | ||
Music-based video games | Video game system using generic or bespoke to the game controllers | Familiar to users, enticing technology, easy setup, can potentially use existing resources. Blaine (2005) provided a review of alternate music-based video game controllers. Wikipedia (2017) provides a list of music-based video games. Use of the WiiMote in music therapy has been explored (Benveniste, Jouvelot, & Michel, 2008), and developments such as the Wiinstrument can be downloaded to utilize the myriad of data that the WiiMote produces (The Wiinstrument, 2018). Games available include Wii Music by Nintendo. | ||
Specific devices | Description | Details | ||
Skoog 2 | Wireless foam cube with five tactile pressure-sensitive pads. | Wireless, portable, easy to set up. Simple and intuitive. Own app and software to customize sensitivity and sound created, MIDI-compatible. Many resources for use in special education on website. Provides direct correlation between physical contact and sound produced, using virtual musical instruments, samples, or MIDI. Offering dynamic control over musical gestures (Skoogmusic, 2018). “The instrument does not simply trigger samples when pressed but uses sophisticated synthesis to dynamically manipulate the various instrument sounds though pressing, squeezing, rubbing, stroking, tilting or manipulating the Skoog” (Farrimond et al., 2011, p. 28). | ||
Makey Makey | Microcontroller board with six connectors that emulate QWERTY keyboard presses and mouse control | Turn everyday objects into touchpads, no software to install, fast setup. Can be connected to conductive objects such as fruit, conductive tape, pencil graphite, and clients touching each other in chains as a means to trigger sounds (Makey Makey, 2018). The process allows appropriation enabling clients to gain authorship of their instruments, and become an intrinsic part of their deployment (Hayes, 2016). | ||
AlphaSphere | Globe-shaped MIDI controller with 48 playable velocity-sensitive pads and own software. | AlphaSphere is a tangible controller, when connected to AlphaLive software/used with other MIDI software can be set up to trigger and manipulate sound and provide a unique modality of access that can be enticing to clients (Place, Lacey, & Mitchell, 2014). | ||
MIDICreator | Device to convert signals from electronic sensors into MIDI data. | Clients can control sounds with physical actions and gestures, can be used to detect simple body movements (Krout, 2014a). Can be connected to MIDI to be used with other synthesizers. Sensors available include pressure, distance, proximity, direction, etc. (Meckin & Bryan-Kinns, 2013). | ||
I-CubeX | Software and digitizer for creating systems with a variety of sensors available | Tools to capture the human actions and/or environmental variables and make these signals available to other equipment such as a computer or a musical instrument to trigger sound, music, video, graphics, animation, robotic movement, etc. MIDI data are used, transmitted via MIDI cable, USB cable, or Bluetooth wireless (I-CubeX, 2018). | ||
Kyub | Eleven feather touch keypads on five surfaces of a 3-inch wooden cube with accelerometer | Maker friendly, open-source DIY MIDI keyboard featuring capacitive sensing and accelerometer. Fully programmable, set scale, tweak note velocity curves, map different instruments to different pads to configure to musical taste (Kyub MIDI keyboard, 2018). User can also configure the way the instrument looks and sounds by designing their own interface. | ||
Suzuki QChord | Electronic instrument with on-board speaker and LCD display. | A device for composition, teaching, and therapy. Incorporating technology from a basic keyboard and electric guitar and combining both in a portable, easy-to-use way. Eighty-four different chord combinations, 100 instrument voices, orchestrated rhythms. Features three sections; a touch-sensitive 4 octave “strum plate,” a rhythm section, and chord button section. Each of these areas can be used independently or combined with a variety of sounds obtainable. Sounds are always in tune. Can be adapted to all ability levels. Can stimulate interaction, increase coordination, stimulate gross and fine motor skills, and increase self-expression. Pitch bend wheel for expression. Changeable song cartridges. Can connect to speaker or MIDI device (Suzuki Qchord, 2018). | ||
Yamaha DJX-iiB | Groove machine in a box-shaped desktop module form with scratch pad and fader. | Can select from 70 patterns, mute parts, add preset hits or fills, and shift key of the patterns playing and use effects. Provides opportunities for the clients to compose music by assembling loops, importing other songs or sections of songs, and/or recording her or his own music, offering accessible and dynamic means of expressing emotions (Whitehead-Pleaux et al., 2011) | ||
Musii | Soft inflatable self-contained portable unit that emits sound and illuminates when three inflated cones are touched, with wireless control | Simple to use, needs no extra equipment or training to operate. Abilities to make music and colour through touch and movement with expansive library of soundscapes (over 50) and innovative musical system. Cannot play out of time or out of tune. A number of people can play harmoniously as a group or an individual can become an orchestra. Tactile physical form that enables the user to see and feel the beam that they are interacting with. “Musii has been designed for developmental play in the SEN sector...The multi-sensory experience of playing Musii has many therapeutic possibilities including encouraging movement, development and awareness of proprioception, turn taking, cause and effect, creativity, expression and communication. It can be used for stimulation or for calming. The synchronised sound and lighting as well as the visual and tactile feedback of the inflatable enable a deeper understanding of the music you are making” (Musii Ltd, 2017) | ||
Reactable | Table with touch screen and moveable objects (Reactable Technology, 2017) | Objects interact with the table surface and each other to make music. Allows intuitive and collective creation of complex musical pieces, collaborative sharing space between users, promotes imitation games, increases visibility of actions, enables monitoring other participant’s work, aids in reducing solitary play sequences, facilitates associative play (Villafuerte, Markova, & Jorda, 2012) | ||
Tenori-On | Handheld screen with 16 × 16 grid of LED switches. Built-in speakers, dial and buttons control sound and beats per minute produced | Create, control, or perform musical material on visually rich touch-sensitive interface (Farrimond et al., 2011). Switches activated in different ways create music from 256 sounds. Engaging, motivating, sensory, well suited for improvisation, easy to use/hard to master. Combines visual and melodic sensory information, can stimulate cognition, memory, and perception. Can function as a rhythm machine with basic or complex rhythmic beats that can be looped or changed. Notes and melodic phrases can be added as well, creating up to 16 layers of sound (Clements-Cortes, 2014). Clements-Cortes (2014) provided an in-depth study of the Tenori-On in a clinical music therapy setting. | ||
Roland Handsonic | Device with 13 ultra-sensitive touch pads | Eight hundred and fifty ready-to-play sounds, or import custom sounds. Responsive and therefore not overly demanding to play, with easy adjustment of volume (Challis & Smith, 2012). Can be calibrated to client need in terms of sensitivity. Can connect to other MIDI devices and MIDI can be recorded from the device for analysis. | ||
Korg Kaossilator/Kaossilator Pro/Korg Mini Kaoss Pad | Audio effects unit with an X/Y touch screen. Mini Kaoss pad features on-board speaker and microphone | Positioning a finger-tip on the touch screen triggers specific sound programs. Ability to trigger individual notes or patterns of notes depending on the nature of the chosen sound and settings selected. Moving around the screen moves between notes within a predefined scale or changes the nature of the sound. Allows the results of actions to be sampled as repeating loops. Particularly effective for those with restricted hand and finger movement, being easy to interact with, to produce complex musical ideas and patterns (Challis & Smith, 2012). | ||
SoundHouse Special Access Kit (Banana Keyboard)—discontinued | Sixteen keys configured like an oversized piano with software component | Curved to suit the radial movement of an arm. Light touch activates music, sounds, or speech programmed into each key. Allows connection of up to eight switches for activation of keys on the keyboard (Sound House, 2017). Innovative, adaptable control surface that require musicians to press, squeeze, or strike them to create and control music through corresponding musical software (Farrimond et al., 2011). User friendly, easy to learn. Arrange combinations of sounds (MIDI sounds or wave files). Recording feature enables real-life performances to be saved and voice and CD segments to be recorded. Arrangements can be printed off and formatted. Variety of global settings cutting down on individually programming each key or switch. Aimed at fostering development of switch use, cause-and-effect, timing, choosing (Sound House, 2017). | ||
Numark Orbit | Wireless MIDI controller | Sixteen backlit customizable pads, four selectable banks to assign up to 64 cues, lighting transitions, video clips, samples, etc. Control wheel and on-board accelerometer. Can be configured and mapped to control other MIDI software, accelerometer for motion control. Comes with demo software that show how to use with tracks and effects (Numark.com, 2018). | ||
Mogees | Resonance contact microphone | Enables instruments to be created from any surface/object alongside configuring of the sounds created with dedicated iPhone/iPad app. Integrates with standard digital audio workstations via Audio Units or VST plug-in. Can be used to provide expressive instrument by using different areas of surface trigger different sound (Mogees, 2018), allowing for objects to be used that are motivating, familiar, or engaging to clients. | ||
Software- based | General technology | Description | Details | |
Generic digital audio workstations (Ableton Live/Audacity/Reaper/Cubase/Logic/Sonare by Cakewalk/Garageband) | Music recording/composition software | Uses preset or user determined settings via on-screen, or pull-down menus. Allow for recording, composing, playback, and creation of music. Some come with content such as instruments and samples available for instant use. Ability to use software instruments (VSTs), input microphones, or electronic instruments. Garageband is preinstalled on Mac computers with out-of-the-box samples and instruments available. Reaper is affordably priced, Ableton is prized for live performance. Audacity is freeware. Can export notation in some cases. Krout (2014a) provides a review on using software for music composition, arranging, notating, improvising, and sequencing. | ||
Specific devices | Description | Details | ||
Clarion | Software instrument | Allows user to change every element instrument including sound, notes, shape, position and colour of notes, and how those notes are played. Integrates with eye gaze systems, SmartNav and iPad. Package offered by Open Orchestras including repertoire, training resources, support, and an evaluation framework (Open Orchestras, 2018). | ||
Magix Music Maker | Digital audio workstation | Provides 425 sounds & loops, seven free Soundpools (1,927 sounds & loops) three software instruments, eight tracks, and eight effects. Can be used with smart boards. | ||
HyperScore | A graphical composition environment | Users draw strokes and lines to explore musical ideas. Graphical elements are mapped to musical structures, allowing users to shape musical progressions visually (Grierson & Kiefer, 2013; Machover, 2004). | ||
MIDIGrid | Music software | Uses mouse and keyboard movements within a grid to trigger notes, chords, sequences, or patterns of sound that can be played back and looped (Hunt & Kirk, 2003). | ||
Microsoft Songsmith | Music software | Generates musical accompaniment to match a singer’s voice using computer microphone input. Musical style and feel of song can be selected. Songs can be posted songs online, or used to create music videos | ||
2Simple music toolkit | Software applications | Six programs that introduce key musical concepts in an interactive way (2Simple, 2017). | ||
Empty-handed | Camera-based | Microsoft Kinect | RGB camera, depth sensor, and multi-array microphone | Provides full body 3D motion capture, voice and face recognition (KinectSEN, 2018). |
EyesWeb | Open development software platform | Real-time multimodal system and interface that has been used extensively in research. System supports input devices including motion capture systems, video cameras, game interfaces (e.g., Kinect, Wii), audio input, analog inputs (e.g., for physiological signals). Outputs include multichannel audio, video, analog devices, robotic platforms (Camurri et al., 2000). Website features information on development (http://www.infomus.org/eyesweb_ita.php) | ||
MotionComposer (available 2019) | System using two types of camera to detect movement | Allows gestures to be used to explore sound environments (Bergsland & Wechsler, 2016). | ||
AUMI | Free software application | Interface that enables the user to play sounds and musical phrases through movement and gestures captured via webcam (Larsen, Overholt, & Moeslund, 2016; Oliveros, Miller, Heyen, Siddall, & Hazard, 2011). | ||
VMI (Virtual Musical Instrument) | Free software | Uses web camera to detect motion. User virtually “touch” shapes on screen to trigger sounds. Requires no special equipment, Windows only based computer. Designed for use by therapists and educators, it is customizable according to the preferences and needs of the user, and can be used for specific therapy or educational goals (Virtual Musical Instrument, 2018). | ||
BigEye—discontinued still available for download | Macintosh only software program | Uses video information to convert into MIDI messages. Allows tracking of objects through space converting their parameters into MIDI in real-time (Legacy Products, 2018). | ||
Camera Theremin | Free test application | Create sound from movement using webcam (Camera Theremin, 2018). | ||
Musical Gestures Toolbox | Toolkit for experienced Max programmers. | Collection of modules and abstractions for the graphical programming environment Max 5 to enable extraction of movement data from video (Jensenius, Godoy, & Wanderley, 2005). | ||
Aerodrum | Package featuring drumsticks, software, feet markers, and camera | An air-drumming instrument. Runs on computer using a high-speed camera to track movements to trigger drums (Knight & Krout 2017). | ||
Break-beam | Beamz | Device featuring four breakable laser beams | Can be purchased as a professional package featuring software, songs, structured activities, therapy guides, and lesson plans | |
Soundbeam | Device featuring ultrasonic beams and switches (both wired or wireless) and a synthesizer unit | Device which uses sensor technology (up to four ultrasonic or eight switches) to translate body movement into music and sound using MIDI. New unit features touch screen interface, extensive library of sounds, recording and composing functions, training programs also available. Extensive support for use available through online resources (https://www.soundbeam.co.uk/). | ||
Theremin (Magee, 2006) | Moog Theremini—device featuring 32 wavetable preset sounds, and on-board speaker and sound engine | Can be used at any skill level, providing new ways to experiment with music and gestural control. Assistive pitch quantization allows each player to adjust the instrument’s level of playing difficulty. “At the maximum position, the Theremini will play every note in a selected scale perfectly, making it impossible to play a wrong note. As control is decreased, more expressive control of pitch and vibrato becomes possible. When set to minimum, the Theremini will perform as a traditional Theremin” (Theremini, 2018). Built-in tuner with real-time visual feedback of played notes and proximity (useful for correcting playing position). Store selected scale & root note, set and recall a specified playing range, and specify patch settings. Silent rehearsal available via headphone jack. Two line level audio outputs, a pitch CV output with selectable range, and a mini USB jack for MIDI I/O and connectivity (Theremini, 2018). | ||
Optimusic/OptiBeams (Knight and Krout 2017) | Interactive light beams | Package with interactive light units (the beams), laptop with OptiMusic software, USB controller box, user button box, reflective pads/bats. Interact with colorful beams of light (2, 4, 6, 8, or 12 beams), pass hand or body through the beams or use reflective wand to trigger audiovisual events in real-time. Comes with over 80 interactive settings. Package also comes with training (on-site or e-training) (OptiBeam, 2018). | ||
MidiGesture | Ultrasonic beam sensor | Sensor that plugs into the MIDICreator system (see MIDICreator). | ||
Leap Motion (Leap Motion, 2017) | Small device to track hand movement | Uses two monochromatic IR cameras and three infrared LEDs to track hand and finger movement above device. Dickens, Greenhalgh, and Koleva (2017) provide an in-depth description of research conducted using the Leap Motion for music performance with users with complex disabilities. | ||
Brain computer interface (BCI) | Brainfingers (Brainfingers, 2018) | Headband fitted with sensors | “Detects electrical signals from facial muscles, eye movement and brain waves. Brainfingers does not directly target music creation, as it can solve many tasks such as simple clicking, to complex combinations of controls. It is software that converts all the sensor input data into controls termed Brainfingers. This software is useful for a broad range of users, especially people with severe disabilities” (Larsen et al., 2016, p. 329). Controls most AAC software, educational software, and video games. | |
Eye gaze | EyeMusic (Larsen et al., 2016)—legacy files available online, requires technical skill to install (EyeMusic, 2018) | Software utilizing generic eye gaze equipment | System that uses eye movements as input to electronic music compositions. Can be used with established composition software allowing prerecorded eye movement data to control musical compositions (Hornof & Sato, 2004). | |
E-Scape (Anderson, 2018) | Software utilizing generic eye gaze equipment (or switches) | Music software specifically designed to be used by people with disabilities to create or perform music. System operates via large guided pop-up menus controllable by one or more switches, mouse, keyboard, eye gaze, or MIDI controllers or sensors. “At every stage, E-Scape asks the user what they want to do and offers a range of options depending upon which level of complexity the user has chosen to work at” (Farrimond et al., 2011, p. 23). Two modes of operation—composition and performance. Can output MIDI data (Farrimond et al., 2011). | ||
EyeHarp | Free software utilizing generic eye gaze equipment | Gaze-controlled or head-controlled music interface to help users learn and play music. Vamvakousis and Ramirez (2016) provide a comprehensive article on the development of the EyeHarp. The website theeyeharp.org also provides a wealth of information about the project (The EyeHarp, 2018). | ||
Eye Play Music | Free software utilizing generic eye gaze equipment | Trigger notes from a range of instruments available with adjustable note length and transposition. Create own scales. Load and save settings. Website features resource for use (MBMM, 2017). | ||
Breath | Jamboxx | MIDI controller device | Hands-free electronic, breath-powered instrument. Uses sip and puff to determine amplitude of note. Software included to configure device and on-board modulation wheel, button, and jack socket to allow switch connectivity (Jamboxx, 2018). | |
Yamaha WX5, WX11 | MIDI controller devices | Breath powered MIDI controller that allows for one-handed playing. WX5 features MIDI output; however, WX11 requires an additional MIDI connection box (MBMM, 2017). | ||
Magic Flute | Stand-alone instrument | Self-contained instrument with built-in tone generating hardware. Plugs into external speakers. Two separate parts, the flute and control module with display. The flute being the remote control for the control module. Musicians can select different sounds or access the user settings without the help of another person (MBMM, 2017). The volume is controlled by blowing in a mouthpiece and the pitch by moving the mouthpiece up/down with the mouth (Vamvakousis & Ramirez, 2016). “The instrument reduces the physical and cognitive challenges inherent within conventional wind instruments. One musician, with very limited lung volume, is nonetheless able to realize the full dynamic range of the instrument” (Farrimond et al., 2011, p. 29). | ||
Interaction modality | Research only | Description | Details | |
Touch | General research | |||
Collaborative interfaces review of literature (Blaine and Fels, 2003) | Review paper of interfaces used for collaborative music-making | Comprehensive review of context and design of a number musical experiences for novices | ||
Specific research | ||||
Musicking Tangibles—RHYME Project | Tangible interfaces consisting of interactive digital “furniture” | Examining the development and benefits of using interactive digital music furniture for disabled children by using two co-creative tangible instruments. ORFI-26 soft pyramid-shaped, pillow-like modules, in three different sizes (30–90 cm) featuring bend sensors and lights, the units can communicate wirelessly with each other. Wave Carpet-7-branched, wired, interactive, soft, dark carpet with orange velvet tips that glow. Central arm contains microphone, two arms contain accelerometers that change the recorded sound. Two arms contain bend sensors that create rhythmical background music. One arm contains a web camera. Contain five software programs, offering different music and dynamic graphics to show via projector embedded in one arm, or via full wall projection. Center contains two speakers and strong vibrator in. Contains IR-sensors allowing interaction with RGB LED lights (Andersson & Cappelen, 2013). | ||
NoiseBear (Grierson and Kiefer, 2013) | Malleable controller | Development of robust, wireless, malleable controller for children with cognitive or physical disabilities | ||
Bean | Gesturally controlled digital instrument | Device designed around a Wii nunchuck controller for use in a music therapy setting (Kirwan, Overholt, & Erkut, 2015) | ||
MAWii | Digital musical instrument using generic WiiMote controller | Research exploring WiiMotes as virtual instruments for children with behavioral disorders (Benveniste et al., 2008). | ||
WamBam | Self-contained electronic hand drum | Created using piezo sensors. Paper describes development and testing of device used in for music therapy sessions with severely intellectually disabled clients (Jense & Leeuw, 2015). | ||
TouchTone | Digital musical instrument | Device featuring touch-sensitive pads designed to develop musical ability, bimanual coordination and increase social participation of children with hemiplegia (Bhat, 2010). | ||
Computer-Assisted Music Therapy | Augmented reality software | Details system developed with Augmented Reality techniques allowing music composition and creation activities using sound and colour, via cards (Correa, Ficheman, do Nascimento, & de Deus Lopes, 2009). | ||
SenseEgg | Wireless controller device | Development of a handheld egg-shaped device featuring seven on-board sensors (button, slider, accelerometer, wind sensor, ultrasonic distance sensor) and a suite of software patches aimed at for musical exploration and teaching. Featured a component that allowed control of settings via an iPad (Blatherwick & Cobb, 2015). | ||
Software-based | DIYSE software | Software that utilizes Guitar Hero controllers | Details development of software allowing connection of existing controllers (Guitar Hero & WiiMote, etc.) to compose and restore music tracks, and design mapping strategies between interface and played sounds, for people who with intellectual learning disabilities (Luhtala, Kymäläinen, & Plomp, 2011). | |
Empty-handed | Camera-based | Movement to Music-MTM | Web camera and software system | Developed to address the need for affordable home-based musical play system, incorporating automatic movement recognition technology that is noncontact and noninvasive (Tam, Schwellnus, Ceilidh, Hamdani, Lamont, & Chau, 2007). |
Break-beam | Benemic/Octonic | Stand-alone instrument | Device with array of eight low-cost infrared distance sensors. Enabling triggering and manipulation of sounds using MIDI messages (Challis, 2011). | |
Eye gaze | Eye conductor | Software uses webcam | Software-based musical interface to play music through eye movements and facial gesture using eye tracker equipment and webcam. Detects gaze and selected facial movements enabling playing of instruments, beat building, sequencing melodies, or triggering musical effects (Refsgaard, 2018). |
Interaction modality . | . | Commercially/ freely available . | Description . | Details . |
---|---|---|---|---|
Touch . | . | General technology . | . | . |
Switches | Broad range of electronic or mechanical tools to allow on/off selections | Tailorable to user, ease of integration to current resources, can be wired or wireless, trigger or start/stop sound effects, musical notes/phrases, recorded samples, or sequences of samples. “For example, the client may activate a CD player or pre-recorded music track” (Knight & Lagasse, 2012, p. 192). Bache et al. (2014) provide an in-depth review of practical switch use. | ||
iPad | Touch screen handheld device speaker, microphone, and motion detection | Multiuse device, tailorable to user, familiar technology, enticing, direct interaction with apps, self-contained with speaker on-board, commonly available. Useful for quickly recording multitrack music arrangements from live or plugged-in sources. Ability to capture sound using built-in microphone. Apps can emulate instruments enabling the client to play a stringed instrument by touching the screen, or a wind instrument by blowing into the microphone (Knight & Lagasse, 2012). “Apps can be used to record, synthesize, manipulate, or provide feedback to client actions and sounds” (p. 194). Knight (2013) provides an in-depth review of iPad applications in music therapy. | ||
iPod Touch | As iPad | Similar to iPad but smaller in form factor and with enhanced security (not connected to a network) | ||
Apps | Pieces of software for use on tablet or smartphone devices | Wide variety, some free. Apps for Children with Special Needs (2018) is a website for finding specific apps aimed at children with special needs featuring music as well as other apps with reviews and videos. Offering the ability to tailor content and interaction to client requirements. | ||
Generic music production controllers (MPCs) | Trigger pads with velocity sensing | MIDI-compatible. Some come with own software instruments, requires technical knowledge to set up. Provides ability to preselect sounds representative of a wide variety of genres enables clients an accessible way to perform their cultural or musical identities. Allows therapists to offer diverse sound palates valuable for playing differing feelings or emotions during improvisation (Crooke & McFerran, 2019). Can also be used in composition to perform and record drum beats and melodic patterns (Crooke, 2018), or to trigger or launch a range of loops or samples, allowing for the playing of precomposed pieces and live remixing. Music Radar (2018) provide an overview of available MPCs. | ||
Music-based video games | Video game system using generic or bespoke to the game controllers | Familiar to users, enticing technology, easy setup, can potentially use existing resources. Blaine (2005) provided a review of alternate music-based video game controllers. Wikipedia (2017) provides a list of music-based video games. Use of the WiiMote in music therapy has been explored (Benveniste, Jouvelot, & Michel, 2008), and developments such as the Wiinstrument can be downloaded to utilize the myriad of data that the WiiMote produces (The Wiinstrument, 2018). Games available include Wii Music by Nintendo. | ||
Specific devices | Description | Details | ||
Skoog 2 | Wireless foam cube with five tactile pressure-sensitive pads. | Wireless, portable, easy to set up. Simple and intuitive. Own app and software to customize sensitivity and sound created, MIDI-compatible. Many resources for use in special education on website. Provides direct correlation between physical contact and sound produced, using virtual musical instruments, samples, or MIDI. Offering dynamic control over musical gestures (Skoogmusic, 2018). “The instrument does not simply trigger samples when pressed but uses sophisticated synthesis to dynamically manipulate the various instrument sounds though pressing, squeezing, rubbing, stroking, tilting or manipulating the Skoog” (Farrimond et al., 2011, p. 28). | ||
Makey Makey | Microcontroller board with six connectors that emulate QWERTY keyboard presses and mouse control | Turn everyday objects into touchpads, no software to install, fast setup. Can be connected to conductive objects such as fruit, conductive tape, pencil graphite, and clients touching each other in chains as a means to trigger sounds (Makey Makey, 2018). The process allows appropriation enabling clients to gain authorship of their instruments, and become an intrinsic part of their deployment (Hayes, 2016). | ||
AlphaSphere | Globe-shaped MIDI controller with 48 playable velocity-sensitive pads and own software. | AlphaSphere is a tangible controller, when connected to AlphaLive software/used with other MIDI software can be set up to trigger and manipulate sound and provide a unique modality of access that can be enticing to clients (Place, Lacey, & Mitchell, 2014). | ||
MIDICreator | Device to convert signals from electronic sensors into MIDI data. | Clients can control sounds with physical actions and gestures, can be used to detect simple body movements (Krout, 2014a). Can be connected to MIDI to be used with other synthesizers. Sensors available include pressure, distance, proximity, direction, etc. (Meckin & Bryan-Kinns, 2013). | ||
I-CubeX | Software and digitizer for creating systems with a variety of sensors available | Tools to capture the human actions and/or environmental variables and make these signals available to other equipment such as a computer or a musical instrument to trigger sound, music, video, graphics, animation, robotic movement, etc. MIDI data are used, transmitted via MIDI cable, USB cable, or Bluetooth wireless (I-CubeX, 2018). | ||
Kyub | Eleven feather touch keypads on five surfaces of a 3-inch wooden cube with accelerometer | Maker friendly, open-source DIY MIDI keyboard featuring capacitive sensing and accelerometer. Fully programmable, set scale, tweak note velocity curves, map different instruments to different pads to configure to musical taste (Kyub MIDI keyboard, 2018). User can also configure the way the instrument looks and sounds by designing their own interface. | ||
Suzuki QChord | Electronic instrument with on-board speaker and LCD display. | A device for composition, teaching, and therapy. Incorporating technology from a basic keyboard and electric guitar and combining both in a portable, easy-to-use way. Eighty-four different chord combinations, 100 instrument voices, orchestrated rhythms. Features three sections; a touch-sensitive 4 octave “strum plate,” a rhythm section, and chord button section. Each of these areas can be used independently or combined with a variety of sounds obtainable. Sounds are always in tune. Can be adapted to all ability levels. Can stimulate interaction, increase coordination, stimulate gross and fine motor skills, and increase self-expression. Pitch bend wheel for expression. Changeable song cartridges. Can connect to speaker or MIDI device (Suzuki Qchord, 2018). | ||
Yamaha DJX-iiB | Groove machine in a box-shaped desktop module form with scratch pad and fader. | Can select from 70 patterns, mute parts, add preset hits or fills, and shift key of the patterns playing and use effects. Provides opportunities for the clients to compose music by assembling loops, importing other songs or sections of songs, and/or recording her or his own music, offering accessible and dynamic means of expressing emotions (Whitehead-Pleaux et al., 2011) | ||
Musii | Soft inflatable self-contained portable unit that emits sound and illuminates when three inflated cones are touched, with wireless control | Simple to use, needs no extra equipment or training to operate. Abilities to make music and colour through touch and movement with expansive library of soundscapes (over 50) and innovative musical system. Cannot play out of time or out of tune. A number of people can play harmoniously as a group or an individual can become an orchestra. Tactile physical form that enables the user to see and feel the beam that they are interacting with. “Musii has been designed for developmental play in the SEN sector...The multi-sensory experience of playing Musii has many therapeutic possibilities including encouraging movement, development and awareness of proprioception, turn taking, cause and effect, creativity, expression and communication. It can be used for stimulation or for calming. The synchronised sound and lighting as well as the visual and tactile feedback of the inflatable enable a deeper understanding of the music you are making” (Musii Ltd, 2017) | ||
Reactable | Table with touch screen and moveable objects (Reactable Technology, 2017) | Objects interact with the table surface and each other to make music. Allows intuitive and collective creation of complex musical pieces, collaborative sharing space between users, promotes imitation games, increases visibility of actions, enables monitoring other participant’s work, aids in reducing solitary play sequences, facilitates associative play (Villafuerte, Markova, & Jorda, 2012) | ||
Tenori-On | Handheld screen with 16 × 16 grid of LED switches. Built-in speakers, dial and buttons control sound and beats per minute produced | Create, control, or perform musical material on visually rich touch-sensitive interface (Farrimond et al., 2011). Switches activated in different ways create music from 256 sounds. Engaging, motivating, sensory, well suited for improvisation, easy to use/hard to master. Combines visual and melodic sensory information, can stimulate cognition, memory, and perception. Can function as a rhythm machine with basic or complex rhythmic beats that can be looped or changed. Notes and melodic phrases can be added as well, creating up to 16 layers of sound (Clements-Cortes, 2014). Clements-Cortes (2014) provided an in-depth study of the Tenori-On in a clinical music therapy setting. | ||
Roland Handsonic | Device with 13 ultra-sensitive touch pads | Eight hundred and fifty ready-to-play sounds, or import custom sounds. Responsive and therefore not overly demanding to play, with easy adjustment of volume (Challis & Smith, 2012). Can be calibrated to client need in terms of sensitivity. Can connect to other MIDI devices and MIDI can be recorded from the device for analysis. | ||
Korg Kaossilator/Kaossilator Pro/Korg Mini Kaoss Pad | Audio effects unit with an X/Y touch screen. Mini Kaoss pad features on-board speaker and microphone | Positioning a finger-tip on the touch screen triggers specific sound programs. Ability to trigger individual notes or patterns of notes depending on the nature of the chosen sound and settings selected. Moving around the screen moves between notes within a predefined scale or changes the nature of the sound. Allows the results of actions to be sampled as repeating loops. Particularly effective for those with restricted hand and finger movement, being easy to interact with, to produce complex musical ideas and patterns (Challis & Smith, 2012). | ||
SoundHouse Special Access Kit (Banana Keyboard)—discontinued | Sixteen keys configured like an oversized piano with software component | Curved to suit the radial movement of an arm. Light touch activates music, sounds, or speech programmed into each key. Allows connection of up to eight switches for activation of keys on the keyboard (Sound House, 2017). Innovative, adaptable control surface that require musicians to press, squeeze, or strike them to create and control music through corresponding musical software (Farrimond et al., 2011). User friendly, easy to learn. Arrange combinations of sounds (MIDI sounds or wave files). Recording feature enables real-life performances to be saved and voice and CD segments to be recorded. Arrangements can be printed off and formatted. Variety of global settings cutting down on individually programming each key or switch. Aimed at fostering development of switch use, cause-and-effect, timing, choosing (Sound House, 2017). | ||
Numark Orbit | Wireless MIDI controller | Sixteen backlit customizable pads, four selectable banks to assign up to 64 cues, lighting transitions, video clips, samples, etc. Control wheel and on-board accelerometer. Can be configured and mapped to control other MIDI software, accelerometer for motion control. Comes with demo software that show how to use with tracks and effects (Numark.com, 2018). | ||
Mogees | Resonance contact microphone | Enables instruments to be created from any surface/object alongside configuring of the sounds created with dedicated iPhone/iPad app. Integrates with standard digital audio workstations via Audio Units or VST plug-in. Can be used to provide expressive instrument by using different areas of surface trigger different sound (Mogees, 2018), allowing for objects to be used that are motivating, familiar, or engaging to clients. | ||
Software- based | General technology | Description | Details | |
Generic digital audio workstations (Ableton Live/Audacity/Reaper/Cubase/Logic/Sonare by Cakewalk/Garageband) | Music recording/composition software | Uses preset or user determined settings via on-screen, or pull-down menus. Allow for recording, composing, playback, and creation of music. Some come with content such as instruments and samples available for instant use. Ability to use software instruments (VSTs), input microphones, or electronic instruments. Garageband is preinstalled on Mac computers with out-of-the-box samples and instruments available. Reaper is affordably priced, Ableton is prized for live performance. Audacity is freeware. Can export notation in some cases. Krout (2014a) provides a review on using software for music composition, arranging, notating, improvising, and sequencing. | ||
Specific devices | Description | Details | ||
Clarion | Software instrument | Allows user to change every element instrument including sound, notes, shape, position and colour of notes, and how those notes are played. Integrates with eye gaze systems, SmartNav and iPad. Package offered by Open Orchestras including repertoire, training resources, support, and an evaluation framework (Open Orchestras, 2018). | ||
Magix Music Maker | Digital audio workstation | Provides 425 sounds & loops, seven free Soundpools (1,927 sounds & loops) three software instruments, eight tracks, and eight effects. Can be used with smart boards. | ||
HyperScore | A graphical composition environment | Users draw strokes and lines to explore musical ideas. Graphical elements are mapped to musical structures, allowing users to shape musical progressions visually (Grierson & Kiefer, 2013; Machover, 2004). | ||
MIDIGrid | Music software | Uses mouse and keyboard movements within a grid to trigger notes, chords, sequences, or patterns of sound that can be played back and looped (Hunt & Kirk, 2003). | ||
Microsoft Songsmith | Music software | Generates musical accompaniment to match a singer’s voice using computer microphone input. Musical style and feel of song can be selected. Songs can be posted songs online, or used to create music videos | ||
2Simple music toolkit | Software applications | Six programs that introduce key musical concepts in an interactive way (2Simple, 2017). | ||
Empty-handed | Camera-based | Microsoft Kinect | RGB camera, depth sensor, and multi-array microphone | Provides full body 3D motion capture, voice and face recognition (KinectSEN, 2018). |
EyesWeb | Open development software platform | Real-time multimodal system and interface that has been used extensively in research. System supports input devices including motion capture systems, video cameras, game interfaces (e.g., Kinect, Wii), audio input, analog inputs (e.g., for physiological signals). Outputs include multichannel audio, video, analog devices, robotic platforms (Camurri et al., 2000). Website features information on development (http://www.infomus.org/eyesweb_ita.php) | ||
MotionComposer (available 2019) | System using two types of camera to detect movement | Allows gestures to be used to explore sound environments (Bergsland & Wechsler, 2016). | ||
AUMI | Free software application | Interface that enables the user to play sounds and musical phrases through movement and gestures captured via webcam (Larsen, Overholt, & Moeslund, 2016; Oliveros, Miller, Heyen, Siddall, & Hazard, 2011). | ||
VMI (Virtual Musical Instrument) | Free software | Uses web camera to detect motion. User virtually “touch” shapes on screen to trigger sounds. Requires no special equipment, Windows only based computer. Designed for use by therapists and educators, it is customizable according to the preferences and needs of the user, and can be used for specific therapy or educational goals (Virtual Musical Instrument, 2018). | ||
BigEye—discontinued still available for download | Macintosh only software program | Uses video information to convert into MIDI messages. Allows tracking of objects through space converting their parameters into MIDI in real-time (Legacy Products, 2018). | ||
Camera Theremin | Free test application | Create sound from movement using webcam (Camera Theremin, 2018). | ||
Musical Gestures Toolbox | Toolkit for experienced Max programmers. | Collection of modules and abstractions for the graphical programming environment Max 5 to enable extraction of movement data from video (Jensenius, Godoy, & Wanderley, 2005). | ||
Aerodrum | Package featuring drumsticks, software, feet markers, and camera | An air-drumming instrument. Runs on computer using a high-speed camera to track movements to trigger drums (Knight & Krout 2017). | ||
Break-beam | Beamz | Device featuring four breakable laser beams | Can be purchased as a professional package featuring software, songs, structured activities, therapy guides, and lesson plans | |
Soundbeam | Device featuring ultrasonic beams and switches (both wired or wireless) and a synthesizer unit | Device which uses sensor technology (up to four ultrasonic or eight switches) to translate body movement into music and sound using MIDI. New unit features touch screen interface, extensive library of sounds, recording and composing functions, training programs also available. Extensive support for use available through online resources (https://www.soundbeam.co.uk/). | ||
Theremin (Magee, 2006) | Moog Theremini—device featuring 32 wavetable preset sounds, and on-board speaker and sound engine | Can be used at any skill level, providing new ways to experiment with music and gestural control. Assistive pitch quantization allows each player to adjust the instrument’s level of playing difficulty. “At the maximum position, the Theremini will play every note in a selected scale perfectly, making it impossible to play a wrong note. As control is decreased, more expressive control of pitch and vibrato becomes possible. When set to minimum, the Theremini will perform as a traditional Theremin” (Theremini, 2018). Built-in tuner with real-time visual feedback of played notes and proximity (useful for correcting playing position). Store selected scale & root note, set and recall a specified playing range, and specify patch settings. Silent rehearsal available via headphone jack. Two line level audio outputs, a pitch CV output with selectable range, and a mini USB jack for MIDI I/O and connectivity (Theremini, 2018). | ||
Optimusic/OptiBeams (Knight and Krout 2017) | Interactive light beams | Package with interactive light units (the beams), laptop with OptiMusic software, USB controller box, user button box, reflective pads/bats. Interact with colorful beams of light (2, 4, 6, 8, or 12 beams), pass hand or body through the beams or use reflective wand to trigger audiovisual events in real-time. Comes with over 80 interactive settings. Package also comes with training (on-site or e-training) (OptiBeam, 2018). | ||
MidiGesture | Ultrasonic beam sensor | Sensor that plugs into the MIDICreator system (see MIDICreator). | ||
Leap Motion (Leap Motion, 2017) | Small device to track hand movement | Uses two monochromatic IR cameras and three infrared LEDs to track hand and finger movement above device. Dickens, Greenhalgh, and Koleva (2017) provide an in-depth description of research conducted using the Leap Motion for music performance with users with complex disabilities. | ||
Brain computer interface (BCI) | Brainfingers (Brainfingers, 2018) | Headband fitted with sensors | “Detects electrical signals from facial muscles, eye movement and brain waves. Brainfingers does not directly target music creation, as it can solve many tasks such as simple clicking, to complex combinations of controls. It is software that converts all the sensor input data into controls termed Brainfingers. This software is useful for a broad range of users, especially people with severe disabilities” (Larsen et al., 2016, p. 329). Controls most AAC software, educational software, and video games. | |
Eye gaze | EyeMusic (Larsen et al., 2016)—legacy files available online, requires technical skill to install (EyeMusic, 2018) | Software utilizing generic eye gaze equipment | System that uses eye movements as input to electronic music compositions. Can be used with established composition software allowing prerecorded eye movement data to control musical compositions (Hornof & Sato, 2004). | |
E-Scape (Anderson, 2018) | Software utilizing generic eye gaze equipment (or switches) | Music software specifically designed to be used by people with disabilities to create or perform music. System operates via large guided pop-up menus controllable by one or more switches, mouse, keyboard, eye gaze, or MIDI controllers or sensors. “At every stage, E-Scape asks the user what they want to do and offers a range of options depending upon which level of complexity the user has chosen to work at” (Farrimond et al., 2011, p. 23). Two modes of operation—composition and performance. Can output MIDI data (Farrimond et al., 2011). | ||
EyeHarp | Free software utilizing generic eye gaze equipment | Gaze-controlled or head-controlled music interface to help users learn and play music. Vamvakousis and Ramirez (2016) provide a comprehensive article on the development of the EyeHarp. The website theeyeharp.org also provides a wealth of information about the project (The EyeHarp, 2018). | ||
Eye Play Music | Free software utilizing generic eye gaze equipment | Trigger notes from a range of instruments available with adjustable note length and transposition. Create own scales. Load and save settings. Website features resource for use (MBMM, 2017). | ||
Breath | Jamboxx | MIDI controller device | Hands-free electronic, breath-powered instrument. Uses sip and puff to determine amplitude of note. Software included to configure device and on-board modulation wheel, button, and jack socket to allow switch connectivity (Jamboxx, 2018). | |
Yamaha WX5, WX11 | MIDI controller devices | Breath powered MIDI controller that allows for one-handed playing. WX5 features MIDI output; however, WX11 requires an additional MIDI connection box (MBMM, 2017). | ||
Magic Flute | Stand-alone instrument | Self-contained instrument with built-in tone generating hardware. Plugs into external speakers. Two separate parts, the flute and control module with display. The flute being the remote control for the control module. Musicians can select different sounds or access the user settings without the help of another person (MBMM, 2017). The volume is controlled by blowing in a mouthpiece and the pitch by moving the mouthpiece up/down with the mouth (Vamvakousis & Ramirez, 2016). “The instrument reduces the physical and cognitive challenges inherent within conventional wind instruments. One musician, with very limited lung volume, is nonetheless able to realize the full dynamic range of the instrument” (Farrimond et al., 2011, p. 29). | ||
Interaction modality | Research only | Description | Details | |
Touch | General research | |||
Collaborative interfaces review of literature (Blaine and Fels, 2003) | Review paper of interfaces used for collaborative music-making | Comprehensive review of context and design of a number musical experiences for novices | ||
Specific research | ||||
Musicking Tangibles—RHYME Project | Tangible interfaces consisting of interactive digital “furniture” | Examining the development and benefits of using interactive digital music furniture for disabled children by using two co-creative tangible instruments. ORFI-26 soft pyramid-shaped, pillow-like modules, in three different sizes (30–90 cm) featuring bend sensors and lights, the units can communicate wirelessly with each other. Wave Carpet-7-branched, wired, interactive, soft, dark carpet with orange velvet tips that glow. Central arm contains microphone, two arms contain accelerometers that change the recorded sound. Two arms contain bend sensors that create rhythmical background music. One arm contains a web camera. Contain five software programs, offering different music and dynamic graphics to show via projector embedded in one arm, or via full wall projection. Center contains two speakers and strong vibrator in. Contains IR-sensors allowing interaction with RGB LED lights (Andersson & Cappelen, 2013). | ||
NoiseBear (Grierson and Kiefer, 2013) | Malleable controller | Development of robust, wireless, malleable controller for children with cognitive or physical disabilities | ||
Bean | Gesturally controlled digital instrument | Device designed around a Wii nunchuck controller for use in a music therapy setting (Kirwan, Overholt, & Erkut, 2015) | ||
MAWii | Digital musical instrument using generic WiiMote controller | Research exploring WiiMotes as virtual instruments for children with behavioral disorders (Benveniste et al., 2008). | ||
WamBam | Self-contained electronic hand drum | Created using piezo sensors. Paper describes development and testing of device used in for music therapy sessions with severely intellectually disabled clients (Jense & Leeuw, 2015). | ||
TouchTone | Digital musical instrument | Device featuring touch-sensitive pads designed to develop musical ability, bimanual coordination and increase social participation of children with hemiplegia (Bhat, 2010). | ||
Computer-Assisted Music Therapy | Augmented reality software | Details system developed with Augmented Reality techniques allowing music composition and creation activities using sound and colour, via cards (Correa, Ficheman, do Nascimento, & de Deus Lopes, 2009). | ||
SenseEgg | Wireless controller device | Development of a handheld egg-shaped device featuring seven on-board sensors (button, slider, accelerometer, wind sensor, ultrasonic distance sensor) and a suite of software patches aimed at for musical exploration and teaching. Featured a component that allowed control of settings via an iPad (Blatherwick & Cobb, 2015). | ||
Software-based | DIYSE software | Software that utilizes Guitar Hero controllers | Details development of software allowing connection of existing controllers (Guitar Hero & WiiMote, etc.) to compose and restore music tracks, and design mapping strategies between interface and played sounds, for people who with intellectual learning disabilities (Luhtala, Kymäläinen, & Plomp, 2011). | |
Empty-handed | Camera-based | Movement to Music-MTM | Web camera and software system | Developed to address the need for affordable home-based musical play system, incorporating automatic movement recognition technology that is noncontact and noninvasive (Tam, Schwellnus, Ceilidh, Hamdani, Lamont, & Chau, 2007). |
Break-beam | Benemic/Octonic | Stand-alone instrument | Device with array of eight low-cost infrared distance sensors. Enabling triggering and manipulation of sounds using MIDI messages (Challis, 2011). | |
Eye gaze | Eye conductor | Software uses webcam | Software-based musical interface to play music through eye movements and facial gesture using eye tracker equipment and webcam. Detects gaze and selected facial movements enabling playing of instruments, beat building, sequencing melodies, or triggering musical effects (Refsgaard, 2018). |
Note. MIDI, musical instrument digital interface.
Interaction modality . | . | Commercially/ freely available . | Description . | Details . |
---|---|---|---|---|
Touch . | . | General technology . | . | . |
Switches | Broad range of electronic or mechanical tools to allow on/off selections | Tailorable to user, ease of integration to current resources, can be wired or wireless, trigger or start/stop sound effects, musical notes/phrases, recorded samples, or sequences of samples. “For example, the client may activate a CD player or pre-recorded music track” (Knight & Lagasse, 2012, p. 192). Bache et al. (2014) provide an in-depth review of practical switch use. | ||
iPad | Touch screen handheld device speaker, microphone, and motion detection | Multiuse device, tailorable to user, familiar technology, enticing, direct interaction with apps, self-contained with speaker on-board, commonly available. Useful for quickly recording multitrack music arrangements from live or plugged-in sources. Ability to capture sound using built-in microphone. Apps can emulate instruments enabling the client to play a stringed instrument by touching the screen, or a wind instrument by blowing into the microphone (Knight & Lagasse, 2012). “Apps can be used to record, synthesize, manipulate, or provide feedback to client actions and sounds” (p. 194). Knight (2013) provides an in-depth review of iPad applications in music therapy. | ||
iPod Touch | As iPad | Similar to iPad but smaller in form factor and with enhanced security (not connected to a network) | ||
Apps | Pieces of software for use on tablet or smartphone devices | Wide variety, some free. Apps for Children with Special Needs (2018) is a website for finding specific apps aimed at children with special needs featuring music as well as other apps with reviews and videos. Offering the ability to tailor content and interaction to client requirements. | ||
Generic music production controllers (MPCs) | Trigger pads with velocity sensing | MIDI-compatible. Some come with own software instruments, requires technical knowledge to set up. Provides ability to preselect sounds representative of a wide variety of genres enables clients an accessible way to perform their cultural or musical identities. Allows therapists to offer diverse sound palates valuable for playing differing feelings or emotions during improvisation (Crooke & McFerran, 2019). Can also be used in composition to perform and record drum beats and melodic patterns (Crooke, 2018), or to trigger or launch a range of loops or samples, allowing for the playing of precomposed pieces and live remixing. Music Radar (2018) provide an overview of available MPCs. | ||
Music-based video games | Video game system using generic or bespoke to the game controllers | Familiar to users, enticing technology, easy setup, can potentially use existing resources. Blaine (2005) provided a review of alternate music-based video game controllers. Wikipedia (2017) provides a list of music-based video games. Use of the WiiMote in music therapy has been explored (Benveniste, Jouvelot, & Michel, 2008), and developments such as the Wiinstrument can be downloaded to utilize the myriad of data that the WiiMote produces (The Wiinstrument, 2018). Games available include Wii Music by Nintendo. | ||
Specific devices | Description | Details | ||
Skoog 2 | Wireless foam cube with five tactile pressure-sensitive pads. | Wireless, portable, easy to set up. Simple and intuitive. Own app and software to customize sensitivity and sound created, MIDI-compatible. Many resources for use in special education on website. Provides direct correlation between physical contact and sound produced, using virtual musical instruments, samples, or MIDI. Offering dynamic control over musical gestures (Skoogmusic, 2018). “The instrument does not simply trigger samples when pressed but uses sophisticated synthesis to dynamically manipulate the various instrument sounds though pressing, squeezing, rubbing, stroking, tilting or manipulating the Skoog” (Farrimond et al., 2011, p. 28). | ||
Makey Makey | Microcontroller board with six connectors that emulate QWERTY keyboard presses and mouse control | Turn everyday objects into touchpads, no software to install, fast setup. Can be connected to conductive objects such as fruit, conductive tape, pencil graphite, and clients touching each other in chains as a means to trigger sounds (Makey Makey, 2018). The process allows appropriation enabling clients to gain authorship of their instruments, and become an intrinsic part of their deployment (Hayes, 2016). | ||
AlphaSphere | Globe-shaped MIDI controller with 48 playable velocity-sensitive pads and own software. | AlphaSphere is a tangible controller, when connected to AlphaLive software/used with other MIDI software can be set up to trigger and manipulate sound and provide a unique modality of access that can be enticing to clients (Place, Lacey, & Mitchell, 2014). | ||
MIDICreator | Device to convert signals from electronic sensors into MIDI data. | Clients can control sounds with physical actions and gestures, can be used to detect simple body movements (Krout, 2014a). Can be connected to MIDI to be used with other synthesizers. Sensors available include pressure, distance, proximity, direction, etc. (Meckin & Bryan-Kinns, 2013). | ||
I-CubeX | Software and digitizer for creating systems with a variety of sensors available | Tools to capture the human actions and/or environmental variables and make these signals available to other equipment such as a computer or a musical instrument to trigger sound, music, video, graphics, animation, robotic movement, etc. MIDI data are used, transmitted via MIDI cable, USB cable, or Bluetooth wireless (I-CubeX, 2018). | ||
Kyub | Eleven feather touch keypads on five surfaces of a 3-inch wooden cube with accelerometer | Maker friendly, open-source DIY MIDI keyboard featuring capacitive sensing and accelerometer. Fully programmable, set scale, tweak note velocity curves, map different instruments to different pads to configure to musical taste (Kyub MIDI keyboard, 2018). User can also configure the way the instrument looks and sounds by designing their own interface. | ||
Suzuki QChord | Electronic instrument with on-board speaker and LCD display. | A device for composition, teaching, and therapy. Incorporating technology from a basic keyboard and electric guitar and combining both in a portable, easy-to-use way. Eighty-four different chord combinations, 100 instrument voices, orchestrated rhythms. Features three sections; a touch-sensitive 4 octave “strum plate,” a rhythm section, and chord button section. Each of these areas can be used independently or combined with a variety of sounds obtainable. Sounds are always in tune. Can be adapted to all ability levels. Can stimulate interaction, increase coordination, stimulate gross and fine motor skills, and increase self-expression. Pitch bend wheel for expression. Changeable song cartridges. Can connect to speaker or MIDI device (Suzuki Qchord, 2018). | ||
Yamaha DJX-iiB | Groove machine in a box-shaped desktop module form with scratch pad and fader. | Can select from 70 patterns, mute parts, add preset hits or fills, and shift key of the patterns playing and use effects. Provides opportunities for the clients to compose music by assembling loops, importing other songs or sections of songs, and/or recording her or his own music, offering accessible and dynamic means of expressing emotions (Whitehead-Pleaux et al., 2011) | ||
Musii | Soft inflatable self-contained portable unit that emits sound and illuminates when three inflated cones are touched, with wireless control | Simple to use, needs no extra equipment or training to operate. Abilities to make music and colour through touch and movement with expansive library of soundscapes (over 50) and innovative musical system. Cannot play out of time or out of tune. A number of people can play harmoniously as a group or an individual can become an orchestra. Tactile physical form that enables the user to see and feel the beam that they are interacting with. “Musii has been designed for developmental play in the SEN sector...The multi-sensory experience of playing Musii has many therapeutic possibilities including encouraging movement, development and awareness of proprioception, turn taking, cause and effect, creativity, expression and communication. It can be used for stimulation or for calming. The synchronised sound and lighting as well as the visual and tactile feedback of the inflatable enable a deeper understanding of the music you are making” (Musii Ltd, 2017) | ||
Reactable | Table with touch screen and moveable objects (Reactable Technology, 2017) | Objects interact with the table surface and each other to make music. Allows intuitive and collective creation of complex musical pieces, collaborative sharing space between users, promotes imitation games, increases visibility of actions, enables monitoring other participant’s work, aids in reducing solitary play sequences, facilitates associative play (Villafuerte, Markova, & Jorda, 2012) | ||
Tenori-On | Handheld screen with 16 × 16 grid of LED switches. Built-in speakers, dial and buttons control sound and beats per minute produced | Create, control, or perform musical material on visually rich touch-sensitive interface (Farrimond et al., 2011). Switches activated in different ways create music from 256 sounds. Engaging, motivating, sensory, well suited for improvisation, easy to use/hard to master. Combines visual and melodic sensory information, can stimulate cognition, memory, and perception. Can function as a rhythm machine with basic or complex rhythmic beats that can be looped or changed. Notes and melodic phrases can be added as well, creating up to 16 layers of sound (Clements-Cortes, 2014). Clements-Cortes (2014) provided an in-depth study of the Tenori-On in a clinical music therapy setting. | ||
Roland Handsonic | Device with 13 ultra-sensitive touch pads | Eight hundred and fifty ready-to-play sounds, or import custom sounds. Responsive and therefore not overly demanding to play, with easy adjustment of volume (Challis & Smith, 2012). Can be calibrated to client need in terms of sensitivity. Can connect to other MIDI devices and MIDI can be recorded from the device for analysis. | ||
Korg Kaossilator/Kaossilator Pro/Korg Mini Kaoss Pad | Audio effects unit with an X/Y touch screen. Mini Kaoss pad features on-board speaker and microphone | Positioning a finger-tip on the touch screen triggers specific sound programs. Ability to trigger individual notes or patterns of notes depending on the nature of the chosen sound and settings selected. Moving around the screen moves between notes within a predefined scale or changes the nature of the sound. Allows the results of actions to be sampled as repeating loops. Particularly effective for those with restricted hand and finger movement, being easy to interact with, to produce complex musical ideas and patterns (Challis & Smith, 2012). | ||
SoundHouse Special Access Kit (Banana Keyboard)—discontinued | Sixteen keys configured like an oversized piano with software component | Curved to suit the radial movement of an arm. Light touch activates music, sounds, or speech programmed into each key. Allows connection of up to eight switches for activation of keys on the keyboard (Sound House, 2017). Innovative, adaptable control surface that require musicians to press, squeeze, or strike them to create and control music through corresponding musical software (Farrimond et al., 2011). User friendly, easy to learn. Arrange combinations of sounds (MIDI sounds or wave files). Recording feature enables real-life performances to be saved and voice and CD segments to be recorded. Arrangements can be printed off and formatted. Variety of global settings cutting down on individually programming each key or switch. Aimed at fostering development of switch use, cause-and-effect, timing, choosing (Sound House, 2017). | ||
Numark Orbit | Wireless MIDI controller | Sixteen backlit customizable pads, four selectable banks to assign up to 64 cues, lighting transitions, video clips, samples, etc. Control wheel and on-board accelerometer. Can be configured and mapped to control other MIDI software, accelerometer for motion control. Comes with demo software that show how to use with tracks and effects (Numark.com, 2018). | ||
Mogees | Resonance contact microphone | Enables instruments to be created from any surface/object alongside configuring of the sounds created with dedicated iPhone/iPad app. Integrates with standard digital audio workstations via Audio Units or VST plug-in. Can be used to provide expressive instrument by using different areas of surface trigger different sound (Mogees, 2018), allowing for objects to be used that are motivating, familiar, or engaging to clients. | ||
Software- based | General technology | Description | Details | |
Generic digital audio workstations (Ableton Live/Audacity/Reaper/Cubase/Logic/Sonare by Cakewalk/Garageband) | Music recording/composition software | Uses preset or user determined settings via on-screen, or pull-down menus. Allow for recording, composing, playback, and creation of music. Some come with content such as instruments and samples available for instant use. Ability to use software instruments (VSTs), input microphones, or electronic instruments. Garageband is preinstalled on Mac computers with out-of-the-box samples and instruments available. Reaper is affordably priced, Ableton is prized for live performance. Audacity is freeware. Can export notation in some cases. Krout (2014a) provides a review on using software for music composition, arranging, notating, improvising, and sequencing. | ||
Specific devices | Description | Details | ||
Clarion | Software instrument | Allows user to change every element instrument including sound, notes, shape, position and colour of notes, and how those notes are played. Integrates with eye gaze systems, SmartNav and iPad. Package offered by Open Orchestras including repertoire, training resources, support, and an evaluation framework (Open Orchestras, 2018). | ||
Magix Music Maker | Digital audio workstation | Provides 425 sounds & loops, seven free Soundpools (1,927 sounds & loops) three software instruments, eight tracks, and eight effects. Can be used with smart boards. | ||
HyperScore | A graphical composition environment | Users draw strokes and lines to explore musical ideas. Graphical elements are mapped to musical structures, allowing users to shape musical progressions visually (Grierson & Kiefer, 2013; Machover, 2004). | ||
MIDIGrid | Music software | Uses mouse and keyboard movements within a grid to trigger notes, chords, sequences, or patterns of sound that can be played back and looped (Hunt & Kirk, 2003). | ||
Microsoft Songsmith | Music software | Generates musical accompaniment to match a singer’s voice using computer microphone input. Musical style and feel of song can be selected. Songs can be posted songs online, or used to create music videos | ||
2Simple music toolkit | Software applications | Six programs that introduce key musical concepts in an interactive way (2Simple, 2017). | ||
Empty-handed | Camera-based | Microsoft Kinect | RGB camera, depth sensor, and multi-array microphone | Provides full body 3D motion capture, voice and face recognition (KinectSEN, 2018). |
EyesWeb | Open development software platform | Real-time multimodal system and interface that has been used extensively in research. System supports input devices including motion capture systems, video cameras, game interfaces (e.g., Kinect, Wii), audio input, analog inputs (e.g., for physiological signals). Outputs include multichannel audio, video, analog devices, robotic platforms (Camurri et al., 2000). Website features information on development (http://www.infomus.org/eyesweb_ita.php) | ||
MotionComposer (available 2019) | System using two types of camera to detect movement | Allows gestures to be used to explore sound environments (Bergsland & Wechsler, 2016). | ||
AUMI | Free software application | Interface that enables the user to play sounds and musical phrases through movement and gestures captured via webcam (Larsen, Overholt, & Moeslund, 2016; Oliveros, Miller, Heyen, Siddall, & Hazard, 2011). | ||
VMI (Virtual Musical Instrument) | Free software | Uses web camera to detect motion. User virtually “touch” shapes on screen to trigger sounds. Requires no special equipment, Windows only based computer. Designed for use by therapists and educators, it is customizable according to the preferences and needs of the user, and can be used for specific therapy or educational goals (Virtual Musical Instrument, 2018). | ||
BigEye—discontinued still available for download | Macintosh only software program | Uses video information to convert into MIDI messages. Allows tracking of objects through space converting their parameters into MIDI in real-time (Legacy Products, 2018). | ||
Camera Theremin | Free test application | Create sound from movement using webcam (Camera Theremin, 2018). | ||
Musical Gestures Toolbox | Toolkit for experienced Max programmers. | Collection of modules and abstractions for the graphical programming environment Max 5 to enable extraction of movement data from video (Jensenius, Godoy, & Wanderley, 2005). | ||
Aerodrum | Package featuring drumsticks, software, feet markers, and camera | An air-drumming instrument. Runs on computer using a high-speed camera to track movements to trigger drums (Knight & Krout 2017). | ||
Break-beam | Beamz | Device featuring four breakable laser beams | Can be purchased as a professional package featuring software, songs, structured activities, therapy guides, and lesson plans | |
Soundbeam | Device featuring ultrasonic beams and switches (both wired or wireless) and a synthesizer unit | Device which uses sensor technology (up to four ultrasonic or eight switches) to translate body movement into music and sound using MIDI. New unit features touch screen interface, extensive library of sounds, recording and composing functions, training programs also available. Extensive support for use available through online resources (https://www.soundbeam.co.uk/). | ||
Theremin (Magee, 2006) | Moog Theremini—device featuring 32 wavetable preset sounds, and on-board speaker and sound engine | Can be used at any skill level, providing new ways to experiment with music and gestural control. Assistive pitch quantization allows each player to adjust the instrument’s level of playing difficulty. “At the maximum position, the Theremini will play every note in a selected scale perfectly, making it impossible to play a wrong note. As control is decreased, more expressive control of pitch and vibrato becomes possible. When set to minimum, the Theremini will perform as a traditional Theremin” (Theremini, 2018). Built-in tuner with real-time visual feedback of played notes and proximity (useful for correcting playing position). Store selected scale & root note, set and recall a specified playing range, and specify patch settings. Silent rehearsal available via headphone jack. Two line level audio outputs, a pitch CV output with selectable range, and a mini USB jack for MIDI I/O and connectivity (Theremini, 2018). | ||
Optimusic/OptiBeams (Knight and Krout 2017) | Interactive light beams | Package with interactive light units (the beams), laptop with OptiMusic software, USB controller box, user button box, reflective pads/bats. Interact with colorful beams of light (2, 4, 6, 8, or 12 beams), pass hand or body through the beams or use reflective wand to trigger audiovisual events in real-time. Comes with over 80 interactive settings. Package also comes with training (on-site or e-training) (OptiBeam, 2018). | ||
MidiGesture | Ultrasonic beam sensor | Sensor that plugs into the MIDICreator system (see MIDICreator). | ||
Leap Motion (Leap Motion, 2017) | Small device to track hand movement | Uses two monochromatic IR cameras and three infrared LEDs to track hand and finger movement above device. Dickens, Greenhalgh, and Koleva (2017) provide an in-depth description of research conducted using the Leap Motion for music performance with users with complex disabilities. | ||
Brain computer interface (BCI) | Brainfingers (Brainfingers, 2018) | Headband fitted with sensors | “Detects electrical signals from facial muscles, eye movement and brain waves. Brainfingers does not directly target music creation, as it can solve many tasks such as simple clicking, to complex combinations of controls. It is software that converts all the sensor input data into controls termed Brainfingers. This software is useful for a broad range of users, especially people with severe disabilities” (Larsen et al., 2016, p. 329). Controls most AAC software, educational software, and video games. | |
Eye gaze | EyeMusic (Larsen et al., 2016)—legacy files available online, requires technical skill to install (EyeMusic, 2018) | Software utilizing generic eye gaze equipment | System that uses eye movements as input to electronic music compositions. Can be used with established composition software allowing prerecorded eye movement data to control musical compositions (Hornof & Sato, 2004). | |
E-Scape (Anderson, 2018) | Software utilizing generic eye gaze equipment (or switches) | Music software specifically designed to be used by people with disabilities to create or perform music. System operates via large guided pop-up menus controllable by one or more switches, mouse, keyboard, eye gaze, or MIDI controllers or sensors. “At every stage, E-Scape asks the user what they want to do and offers a range of options depending upon which level of complexity the user has chosen to work at” (Farrimond et al., 2011, p. 23). Two modes of operation—composition and performance. Can output MIDI data (Farrimond et al., 2011). | ||
EyeHarp | Free software utilizing generic eye gaze equipment | Gaze-controlled or head-controlled music interface to help users learn and play music. Vamvakousis and Ramirez (2016) provide a comprehensive article on the development of the EyeHarp. The website theeyeharp.org also provides a wealth of information about the project (The EyeHarp, 2018). | ||
Eye Play Music | Free software utilizing generic eye gaze equipment | Trigger notes from a range of instruments available with adjustable note length and transposition. Create own scales. Load and save settings. Website features resource for use (MBMM, 2017). | ||
Breath | Jamboxx | MIDI controller device | Hands-free electronic, breath-powered instrument. Uses sip and puff to determine amplitude of note. Software included to configure device and on-board modulation wheel, button, and jack socket to allow switch connectivity (Jamboxx, 2018). | |
Yamaha WX5, WX11 | MIDI controller devices | Breath powered MIDI controller that allows for one-handed playing. WX5 features MIDI output; however, WX11 requires an additional MIDI connection box (MBMM, 2017). | ||
Magic Flute | Stand-alone instrument | Self-contained instrument with built-in tone generating hardware. Plugs into external speakers. Two separate parts, the flute and control module with display. The flute being the remote control for the control module. Musicians can select different sounds or access the user settings without the help of another person (MBMM, 2017). The volume is controlled by blowing in a mouthpiece and the pitch by moving the mouthpiece up/down with the mouth (Vamvakousis & Ramirez, 2016). “The instrument reduces the physical and cognitive challenges inherent within conventional wind instruments. One musician, with very limited lung volume, is nonetheless able to realize the full dynamic range of the instrument” (Farrimond et al., 2011, p. 29). | ||
Interaction modality | Research only | Description | Details | |
Touch | General research | |||
Collaborative interfaces review of literature (Blaine and Fels, 2003) | Review paper of interfaces used for collaborative music-making | Comprehensive review of context and design of a number musical experiences for novices | ||
Specific research | ||||
Musicking Tangibles—RHYME Project | Tangible interfaces consisting of interactive digital “furniture” | Examining the development and benefits of using interactive digital music furniture for disabled children by using two co-creative tangible instruments. ORFI-26 soft pyramid-shaped, pillow-like modules, in three different sizes (30–90 cm) featuring bend sensors and lights, the units can communicate wirelessly with each other. Wave Carpet-7-branched, wired, interactive, soft, dark carpet with orange velvet tips that glow. Central arm contains microphone, two arms contain accelerometers that change the recorded sound. Two arms contain bend sensors that create rhythmical background music. One arm contains a web camera. Contain five software programs, offering different music and dynamic graphics to show via projector embedded in one arm, or via full wall projection. Center contains two speakers and strong vibrator in. Contains IR-sensors allowing interaction with RGB LED lights (Andersson & Cappelen, 2013). | ||
NoiseBear (Grierson and Kiefer, 2013) | Malleable controller | Development of robust, wireless, malleable controller for children with cognitive or physical disabilities | ||
Bean | Gesturally controlled digital instrument | Device designed around a Wii nunchuck controller for use in a music therapy setting (Kirwan, Overholt, & Erkut, 2015) | ||
MAWii | Digital musical instrument using generic WiiMote controller | Research exploring WiiMotes as virtual instruments for children with behavioral disorders (Benveniste et al., 2008). | ||
WamBam | Self-contained electronic hand drum | Created using piezo sensors. Paper describes development and testing of device used in for music therapy sessions with severely intellectually disabled clients (Jense & Leeuw, 2015). | ||
TouchTone | Digital musical instrument | Device featuring touch-sensitive pads designed to develop musical ability, bimanual coordination and increase social participation of children with hemiplegia (Bhat, 2010). | ||
Computer-Assisted Music Therapy | Augmented reality software | Details system developed with Augmented Reality techniques allowing music composition and creation activities using sound and colour, via cards (Correa, Ficheman, do Nascimento, & de Deus Lopes, 2009). | ||
SenseEgg | Wireless controller device | Development of a handheld egg-shaped device featuring seven on-board sensors (button, slider, accelerometer, wind sensor, ultrasonic distance sensor) and a suite of software patches aimed at for musical exploration and teaching. Featured a component that allowed control of settings via an iPad (Blatherwick & Cobb, 2015). | ||
Software-based | DIYSE software | Software that utilizes Guitar Hero controllers | Details development of software allowing connection of existing controllers (Guitar Hero & WiiMote, etc.) to compose and restore music tracks, and design mapping strategies between interface and played sounds, for people who with intellectual learning disabilities (Luhtala, Kymäläinen, & Plomp, 2011). | |
Empty-handed | Camera-based | Movement to Music-MTM | Web camera and software system | Developed to address the need for affordable home-based musical play system, incorporating automatic movement recognition technology that is noncontact and noninvasive (Tam, Schwellnus, Ceilidh, Hamdani, Lamont, & Chau, 2007). |
Break-beam | Benemic/Octonic | Stand-alone instrument | Device with array of eight low-cost infrared distance sensors. Enabling triggering and manipulation of sounds using MIDI messages (Challis, 2011). | |
Eye gaze | Eye conductor | Software uses webcam | Software-based musical interface to play music through eye movements and facial gesture using eye tracker equipment and webcam. Detects gaze and selected facial movements enabling playing of instruments, beat building, sequencing melodies, or triggering musical effects (Refsgaard, 2018). |
Interaction modality . | . | Commercially/ freely available . | Description . | Details . |
---|---|---|---|---|
Touch . | . | General technology . | . | . |
Switches | Broad range of electronic or mechanical tools to allow on/off selections | Tailorable to user, ease of integration to current resources, can be wired or wireless, trigger or start/stop sound effects, musical notes/phrases, recorded samples, or sequences of samples. “For example, the client may activate a CD player or pre-recorded music track” (Knight & Lagasse, 2012, p. 192). Bache et al. (2014) provide an in-depth review of practical switch use. | ||
iPad | Touch screen handheld device speaker, microphone, and motion detection | Multiuse device, tailorable to user, familiar technology, enticing, direct interaction with apps, self-contained with speaker on-board, commonly available. Useful for quickly recording multitrack music arrangements from live or plugged-in sources. Ability to capture sound using built-in microphone. Apps can emulate instruments enabling the client to play a stringed instrument by touching the screen, or a wind instrument by blowing into the microphone (Knight & Lagasse, 2012). “Apps can be used to record, synthesize, manipulate, or provide feedback to client actions and sounds” (p. 194). Knight (2013) provides an in-depth review of iPad applications in music therapy. | ||
iPod Touch | As iPad | Similar to iPad but smaller in form factor and with enhanced security (not connected to a network) | ||
Apps | Pieces of software for use on tablet or smartphone devices | Wide variety, some free. Apps for Children with Special Needs (2018) is a website for finding specific apps aimed at children with special needs featuring music as well as other apps with reviews and videos. Offering the ability to tailor content and interaction to client requirements. | ||
Generic music production controllers (MPCs) | Trigger pads with velocity sensing | MIDI-compatible. Some come with own software instruments, requires technical knowledge to set up. Provides ability to preselect sounds representative of a wide variety of genres enables clients an accessible way to perform their cultural or musical identities. Allows therapists to offer diverse sound palates valuable for playing differing feelings or emotions during improvisation (Crooke & McFerran, 2019). Can also be used in composition to perform and record drum beats and melodic patterns (Crooke, 2018), or to trigger or launch a range of loops or samples, allowing for the playing of precomposed pieces and live remixing. Music Radar (2018) provide an overview of available MPCs. | ||
Music-based video games | Video game system using generic or bespoke to the game controllers | Familiar to users, enticing technology, easy setup, can potentially use existing resources. Blaine (2005) provided a review of alternate music-based video game controllers. Wikipedia (2017) provides a list of music-based video games. Use of the WiiMote in music therapy has been explored (Benveniste, Jouvelot, & Michel, 2008), and developments such as the Wiinstrument can be downloaded to utilize the myriad of data that the WiiMote produces (The Wiinstrument, 2018). Games available include Wii Music by Nintendo. | ||
Specific devices | Description | Details | ||
Skoog 2 | Wireless foam cube with five tactile pressure-sensitive pads. | Wireless, portable, easy to set up. Simple and intuitive. Own app and software to customize sensitivity and sound created, MIDI-compatible. Many resources for use in special education on website. Provides direct correlation between physical contact and sound produced, using virtual musical instruments, samples, or MIDI. Offering dynamic control over musical gestures (Skoogmusic, 2018). “The instrument does not simply trigger samples when pressed but uses sophisticated synthesis to dynamically manipulate the various instrument sounds though pressing, squeezing, rubbing, stroking, tilting or manipulating the Skoog” (Farrimond et al., 2011, p. 28). | ||
Makey Makey | Microcontroller board with six connectors that emulate QWERTY keyboard presses and mouse control | Turn everyday objects into touchpads, no software to install, fast setup. Can be connected to conductive objects such as fruit, conductive tape, pencil graphite, and clients touching each other in chains as a means to trigger sounds (Makey Makey, 2018). The process allows appropriation enabling clients to gain authorship of their instruments, and become an intrinsic part of their deployment (Hayes, 2016). | ||
AlphaSphere | Globe-shaped MIDI controller with 48 playable velocity-sensitive pads and own software. | AlphaSphere is a tangible controller, when connected to AlphaLive software/used with other MIDI software can be set up to trigger and manipulate sound and provide a unique modality of access that can be enticing to clients (Place, Lacey, & Mitchell, 2014). | ||
MIDICreator | Device to convert signals from electronic sensors into MIDI data. | Clients can control sounds with physical actions and gestures, can be used to detect simple body movements (Krout, 2014a). Can be connected to MIDI to be used with other synthesizers. Sensors available include pressure, distance, proximity, direction, etc. (Meckin & Bryan-Kinns, 2013). | ||
I-CubeX | Software and digitizer for creating systems with a variety of sensors available | Tools to capture the human actions and/or environmental variables and make these signals available to other equipment such as a computer or a musical instrument to trigger sound, music, video, graphics, animation, robotic movement, etc. MIDI data are used, transmitted via MIDI cable, USB cable, or Bluetooth wireless (I-CubeX, 2018). | ||
Kyub | Eleven feather touch keypads on five surfaces of a 3-inch wooden cube with accelerometer | Maker friendly, open-source DIY MIDI keyboard featuring capacitive sensing and accelerometer. Fully programmable, set scale, tweak note velocity curves, map different instruments to different pads to configure to musical taste (Kyub MIDI keyboard, 2018). User can also configure the way the instrument looks and sounds by designing their own interface. | ||
Suzuki QChord | Electronic instrument with on-board speaker and LCD display. | A device for composition, teaching, and therapy. Incorporating technology from a basic keyboard and electric guitar and combining both in a portable, easy-to-use way. Eighty-four different chord combinations, 100 instrument voices, orchestrated rhythms. Features three sections; a touch-sensitive 4 octave “strum plate,” a rhythm section, and chord button section. Each of these areas can be used independently or combined with a variety of sounds obtainable. Sounds are always in tune. Can be adapted to all ability levels. Can stimulate interaction, increase coordination, stimulate gross and fine motor skills, and increase self-expression. Pitch bend wheel for expression. Changeable song cartridges. Can connect to speaker or MIDI device (Suzuki Qchord, 2018). | ||
Yamaha DJX-iiB | Groove machine in a box-shaped desktop module form with scratch pad and fader. | Can select from 70 patterns, mute parts, add preset hits or fills, and shift key of the patterns playing and use effects. Provides opportunities for the clients to compose music by assembling loops, importing other songs or sections of songs, and/or recording her or his own music, offering accessible and dynamic means of expressing emotions (Whitehead-Pleaux et al., 2011) | ||
Musii | Soft inflatable self-contained portable unit that emits sound and illuminates when three inflated cones are touched, with wireless control | Simple to use, needs no extra equipment or training to operate. Abilities to make music and colour through touch and movement with expansive library of soundscapes (over 50) and innovative musical system. Cannot play out of time or out of tune. A number of people can play harmoniously as a group or an individual can become an orchestra. Tactile physical form that enables the user to see and feel the beam that they are interacting with. “Musii has been designed for developmental play in the SEN sector...The multi-sensory experience of playing Musii has many therapeutic possibilities including encouraging movement, development and awareness of proprioception, turn taking, cause and effect, creativity, expression and communication. It can be used for stimulation or for calming. The synchronised sound and lighting as well as the visual and tactile feedback of the inflatable enable a deeper understanding of the music you are making” (Musii Ltd, 2017) | ||
Reactable | Table with touch screen and moveable objects (Reactable Technology, 2017) | Objects interact with the table surface and each other to make music. Allows intuitive and collective creation of complex musical pieces, collaborative sharing space between users, promotes imitation games, increases visibility of actions, enables monitoring other participant’s work, aids in reducing solitary play sequences, facilitates associative play (Villafuerte, Markova, & Jorda, 2012) | ||
Tenori-On | Handheld screen with 16 × 16 grid of LED switches. Built-in speakers, dial and buttons control sound and beats per minute produced | Create, control, or perform musical material on visually rich touch-sensitive interface (Farrimond et al., 2011). Switches activated in different ways create music from 256 sounds. Engaging, motivating, sensory, well suited for improvisation, easy to use/hard to master. Combines visual and melodic sensory information, can stimulate cognition, memory, and perception. Can function as a rhythm machine with basic or complex rhythmic beats that can be looped or changed. Notes and melodic phrases can be added as well, creating up to 16 layers of sound (Clements-Cortes, 2014). Clements-Cortes (2014) provided an in-depth study of the Tenori-On in a clinical music therapy setting. | ||
Roland Handsonic | Device with 13 ultra-sensitive touch pads | Eight hundred and fifty ready-to-play sounds, or import custom sounds. Responsive and therefore not overly demanding to play, with easy adjustment of volume (Challis & Smith, 2012). Can be calibrated to client need in terms of sensitivity. Can connect to other MIDI devices and MIDI can be recorded from the device for analysis. | ||
Korg Kaossilator/Kaossilator Pro/Korg Mini Kaoss Pad | Audio effects unit with an X/Y touch screen. Mini Kaoss pad features on-board speaker and microphone | Positioning a finger-tip on the touch screen triggers specific sound programs. Ability to trigger individual notes or patterns of notes depending on the nature of the chosen sound and settings selected. Moving around the screen moves between notes within a predefined scale or changes the nature of the sound. Allows the results of actions to be sampled as repeating loops. Particularly effective for those with restricted hand and finger movement, being easy to interact with, to produce complex musical ideas and patterns (Challis & Smith, 2012). | ||
SoundHouse Special Access Kit (Banana Keyboard)—discontinued | Sixteen keys configured like an oversized piano with software component | Curved to suit the radial movement of an arm. Light touch activates music, sounds, or speech programmed into each key. Allows connection of up to eight switches for activation of keys on the keyboard (Sound House, 2017). Innovative, adaptable control surface that require musicians to press, squeeze, or strike them to create and control music through corresponding musical software (Farrimond et al., 2011). User friendly, easy to learn. Arrange combinations of sounds (MIDI sounds or wave files). Recording feature enables real-life performances to be saved and voice and CD segments to be recorded. Arrangements can be printed off and formatted. Variety of global settings cutting down on individually programming each key or switch. Aimed at fostering development of switch use, cause-and-effect, timing, choosing (Sound House, 2017). | ||
Numark Orbit | Wireless MIDI controller | Sixteen backlit customizable pads, four selectable banks to assign up to 64 cues, lighting transitions, video clips, samples, etc. Control wheel and on-board accelerometer. Can be configured and mapped to control other MIDI software, accelerometer for motion control. Comes with demo software that show how to use with tracks and effects (Numark.com, 2018). | ||
Mogees | Resonance contact microphone | Enables instruments to be created from any surface/object alongside configuring of the sounds created with dedicated iPhone/iPad app. Integrates with standard digital audio workstations via Audio Units or VST plug-in. Can be used to provide expressive instrument by using different areas of surface trigger different sound (Mogees, 2018), allowing for objects to be used that are motivating, familiar, or engaging to clients. | ||
Software- based | General technology | Description | Details | |
Generic digital audio workstations (Ableton Live/Audacity/Reaper/Cubase/Logic/Sonare by Cakewalk/Garageband) | Music recording/composition software | Uses preset or user determined settings via on-screen, or pull-down menus. Allow for recording, composing, playback, and creation of music. Some come with content such as instruments and samples available for instant use. Ability to use software instruments (VSTs), input microphones, or electronic instruments. Garageband is preinstalled on Mac computers with out-of-the-box samples and instruments available. Reaper is affordably priced, Ableton is prized for live performance. Audacity is freeware. Can export notation in some cases. Krout (2014a) provides a review on using software for music composition, arranging, notating, improvising, and sequencing. | ||
Specific devices | Description | Details | ||
Clarion | Software instrument | Allows user to change every element instrument including sound, notes, shape, position and colour of notes, and how those notes are played. Integrates with eye gaze systems, SmartNav and iPad. Package offered by Open Orchestras including repertoire, training resources, support, and an evaluation framework (Open Orchestras, 2018). | ||
Magix Music Maker | Digital audio workstation | Provides 425 sounds & loops, seven free Soundpools (1,927 sounds & loops) three software instruments, eight tracks, and eight effects. Can be used with smart boards. | ||
HyperScore | A graphical composition environment | Users draw strokes and lines to explore musical ideas. Graphical elements are mapped to musical structures, allowing users to shape musical progressions visually (Grierson & Kiefer, 2013; Machover, 2004). | ||
MIDIGrid | Music software | Uses mouse and keyboard movements within a grid to trigger notes, chords, sequences, or patterns of sound that can be played back and looped (Hunt & Kirk, 2003). | ||
Microsoft Songsmith | Music software | Generates musical accompaniment to match a singer’s voice using computer microphone input. Musical style and feel of song can be selected. Songs can be posted songs online, or used to create music videos | ||
2Simple music toolkit | Software applications | Six programs that introduce key musical concepts in an interactive way (2Simple, 2017). | ||
Empty-handed | Camera-based | Microsoft Kinect | RGB camera, depth sensor, and multi-array microphone | Provides full body 3D motion capture, voice and face recognition (KinectSEN, 2018). |
EyesWeb | Open development software platform | Real-time multimodal system and interface that has been used extensively in research. System supports input devices including motion capture systems, video cameras, game interfaces (e.g., Kinect, Wii), audio input, analog inputs (e.g., for physiological signals). Outputs include multichannel audio, video, analog devices, robotic platforms (Camurri et al., 2000). Website features information on development (http://www.infomus.org/eyesweb_ita.php) | ||
MotionComposer (available 2019) | System using two types of camera to detect movement | Allows gestures to be used to explore sound environments (Bergsland & Wechsler, 2016). | ||
AUMI | Free software application | Interface that enables the user to play sounds and musical phrases through movement and gestures captured via webcam (Larsen, Overholt, & Moeslund, 2016; Oliveros, Miller, Heyen, Siddall, & Hazard, 2011). | ||
VMI (Virtual Musical Instrument) | Free software | Uses web camera to detect motion. User virtually “touch” shapes on screen to trigger sounds. Requires no special equipment, Windows only based computer. Designed for use by therapists and educators, it is customizable according to the preferences and needs of the user, and can be used for specific therapy or educational goals (Virtual Musical Instrument, 2018). | ||
BigEye—discontinued still available for download | Macintosh only software program | Uses video information to convert into MIDI messages. Allows tracking of objects through space converting their parameters into MIDI in real-time (Legacy Products, 2018). | ||
Camera Theremin | Free test application | Create sound from movement using webcam (Camera Theremin, 2018). | ||
Musical Gestures Toolbox | Toolkit for experienced Max programmers. | Collection of modules and abstractions for the graphical programming environment Max 5 to enable extraction of movement data from video (Jensenius, Godoy, & Wanderley, 2005). | ||
Aerodrum | Package featuring drumsticks, software, feet markers, and camera | An air-drumming instrument. Runs on computer using a high-speed camera to track movements to trigger drums (Knight & Krout 2017). | ||
Break-beam | Beamz | Device featuring four breakable laser beams | Can be purchased as a professional package featuring software, songs, structured activities, therapy guides, and lesson plans | |
Soundbeam | Device featuring ultrasonic beams and switches (both wired or wireless) and a synthesizer unit | Device which uses sensor technology (up to four ultrasonic or eight switches) to translate body movement into music and sound using MIDI. New unit features touch screen interface, extensive library of sounds, recording and composing functions, training programs also available. Extensive support for use available through online resources (https://www.soundbeam.co.uk/). | ||
Theremin (Magee, 2006) | Moog Theremini—device featuring 32 wavetable preset sounds, and on-board speaker and sound engine | Can be used at any skill level, providing new ways to experiment with music and gestural control. Assistive pitch quantization allows each player to adjust the instrument’s level of playing difficulty. “At the maximum position, the Theremini will play every note in a selected scale perfectly, making it impossible to play a wrong note. As control is decreased, more expressive control of pitch and vibrato becomes possible. When set to minimum, the Theremini will perform as a traditional Theremin” (Theremini, 2018). Built-in tuner with real-time visual feedback of played notes and proximity (useful for correcting playing position). Store selected scale & root note, set and recall a specified playing range, and specify patch settings. Silent rehearsal available via headphone jack. Two line level audio outputs, a pitch CV output with selectable range, and a mini USB jack for MIDI I/O and connectivity (Theremini, 2018). | ||
Optimusic/OptiBeams (Knight and Krout 2017) | Interactive light beams | Package with interactive light units (the beams), laptop with OptiMusic software, USB controller box, user button box, reflective pads/bats. Interact with colorful beams of light (2, 4, 6, 8, or 12 beams), pass hand or body through the beams or use reflective wand to trigger audiovisual events in real-time. Comes with over 80 interactive settings. Package also comes with training (on-site or e-training) (OptiBeam, 2018). | ||
MidiGesture | Ultrasonic beam sensor | Sensor that plugs into the MIDICreator system (see MIDICreator). | ||
Leap Motion (Leap Motion, 2017) | Small device to track hand movement | Uses two monochromatic IR cameras and three infrared LEDs to track hand and finger movement above device. Dickens, Greenhalgh, and Koleva (2017) provide an in-depth description of research conducted using the Leap Motion for music performance with users with complex disabilities. | ||
Brain computer interface (BCI) | Brainfingers (Brainfingers, 2018) | Headband fitted with sensors | “Detects electrical signals from facial muscles, eye movement and brain waves. Brainfingers does not directly target music creation, as it can solve many tasks such as simple clicking, to complex combinations of controls. It is software that converts all the sensor input data into controls termed Brainfingers. This software is useful for a broad range of users, especially people with severe disabilities” (Larsen et al., 2016, p. 329). Controls most AAC software, educational software, and video games. | |
Eye gaze | EyeMusic (Larsen et al., 2016)—legacy files available online, requires technical skill to install (EyeMusic, 2018) | Software utilizing generic eye gaze equipment | System that uses eye movements as input to electronic music compositions. Can be used with established composition software allowing prerecorded eye movement data to control musical compositions (Hornof & Sato, 2004). | |
E-Scape (Anderson, 2018) | Software utilizing generic eye gaze equipment (or switches) | Music software specifically designed to be used by people with disabilities to create or perform music. System operates via large guided pop-up menus controllable by one or more switches, mouse, keyboard, eye gaze, or MIDI controllers or sensors. “At every stage, E-Scape asks the user what they want to do and offers a range of options depending upon which level of complexity the user has chosen to work at” (Farrimond et al., 2011, p. 23). Two modes of operation—composition and performance. Can output MIDI data (Farrimond et al., 2011). | ||
EyeHarp | Free software utilizing generic eye gaze equipment | Gaze-controlled or head-controlled music interface to help users learn and play music. Vamvakousis and Ramirez (2016) provide a comprehensive article on the development of the EyeHarp. The website theeyeharp.org also provides a wealth of information about the project (The EyeHarp, 2018). | ||
Eye Play Music | Free software utilizing generic eye gaze equipment | Trigger notes from a range of instruments available with adjustable note length and transposition. Create own scales. Load and save settings. Website features resource for use (MBMM, 2017). | ||
Breath | Jamboxx | MIDI controller device | Hands-free electronic, breath-powered instrument. Uses sip and puff to determine amplitude of note. Software included to configure device and on-board modulation wheel, button, and jack socket to allow switch connectivity (Jamboxx, 2018). | |
Yamaha WX5, WX11 | MIDI controller devices | Breath powered MIDI controller that allows for one-handed playing. WX5 features MIDI output; however, WX11 requires an additional MIDI connection box (MBMM, 2017). | ||
Magic Flute | Stand-alone instrument | Self-contained instrument with built-in tone generating hardware. Plugs into external speakers. Two separate parts, the flute and control module with display. The flute being the remote control for the control module. Musicians can select different sounds or access the user settings without the help of another person (MBMM, 2017). The volume is controlled by blowing in a mouthpiece and the pitch by moving the mouthpiece up/down with the mouth (Vamvakousis & Ramirez, 2016). “The instrument reduces the physical and cognitive challenges inherent within conventional wind instruments. One musician, with very limited lung volume, is nonetheless able to realize the full dynamic range of the instrument” (Farrimond et al., 2011, p. 29). | ||
Interaction modality | Research only | Description | Details | |
Touch | General research | |||
Collaborative interfaces review of literature (Blaine and Fels, 2003) | Review paper of interfaces used for collaborative music-making | Comprehensive review of context and design of a number musical experiences for novices | ||
Specific research | ||||
Musicking Tangibles—RHYME Project | Tangible interfaces consisting of interactive digital “furniture” | Examining the development and benefits of using interactive digital music furniture for disabled children by using two co-creative tangible instruments. ORFI-26 soft pyramid-shaped, pillow-like modules, in three different sizes (30–90 cm) featuring bend sensors and lights, the units can communicate wirelessly with each other. Wave Carpet-7-branched, wired, interactive, soft, dark carpet with orange velvet tips that glow. Central arm contains microphone, two arms contain accelerometers that change the recorded sound. Two arms contain bend sensors that create rhythmical background music. One arm contains a web camera. Contain five software programs, offering different music and dynamic graphics to show via projector embedded in one arm, or via full wall projection. Center contains two speakers and strong vibrator in. Contains IR-sensors allowing interaction with RGB LED lights (Andersson & Cappelen, 2013). | ||
NoiseBear (Grierson and Kiefer, 2013) | Malleable controller | Development of robust, wireless, malleable controller for children with cognitive or physical disabilities | ||
Bean | Gesturally controlled digital instrument | Device designed around a Wii nunchuck controller for use in a music therapy setting (Kirwan, Overholt, & Erkut, 2015) | ||
MAWii | Digital musical instrument using generic WiiMote controller | Research exploring WiiMotes as virtual instruments for children with behavioral disorders (Benveniste et al., 2008). | ||
WamBam | Self-contained electronic hand drum | Created using piezo sensors. Paper describes development and testing of device used in for music therapy sessions with severely intellectually disabled clients (Jense & Leeuw, 2015). | ||
TouchTone | Digital musical instrument | Device featuring touch-sensitive pads designed to develop musical ability, bimanual coordination and increase social participation of children with hemiplegia (Bhat, 2010). | ||
Computer-Assisted Music Therapy | Augmented reality software | Details system developed with Augmented Reality techniques allowing music composition and creation activities using sound and colour, via cards (Correa, Ficheman, do Nascimento, & de Deus Lopes, 2009). | ||
SenseEgg | Wireless controller device | Development of a handheld egg-shaped device featuring seven on-board sensors (button, slider, accelerometer, wind sensor, ultrasonic distance sensor) and a suite of software patches aimed at for musical exploration and teaching. Featured a component that allowed control of settings via an iPad (Blatherwick & Cobb, 2015). | ||
Software-based | DIYSE software | Software that utilizes Guitar Hero controllers | Details development of software allowing connection of existing controllers (Guitar Hero & WiiMote, etc.) to compose and restore music tracks, and design mapping strategies between interface and played sounds, for people who with intellectual learning disabilities (Luhtala, Kymäläinen, & Plomp, 2011). | |
Empty-handed | Camera-based | Movement to Music-MTM | Web camera and software system | Developed to address the need for affordable home-based musical play system, incorporating automatic movement recognition technology that is noncontact and noninvasive (Tam, Schwellnus, Ceilidh, Hamdani, Lamont, & Chau, 2007). |
Break-beam | Benemic/Octonic | Stand-alone instrument | Device with array of eight low-cost infrared distance sensors. Enabling triggering and manipulation of sounds using MIDI messages (Challis, 2011). | |
Eye gaze | Eye conductor | Software uses webcam | Software-based musical interface to play music through eye movements and facial gesture using eye tracker equipment and webcam. Detects gaze and selected facial movements enabling playing of instruments, beat building, sequencing melodies, or triggering musical effects (Refsgaard, 2018). |
Note. MIDI, musical instrument digital interface.
Controlling Sound With Technology
Paine and Drummond (2009) suggest there are two distinct approaches to computer-assisted music: “control of predetermined sequences of sounds (such as the triggering of sound samples) or creation of sounds in real-time by the manipulation of software synthesis variables” (p. 2). Technology offers the ability to control and trigger sound in different ways that extend past that of acoustic instruments. Technology can also provide responses to interaction in ways that acoustic instruments cannot. It can offer physical and/or cognitive support, and scaffold capability to give users access in ways traditional instruments do not allow. Swingler (1998) suggests that few children have the physical coordination or control necessary for traditional performance. As such, technology can help to shift beyond traditional musical qualities toward a new and developing musical aesthetic, one enabled by the introduction of electricity to musical activity. He suggested that this allows the opening up of many musical doors so all can enjoy being expressive with sound: “Many techniques can be made easily available to virtually all kids through technology” (Swingler, 1998, p. 5). Through technology, small motions can lead to sound production and engagement. For example, with even something as simple as a microphone there are great opportunities for utilizing feedback and amplification to allow the “tiniest voice and smallest nuances to be enhanced and extended” (Ellis & Van Leeuwen, 2000, p. 8).
Music technology can therefore help to:
Transduce movement and gestures into musical expression (Hunt et al., 2004)
Make it possible for a client to realize a creative idea regardless of implementation or user and to give the opportunity for an aesthetic experience (Misje, 2013)
Allow people to lose themselves in artistic expression (with a quality of interaction so high that they are not aware they are using technology) (Hunt, Kirk, Abbotson, & Abbotson, 2000)
Give initialization opportunities to usually passive users enabling the concept of selfhood, which can be inhibited for individuals with profound and multiple learning disabilities (PMLD)
Provide, sometimes for the first time (Swingler, 1998), that “make something happen!” moment as described by Ellis (1997), which is a foundational experience of learning
These simple but crucial experiences may help users to encounter and develop communication skills through sound. This control can lead to changes in behavior patterns beyond the environment of a therapy session with individuals becoming more self-aware and interactive outside of the sessions, more tolerant, and with a growing awareness of others (Swingler, 1998). Hunt and colleagues (2004) suggested that technology offers access to real-time sound control to those with limited movement, along with new sound worlds and timbres (Ellis & Van Leeuwen, 2000; Hunt et al., 2000; Kirk, Hunt, Hildred, Neighbour, & North, 2002; Misje, 2013). Computer music can be intriguing, particularly to young people, who may find traditional instruments, which are often associated with strict disciplined methods, “off-putting” (Hunt et al., 2004).
Technology can also offer the sense of control and autonomy (Crowe & Rio, 2004) removing the need for prerequisite skills for learning to occur (Nagler, 2011). This can help clients reach peak experiences that would be difficult using traditional instruments (Misje, 2013). Technology can offer the ability to readily create music, learn to play an electric instrument, use computer programs, and/or to write and record. These activities can be condensed into a small amount of equipment, by offering the potential for many instruments to be accessed from one set up. This provides a “blank sheet” (Kirk et al., 2002) onto which individual instruments can be built for different uses/users. “This aural richness and variety provides the internal motivation.…. In addition, the technology also provides physical access for [people with disabilities]” (Ellis, 1997, p. 176). In cases where affordability is an issue, technology could be beneficial, given how expensive acoustic instruments can be. “It is possible to create sounds with as much musical interest as familiar orchestra instruments but which could not be produced by a known instrument. A new dimension for interaction can then be opened up, offering radical possibilities for performance” (Kirk et al., 2002, p. 1023) that allow for and support unconventional playing (Ellis & Van Leeuwen, 2000). DMIs do not need to sound or play like conventional instruments, and they can be created to be operated by any part of the anatomy with no right or wrong technique, only that which is appropriate to the individual (Ellis & Van Leeuwen, 2000).
Music Technology Used in Music Therapy Practice
Music technology offers up new possibilities for exploration within music as part of the larger framework of music therapy (Misje, 2013). It has been used for many music-making activities both as an active music technique (singing, music composition, instrument playing) and as receptive intervention such as listening. Technology has also enabled the exploration of activities such as songwriting, recording, improvisation, listening, recreative, and multimedia project development as well as studying, learning, and composing and serving the needs of individuals with disabilities both in medical practice and research (Crowe & Rio, 2004; Viega, 2016). Music technology in music therapy has been used to address identity development (Magee, 2006); express thoughts and feelings (Whitehead-Pleaux et al., 2011); promote empowerment (Burland & Magee, 2013; Cappelen & Andersson, 2013); construct meaning (McDowall, 2008); and develop agency (Kruger, 2007). The development of on-task behavior, concentration, cooperation, communication, self-expression, problem solving, and decision-making have all been shown to be supported through the use of technology (Crowe & Rio, 2004). Technology can be particularly useful for those with short attention spans as it can be set up to instantaneously provide a relevant and enticing response, leading to enhanced focus and the potential to transcend disability (Swingler, 1998). Technology can also be used to provide individual control by community participation (Misje, 2013). This can be seen in the work of Andersson and Cappelen (2013), and through the RHYME project, using tangible interfaces for musicking (Small, 2011).
Incorporating Music Technology Into Practice
Nagler (2011) suggests the next steps for the inclusion of technology (specifically digital handheld music-making devices) in music therapy clinical practice are:
1. The creation and development of applications that allow for music therapists to use musical methods analogous with practices achieved using traditional instruments, thus allowing for “demonstration of patient progress toward specific goal attainment” (p. 198)
2. The development of accepted, common guidelines from experts in the field with best practices needed to dictate methods. Nagler (2011) suggests that the development and sharing of technology could be spurred on by the use of Creative Commons licensing and open-source networks. This includes the need to create a taxonomy of understanding (to codify the pitfalls, methods, and potentials) incorporating the vocabulary, structure, and architecture of technology (specifically of handheld music devices) into clinical practice
Farrimond and colleagues (2011) suggest simplifying the complexity of available technology by distinguishing between access needs and learning needs to aid in finding technology that is suitable for providing musical possibilities for clients. This can then lead to an emphasis on the creative preferences and needs of the individual. Magee and Burland (2008) echoed this by advising “recommendations from allied fields advise that access to music-making for an individual with disabilities needs to start with examining the variance of the individual’s abilities, the type of input required to achieve a task, and the possible mappings between the two” (p. 126).
Further, developments in music education such as the Sounds of Intent framework (Vogiatzoglou, Ockelford, Welch, & Himonides, 2011) seek to provide “evidence-based guidance on appropriate music pedagogy for all children in special education (thus informing policy and practice)” (Welch, Ockelford, Zimmermann, Himonides, & Wilde, 2015, p. 3). The resources they provide are aimed at mapping the musical development of children and young people in special education settings.
Finally, a key issue for designers of new technology to consider is the “musicality, usability, accessibility and affordability” of technology (Challis, 2011, p. 6). As such, there are several design considerations that should be reviewed when designing technology specifically for the special education setting (Ward et al., 2017). These considerations aim to maximize the potential for new developments to be incorporated into practice, make technological tools less daunting to everyday users, and foster creativity and communication among users.
Conclusion
While it is clear that incorporating music technology into clinical practice to enable active music-making has a myriad of potential benefits, it is also clear that the ever-changing landscape of technology can be overwhelming This can create gaps between the developer, clinician, and client. This ever-changing landscape may be particularly overwhelming for music therapists not already steeped in technology, as these systems often consist of several layers of technologies that require technical skill to combine. Practitioners may find it difficult to keep up with changes in technology, and figure out how to combine and integrate them into their practice. Still, despite these technical and financial challenges, the utilization of technology provides unique access to music-making for those that could not access traditional instruments or repertoire. Alternate controllers, in particular, provide a means to explore new ways of utilizing an individual’s physical and learning abilities to provide meaningful and motivating musical experiences. Using music technology in this manner, on its own or alongside traditional instruments, requires a different approach to integration, repertoire, and skill set of the users. This approach must take into account the type of technology, how it will be used, and also the intended outcome.
The potential in using technology is evident from the developments presented in this article; this potential, however, must be discussed, shared, and best practices developed. There is still a growing need for a reexamination of the content of education and training that places technology at the forefront of music-making scenarios. The combination of this and new partnerships with those already steeped in technology would lead to a more established use of technology and would thereby build a new generation of clinicians.
Funding
Funding for the Centre for Digital Entertainment granted by the EPSRC (code: EP/L016540/1).
Acknowledgments
With thanks to the Three Ways School, Bath, UK, Luke Woodbury of DotLib. Dr. Tom Davis, and Dr. Ann Bevan of Bournemouth University for supervisory support.
Asha Ward is a research engineer currently undertaking an engineering doctorate (EngD). Her work focuses on making accessible musical instruments and interfaces for those with complex needs.
Dr. Tom Davis is a lecturer in Creative Technology with an interest in instrument building and improvisation.
Dr. Ann Bevan has been a registered nurse for over 40 years and now teaches children’s nursing, she is an action researcher.