Drifting, one of the recent works of my Brushing Project series, is a sonic performance piece incorporating a DIY made Brushing Interface, and a performer in 8-channel audio. The Brushing Interface is a DIY-made interface, which is made with over 200 self-made force sensing resistor (FSR) sensors and 8 piezos that are connected to a custom made printed circuit boards (PCB) and a Teensy microcontroller, that can detect the position/pressure/brushing sound in realtime. This piece aims to deliver the nuanced and sophisticated materiality of a brush through an intersection between sonic timbre and brushing gestures by creating a deep timbre space on a finite two dimensional surface. The deep timbre space is constructed by timbre transfer from the brushing sound inputs and other realtime synthesis, which are mapped with brushing positions and machine-learning-based gesture detections (rbfi for position-based mapping and MuBu for machine-learning-based gesture detection) (Choi 2022). This paper will focus on the compositional process and methodology of this project, which demonstrates a way to integrate bodily experimentation and the building process of electronic/digital medium.
In the electronic music literature, there are viewpoints that the creative process and the implementation of an electronic medium are separate. This division between artistic and technical work has been pretty prominent that major computer music institutes such as IRCAM (Institut de Resherche et Coordination Acoustique/Musique) in Paris and CSC (Centro di Sonologia Computazionale) in Padova, applied to their working methodology (Zattra 2017b). Behind this approach, there is a belief that computer music practice involves the division between the "poietic and aesthetic side," (Zattra 2017a, 88) which the former is technical implementation and the latter is composition. However, by incorporating making and embodied approach, this strict binary can be dismantled and provide an integrated approach in computer/electronic music composition. Through the creative process of making Drifting, this paper will present how it can be realized as a genuine piece of work. In short, Drifting involves a constant iteration between the making process of the interface and embodied experimentation of the interface in a performative way. Also, the concept of affordance will be discussed to illustrate this discussion.
The act of making is not simply an implementation of a pre-configured plan, but rather an improvisation between the hand, body, mind, and material. In this process, the maker's movement, gestures, and even their own life get involved "joining with and following the forces and flows that bring his[their] work to fruition." (Ingold 2013, 31) Therefore, the construction of the instrument is a flow of process in which the maker incorporates their artistic vision/musicality/belief/personal life, where these qualities dynamically engage with the material through the hand and body. As Tim Ingold described in his book, Making: Anthropology, Archaeology, Art and Architecture (Ingold 2013), the final outcome is never fully revealed until its end, but it comes into shape as the interaction proceeds. Also, it is important to mention that the maker themselves also interacts with and gets affected by the larger social/cultural flux surrounding them. The instrument produced from this process reflects a deeply layered assemblage in various degrees.
In this regard, the made object does not exist solely on its explicit functionality. Therefore, framing a musical interface it based on a static definition or fitting it in a category imposes a risk of missing the various musicking involved and its creative potential. Therefore, to conceptualize a self-made musical interface in a way that captures the dynamic musicking and artistic potential innate to it, while preserving its presence as a design object. The concept of affordance was first proposed by James J. Gibson in the field of psychology. Gibson advocated direct perception over representational/inference perception, arguing that "subjects do not need existing knowledge of a situation to disambiguate but instead can perceive directly from the environment and act based on direct perception." (Davis 2020, 27) In this context, Gibson defined affordance as "what offers the animal, what it provides or furnishes, either for good or ill." (Gibson 1979, 127) After about a decade, Donald A. Norman expanded this concept into design studies and human-computer interaction (HCI). (Davis 2020, 29) Norman argued that "objects should be designed in ways that guide users' perceptions and thus guide actions."(Davis 2020, 29–30; Norman 1988) Norman defined affordance as "the perceived and actual properties of the thing, primarily those fundamental that determine just how the thing could possibly be used." (Norman 1988, 9) Kenya Hara very well articulated this concept through the product design work by Naoto Fukasawa.
"Consider the CD player Fukasawa designed. It looks almost exactly like a wall mounted fan. The CD is inserted in the center. If you pull the cord that on a fan would be the on-off cord, the disc begins to spin, just like a fan. We know it's a CD player, but our previous experiences and memories of fans, entrenched in our minds, cause us to prepare ourselves for that object."(Hara 2007, 45)
Moreover, affordance is not only limited to "enabling users' possibilities and constraining their choices" (Nagy and Neff 2015, 4) through design by utilizing social/innate relations into the design, and it can also be interpreted more expansively. Peter Nagy and Gina Neff had coined the term imagined affordance, which included the "users' perception, attitudes, and expectations" to the division of materiality/functionality of technology and the work by designers (5). As affordance takes "emotional, affective, and non-rational" (6) into account, its inter-relation becomes more complex than the simple binary between social and technical, which makes the concept more applicable to an artistic context. In this context, designing and building interfaces in an electronic/computer music project is not simply for realizing a technical function or directing a performer's action in a certain way. It is creating a mediated object that actively interacts between the performer's social and innate network, and evokes their artistic imagination.
The process of creating Drifting involved constant feedback between the making process and bodily experimentation. Related to this, David Wessel suggested a creative methodology for utilizing bodily interaction in a computer music performance. His model was based on a system consisting of "rich and precise gestural interfaces, connectivity devices, real-time gesture analysis and mapping software, and richly controlled sound synthesis and processing." (Wessel 2006, 93) To create an expressive performance utilizing this system, Wessel emphasized babbling, which is a non-intentional bodily interaction with the interface. This term came from how human infants develop their speech through "sequences of commands sent to the motor system in order to learn the relationship between these commands and the sonic result produced by the vocal apparatus."(94) Babbling is highly effective because its "auditory feedback provides information about the relationship between a gesture and a resulting sound,"(94) which enables artists to expand their performative expression through an interface.
Moreover, this notion of babbling can be expanded beyond the mechanical "sensory-motor engagement"(94) suggested by Wessel. As discussed above, the affordance of a musical interface encompasses innate qualities, social-cultural backgrounds, and imaginations of the user and the maker, which makes the interface become a "space of potential" through its artistic affordances, where it inhabits the "moments of emergent perception or imagination, acts of identification of figure, action process, concept, relationship, or technique" for various participants in a nonlinear way(Impett 2021, 121). Therefore, the making process of an interface is equivalent to creating a "space of potential" with an expansive aesthetic affordance, and embodied experimentation with babbling realizes these artistic potentials from the interface.
In this process, the body takes an expansive role in the making process and the embodied experimentation, and reconfigures the relationship between the mind/body/technology. Susan Kozel very well articulated this through her writing, referencing Maurice Merleau-Ponty's phenomenology. Kozel argued that viewing the human body as a "dynamic process" enables the reconfiguration of the human-computer interaction in a non-Cartesian way, rejecting the hierarchy and division of mind and body(Kozel 2008, 33). Kozel writes that "our bodies extend beyond ourselves through the operation of our senses and as such the boundary of the body, skin, is not a boundary at all. We are porous beings, and we are part of flesh as well as being flesh."(33)
In this regard, the compositional process occurs through a constant feedback loop between the making and embodied experimentation through the phenomenological human body. The interface built by the artist evokes artistic affordance that has aesthetic potentiality, and the embodied experimentation explores and realizes the artistic potentiality of the interface, while also inspiring new design ideas for the interface itself.
What was intriguing about the brush was its nuanced and sophisticated materiality as a writing/drawing instrument. Every stroke is unique and small details in the brushing gesture lead to different textures of the drawn outcome. This interest in gesture and texture had transformed into an interest in the relationship between gesture and timbre. There are three main sound textures used in Drifting. This paper will not go into too many details about these realtime audio synthesis techniques but will describe them at a higher level for illustration.
Each sound texture was mapped with the rbfi(radial basis function interpolator) Max object, developed from the Center for New Music and Audio Technologies (CNMAT), which enables multi-dimensional mapping in a two-dimensional plane. This is implemented by calculating the weight of the data point through the "power function of the distance from the control cursor."(Freed et al. 2010, 343) For example, the figure on the left is the image of the rbfi object actually used in the Max patch for Drifting, specifically mapped with the convoluted brushing sound texture. If a pressure beyond a certain threshold is detected from the Brushing Interface at a certain location, the control point of the rbfi object glides to that location. The gliding speed is relative to the amount of pressure detected. Each data point, ones that start with /p, refers to the level of brushing sound convolved with different audio samples, with the /p0 data point representing the unprocessed one. Based on the position of the control point, the X sign on the figure, the mix of 10 different brushing sound textures is manipulated in realtime.
Moreover, the transition between the three main sound textures, which are convolved brushing sounds, granulated wood sounds, and concatenative synthesis of breathing sounds, were mediated by MuBu, developed from IRCAM, which provides a machine learning-based gesture recognition functionality. Integrated with inputs coming from over 200 FSR sensors on the Brushing Interface, it can detect and classify unique brushing activities, which controls the transition/juxtaposition/overlapping between those different sound textures.
This approach enables every sonic parameter and the entire structure of the performance to be manipulated solely from brushing gestures on the Brushing Interface without any other types of controllers such as MIDI foot pedals, sliders, and knobs. At the same time, its architecture allows great versatility and richness only with brushing gestures. This provides a strong aesthetic affordance driven by brushing gestures, which have been formed by the constant iteration between babbling and making based on it. The notion of a deep timbre space has also aroused from this process. Multiple layers of sound textures reside under the Brushing Interface that are evoked through various brushing gestures forming a multi-dimensional sonic space from a two-dimensional Brushing Interface, which is also coherent under the gesture and the materiality of the brush.
In this paper, the integration between the making of technological medium and embodied experimentation was discussed through my work Drifting. This shows that technical implementation and embodied performance are not two separate dichotomies but rather are entangled in technology-mediated practices. The formulation of affordance through making and realization of its imaginaries through the body inspires and references each other simultaneously. This entangled intra-relation between the two provides a vibrant and organic compositional methodology that utilizes technological medium.
For future plans, I am currently focused on pursuing an improvisational direction, which very well fits the dynamic nature of this creative process. Here, improvisation is not only limited to the performance itself, but also includes the broader practice of the Brushing Project, the treatment of the technology, creative practice, planning and formulation of the performance, and the performative gestures involved in the creative process and the performance itself. This broad notion of improvisation resonates with the creative and academic discourse from the Critical Improvisation Studies, which treats "improvisation as a way of life ... finding that social and political formations themselves improvise and that improvisation not only enacts such formations directly but also is fundamentally constitutive of them."(Lewis and Piekut 2016, 13)
Choi, Jaehoon. 2022. Jaehoon Choi’s Website, Drifting page. https://jsonchoi.io/drifting.html; Accessed: 2023-04-13.
Davis, Jenny L. 2020. How artifacts afford: The power and politics of everyday things. MIT Press.
Freed, Adrian, John MacCallum, Andrew Schmeder, and David Wessel. 2010. “Visualizations and Interaction Strategies for Hybridization Interfaces.” In Proceedings of the International Conference on New Interfaces for Musical Expression, 343–347.
Gibson, James J. 1979. The ecological approach to visual perception. Boston: Houghton Mifflin.
Hara, Kenya. 2007. Designing design. Lars Müller Publishers.
Impett, Jonathan. 2021. “Dissociation and Interference in Composers’ Stories about Music.” In Sound Work: Composition as Critical Technical Practice, 117–135. Leuven University Press.
Ingold, Tim. 2013. Making : anthropology, archaeology, art and architecture. Routledge.
Kozel, Susan. 2008. Closer: Performance, technologies, phenomenology. MIT Press.
Lewis, George, and Benjamin Piekut. 2016. “Introduction: On critical improvisation studies,” In The Oxford handbook of critical improvisation studies, edited by George Lewis and Benjamin Piekut, vol. 1. Oxford University Press.
Nagy, Peter, and Gina Neff. 2015. “Imagined affordance: Reconstructing a keyword for communication theory.” Social Media+ Society 1 (2): 1–9.
Norman, Donald A. 1988. The psychology of everyday things. Cambridge, MA: MIT Press.
Wessel, David. 2006. “An enactive approach to computer music performance.” Edited by Yann Orlarey. Le Feedback dans la Creation Musical, 93–98.
Zattra, Laura. 2017a. “Alvise Vidolin interviewed by Laura Zattra: The role of the computer music designers in composition and performance.” In Live Electronic Music: Composition, Performance, Study, edited by Friedemann Sallis, Valentina Bertolani, Jan Burle, and Laura Zattra, 83–100. Routledge.
Zattra, Laura. 2017b. “Collaborating on composition: The role of the musical assistant at IRCAM, CCRMA and CSC.” In Live Electronic Music: Composition, Performance, Study, edited by Friedemann Sallis, Valentina Bertolani, Jan Burle, and Laura Zattra, 59–80. Routledge.
Your browser does not meet the minimum requirements to view this website. The browsers listed below are compatible. If you do not have any of these browsers, click on the icon to download the desired browser.
