Creatively Processing the Pandemic

Name: Orpheus Instituut
Price range: $

A Multidisciplinary Audio Essay

Article by Diana Chester, Benjamin Carey and Liam Bray

Creatively Processing the Pandemic looks at the collaborative process of sonifying data from the COVID-19 pandemic, through a narrative that weaves together conversations between artists, sonic compositions, discussions of the creative process, and a recorded soundtrack of newsreel from the pandemic.
... +

Share this article

go to article

tags: sonification, audio essay, electronic music, sound studies, cybernetics, sound synthesis, practice-based research, COVID-19

This piece is a contribution to the newly formed genre of the audio essay where multiple voices, non-human actors and methods come together as a form of sonic argumentation. This audio essay is born out of a grant funded research project entitled ‘NanoResonance’, a multidisciplinary project where artists and scientists collaborate to find ways of expressing scientific data creatively, and the outcomes of this work. The present essay harnesses the creative practices of the three authors to discover individual and collective strategies for using scientific data to inspire novel compositional ideas. We were interested in discovering, through practice, how such datasets can inform our compositional thinking, and what that might mean for future interdisciplinary practices. The research explores the process of creative practice, the realities of collaborating over distance, the technological approaches to this work, and the often lesser-discussed process of navigating through the expressive creative process as an artist. The audio essay is heavily layered and incorporates three distinct creative responses to COVID-19 data from the three artists, interwoven with candid discussions that took place during Zoom meetings. By considering the role of data, sonification, collaboration, and our personal creative processes, this audio essay offers a perspective on how we as artists grapple with data as it is contextualised in the world around us. The audio essay is interspersed with textual reflections by the authors, providing further context to the audio essay itself. The research highlighted within was made possible through a Catalyst grant from Sydney Nano, the University of Sydney’s multidisciplinary Nanoscience and technology institute.

Opening

00:00 / 00:00

The thing about the use of the COVID-19 data is that it is so situated in where we are right now, it wasn't just a pragmatic choice. It also had resonance and there was a reason for that resonance. And I think as well, that's a useful thing to think of when we're talking about any data, about why this particular data set is useful. What are we trying to communicate if we are doing a direct sonification? And yeah, in an audio essay like that, I'm so down with that Diana, using bits and pieces of that. I mean, this is starting to feel quite ready radiophonic in a way.

Benjamin Carey

Introduction:

Art made from COVID-19 data sets is the focus of this audio essay. The audio essay is a form of sonic argumentation and presentation that leans into the dynamic layering power of sound as a tool to communicate and articulate research, in this case the NanoResonance project (Groth and Samson, 2016). This piece is a contribution to this newly formed genre, where multiple voices, non-human actors and diverse methodologies come together to present a coherent sonic argument. This audio essay brings together sonified creative responses using COVID-19 data from Johns Hopkins University (Dong et al., 2019), Zoom collaboration discussions, and short essays by the three contributing artists about the creative practice that each used in their sonification. In our work, the COVID-19 data is an actor that is used to influence and frame artistic decisions, both technically and conceptually. The data is used in multiple ways through the artists’ individual practices, both in its raw form and in various manipulations. We have chosen to use the audio paper format to communicate the ideas of this research because of its unique affordances for this type of multidisciplinary artistic-research.

In “Audio Papers - a manifesto”, Groth and Samson propose a methodological guide to the audio paper. In it, they state “the audio paper depends on diverse sound environments and human practices in its attempt to assemble aspects, narratives, phenomena and sensations of the world.” (Groth and Samson, 2016). True to this method, our audio essay places field recordings of newsreels, creative sonifications of COVID-19 data, and human voices in conversation with one another to explore the collaborative process of developing creative works of this data and the technologically mediated environments we as artists engage with for our work, and have relied on for our meetings and development of this piece. Sound as a medium for expression allows for unique ideas and characteristics inherent in a piece to be transparently shared with a listener. Differently from text, qualities of the human voice allow us to convey a mood, emotion, and context without description but through tone and temporality.

In the audio essay, each sonic work is contextualised with reference to each artist’s use of the data, as well as their technical and artistic process. Whilst these verbal additions illuminate this useful background of each work, the presentation of the sonic works themselves, with their varying use of gesture, dynamic and emotional qualities, communicate each artists’ response to the data in ways that language cannot. The human voice mixed with the expressive nature of the three creative sonifications allows for the blending of feelings and sensations alongside the semantics of language and sound. (Groth and Samson, 2016).

The NanoResonance research project has been a year and a half in the making and through this time, almost entirely during the pandemic, the research team has relied almost exclusively on remote collaboration. A primary building block of the project is the development of two bespoke software programs designed to parse, and then synthesise scientific data sets into MIDI and control voltage information with a number of controllable parameters. Developed in Cycling '74's 'Max' (Cycling '74, 2022), we have used these programs to sonically articulate, process, and conceptualise the COVID-19 data of positive recorded COVID-19 cases by country and tied to specific dates. Each artist has created a sonification through their individual lens and methodological process, resulting in three distinctive sonifications tied to our individual somatic experiences of the pandemic. Research questions have guided the development of sonic argument within the academic frameworks of each contributor, with sound as our unified presentation format (Groth and Samson, 2016).

As you will hear below, each sonification takes parsed data parameters into account but leans into unique specialisations from modular synthesis to musique concrète techniques to articulate our experiences and notions of the pandemic while remaining true to the data. These sonifications are arguably an extension of our creative and intellectual selves, as well as our thinking about the interconnected nature of the data that expresses this world event, our own somatic and emotional intuitive processes navigating the pandemic, and of course, our personal experiences of meaning making from that data. The audio essay is roughly fifteen minutes in length and broken up here in chunks that correlate to each artist's reflections on the process of creatively responding to scientific data through sound.

Diana Chester

00:00 / 00:00

Dr Diana Chester:

I am a sound studies scholar and artist and I work heavily with recorded sound. I typically do field work as a part of my research and often record all the sounds that I use in my compositional works. For the NanoResonance project I wanted to create an interwoven narrative of the recorded environments of place alongside the sonification of MIDI taken directly from the Johns Hopkins University COVID-19 dataset (Dong et al., 2019). This desire for sounds of place, manifested as a soundscape of newsreel recordings, which I collected online and carefully curated to highlight details of each country’s journey with the pandemic. In some cases, this was bringing bits and pieces of different languages, different perspectives on how the pandemic was handled and the local approach to speaking about the unfolding of the global pandemic from the early days. The process of using MIDI data alone did not feel that it resonated with my own practice and intuitive approach, nor did I feel It could express the geographic scope of the work, which I found to be at the heart of my own interests.

In the audio essay, I used the parsed datasets of the COVID-19 data from China, Japan, Singapore, Taiwan, South Korea, the United States of America, Italy, South Africa, Mexico, Australia, the United Kingdom, Germany, the United Arab Emirates, Brazil, and New Zealand. I also used sound from newsreels of the pandemic from each of these countries' local television networks or radio. I curated the sections of the news in a somewhat chronological order based loosely on the beginning of the pandemic and where and how news started being revealed around the world. I skewed toward using news coverage from these countries in English where possible (though I did use clips in Mandarin and German) largely because I needed to understand the sound clips I was editing and including, but also I was designing the piece for an English-speaking audience. I also included news coverage of the Black Lives Matter movement protests from the USA and Australia, as I felt there were aspects of these movements that were socially and politically intertwined with the pandemic. The coming together of large groups of people during lockdowns, was in itself a political statement about free speech, protest, and human rights, and as an American living in Australia this felt like a critical element of my own awareness and experience of this pandemic period.

Next, I worked with the MIDI data from these same countries, and wove a narrative using different sounds on my Haken Continuum, (Haken Audio, 2022) as a way of sonifying the MIDI data that came out of the Max patches we had created (see Figure 1). This Max patch allowed me to translate the data sets from an individual country over a designated period of time, into a 1-128 spectrum that could then be represented by MIDI. I was using data from February - May 2020. I found this process to be highly informed by the newsreel soundscape I had already put together, as I felt connected to and engrossed in each local context as I was choosing the virtual sounds to represent each country and determining how quickly or slowly I wanted the sequence of data points to cycle from the specified start to end date. Ultimately these MIDI soundscapes of the COVID-19 dataset created a sonic curve that speeds up and slows down differently based on the number of data points (positive cases) by county, but in all cases has a slower start and a dynamic centre or climax, representing the greatest number of COVID positive cases. In real time, I laid new tracks with the MIDI data from a given country on top the newsreel tracks from that same country and then listened back to the flow of the two tracks and how they related to one another. I augmented very little in the MIDI Max patch except for the speed at which the data points were cycled through, resulting in a quicker or slower spitting out of the notes from my controller. I automated panning within a stereo field and the overall gain of each track to develop the sonified piece you hear below my voice in the audio clip.

Liam Bray

00:00 / 00:00

Mr Liam Bray:

I am an interaction designer and design researcher; my main interest is in systems that automate or extend some part of creative processes. The bulk of my time is spent designing, prototyping, and testing experimental tools which collaborate with their user when making art. For my contribution to the NanoResonance project I took inspiration from the early cyberneticians, Norbert Wiener and Ross Ashby. Their descriptions of systems of feedback and control are deeply relevant to the kinds of sociotechnical experiences we have with data embedded all around us.

One year after the end of World War II, Ross Ashby, a psychiatrist who had written and studied the biological process of homeostasis (an example of which, is the body's capacity to regulate its internal temperature), used parts garnered from sperry machine gun turrets and other automatic aiming devices to create what he would go on to call the world first “thinking machine” the homeostat (Ashby, 1960). At the same time, Norbert Wiener, an American mathematician and philosopher at MIT had during this same period begun to develop his general theory of machines which he called Cybernetics. Wiener defined cybernetics in 1948 as "the scientific study of control and communication in the animal and the machine" (Wiener, 2019). His systems describe a feedback loop where the input redefines the output in an iterative process. Cybernetics was the framework by which inventors like Ashby were able to endow their machines with anthropomorphic analogy. Ashby’s claim was that the homeostat wasn’t simply a automatic regulatory system. It had knowledge, and it used that knowledge to act in its environment.

Grounded in this framing, I turned my attention to the agency of the virus itself, its own system of regulatory control and its own knowledge. I wanted to use the data as a representation of that agency, and its ability to destabilise us. I used the curvature of the rising case numbers in Sydney, Australia, where I live, as a waveform, translating this data into a ‘wavetable’ that was readable by a digital oscillator within a hardware synthesiser system. Composing with this bespoke waveform allows us to literally hear the curve itself, now represented as an oscillating, audio-rate wavetable. The sound this made was not analogy, it was the data representation. However, the embedded knowledge in that sound was the impact the virus has had on its environment, its march upwards was the agency of the virus moving towards a homeostatic state, an equilibrium, that we’re not coming back from.

Benjamin Carey

00:00 / 00:00

Dr Benjamin Carey:

I am an electronic musician, composer and practice-based researcher. Much of my work focuses on sound synthesis, performance, and composition with analog hardware, as well as designing interactive approaches to electronic music composition and performance.

The NanoResonance project, at its core, is an exploration of new artistic methodologies towards interpreting and creating with scientific data. We chose to work with datasets on COVID-19 from the Johns Hopkins University database because of its contemporary relevance, as the spread of this pandemic has touched us all in various ways throughout the past 18 months (Dong et al., 2019). Having a quantifiable and up-to-date resource for rising case numbers worldwide provided each of us with a series of curves to interpret artistically, to channel our own musical responses to the pandemic through. It was important to us that this data be used both as a source of inspiration, and as a concrete resource for developing musical and sonic material. The artistic methodology employed in all of our work is that of ‘sonification’, that is, making the inaudible audible in some way. Sonification often makes use of data in some form as its raw material, and can range from approaches that seek to clearly communicate information through sound, to artistic methodologies that take a less literal, communicative approach to the use of data.

In my work, I was interested in embedding traces of the pandemic within complex, interconnected networks of electricity, allowing these to generate new musical gestures, as well as to influence the combination of existing sonic events. My work was not designed to communicate information, but to use the inherent dynamism of this data to suggest and directly inform methods of structuring musical events. Similar to the approach taken by Dr Diana Chester, I made use of COVID-19 data as a form of control, however in my case, I translated the data being output from our Max patches into control voltages for use in a hardware modular synthesiser. I developed a bespoke, generative patch on my modular synthesiser that made heavy use of audio and control feedback in its design. The patch threw sonic events around the stereo field, surfacing diverse sounds in a cacophony of noise and texture. Taking looping datasets derived from case numbers in three countries, USA, UK and Australia, I converted the dynamic curves of rising case numbers into voltages that were then routed around my modular system. This data affected the speed and stereo placement of events, the density of the textures heard in the patch, as well as pushing and pulling upon other aspects of the control architecture of the system.

The analogy that emerged through working with data was the way in which the interconnected nature of our society was made tangible throughout this pandemic. Whilst the data did not generate raw material, its presence forced a complex, interconnected system into a particular set of musical trajectories, none of which could have been predicted in advance. This approach enabled an abstracted yet coherent connection to the theme of the pandemic.

Conclusion:

During the course of this research, the team explored our own creative sonic practices and the way they were influenced by the subject matter of the COVID-19 data. The resulting artistic work and presentation format, the audio paper, highlights synergies between non-human and human actors present in the process of composing and the research's eventual contextualisation in a sonic format. The work of the human actors, the three composers, is connected through their compositions made from the source data from the Johns Hopkins University COVID-19 map and the Max patches programmed to turn that data into MIDI and control voltage. In this way, this audio paper does not simply present the voice-overs of each contributor or their recorded conversations during team meetings. As Groth and Samson suggest, citing Bruno Latour “the audio paper is not limited to narrations performed by human beings. Not only humans act, also landscapes, objects, technologies and politics are rendered active agents” (Groth and Samson, 201; Latour, 2005). The audio paper allows for a deep and rich exploration and discussion of the data, method, and findings of our work. The format we have chosen for the presentation of these research findings includes human voices but also includes the data itself, the technologically composed and mediated data, and the methodological approach for sonification of that data. In line with Groth and Samson’s sentiment, “The audio paper is never conventional, as it always incorporates an awareness of the processes of research and technological production.” (Groth and Samson, 2016)

Conclusion

00:00 / 00:00

References:

Ashby, W.R. (1960). The homeostat. In Design for a Brain (pp. 100-121). Springer, Dordrecht.

Cariani, P.A. (2009). The homeostat as embodiment of adaptive control. International Journal of General Systems, 38(2), pp.139-154.

Cycling '74 (2022), Max, Software, https://cycling74.com/products/max-features

Dong E, Du H, Gardner L. (2019) An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis; published online Feb 19. https://doi.org/10.1016/S1473-3099(20)30120-1

Haken Audio (2022), The Continuum Fingerboard, https://www.hakenaudio.com

Latour, B. 2005. Reassembling the Social: An Introduction to Actor-Network Theory. Oxford: Oxford University Press.

Malafouris, L. (2008). At the Potter’s Wheel: An Argument for Material Agency. In C. Knappett and L. Malafouris, ed., Material Agency: Towards a Non-Anthropocentric Approach. New York: Springer., pp. 19–36.

Pickering, A. (1995). The Mangle of Practice: Time, Agency, and Science. Chicago: University of Chicago Press.

Schön, D. (1983). The Reflective Practitioner: How Professionals Think in Action, edn, 1. Ashgate Publishing Ltd, Aldershot.

Scrivener, S. (2000). Reflection in and on action and practice in creative production doctoral projects in art and design. Working Papers in art and design, 1

Wiener, N. (2019). Cybernetics or Control and Communication in the Animal and the Machine. MIT press.