Between human and algorithmic performance, how does meaning emerge through malfunction?
This essay investigates the possibilities and challenges of interpreting graphic musical notation through both human and computational means, juxtaposing expressive freedom and interpretive precision. Graphic scores, such as those in John Cage’s Notations, challenge traditional notation methods by expanding the field of how musical ideas are represented and interpreted. These deviations enable novel explorations of the dynamics between composer, performer (whether human or machine) and listener. Through an examination of scores selected from Notations, the essay reflects on their interpretive affordances and the kinds of musical interactions they invite.
The second part of the essay explores possible approaches for algorithmic interpretations of graphic scores, which offer rapid analysis and sonic reproduction of visual parameters but lack the indeterminacy and serendipity of human performance. Inspired by glitch principles, this essay proposes embedding unpredictability into these computational systems but as a parallel, complementary mode of expression that mirrors the spontaneity of performative practice while remaining distinctly algorithmic.
Notating music is a peculiar exercise: it seeks to capture sound through symbols that, while striving for clarity and determinism, inevitably omit swathes of information. From this ambiguity, interpretive agency emerges. Graphical notation, emerging prominently in the mid-twentieth century, represents a radical departure from traditional staff notation, often embracing this inherent indeterminacy. But what happens when this uncertainty comes about not by design, but by accident?
Once, while working on a piece of music, I encountered an unexpected problem: a software bug distorted my carefully designed visual score. What should have been a collection of orthogonal lines and tangent vectors became an erratic scatter of pixels. Although I restored the score to its intended state, the accident left me imagining alternative trajectories for the piece, sparking reflections that would later shape the ideas explored here.
In the realm of digital audio, one often finds unexpected sources of inspiration in unintended distortions. When malfunctions intrude upon the compositional process, do they extend the possibilities of notation, or do they undermine its purpose? Such questions are not purely technical; they arise both from compositional practice and a broader curiosity about how our tools and their “failures” shape the ways we relate to sound, to one another, and to the systems through which we make music.
From this perspective, the essay explores graphical notation as a framework for algorithmic interpretation, examining how computational approaches augmented by glitch aesthetics can complement, extend, and challenge human performance to serve as a potential basis for a contemporary notational aesthetic.
The mid-twentieth century was a particularly prolific period for avant-garde composers, marked not only by radical new sounds but also by equally radical ways of writing them down (Hall 2018). Innovation in musical notation accompanied these developments, as composers sought to create scores that could accommodate fluid, ambiguous sonic and performative situations. This concern with process and openness was not limited to music, with comparable emphases on process over fixed form taking shape across artistic disciplines.
At the same time, the conditions under which art and media were produced and experienced were rapidly changing. The spread of new media and mechanical reproduction, as Walter Benjamin observed, had eroded the ‘aura’ of the artwork (Benjamin 1968, 217-251), challenging earlier notions of originality and authorship. Indeterminacy in music can be read as a response to this condition, underlining the unique character of performance in contrast to fixed media.
Within this context, John Cage’s use of the I Ching as a tool for indeterminate composition exemplifies the shift from authorial control to procedural generation. His 1951 work Music of Changes demonstrates how compositional decisions could be delegated to chance (J. Cage 1961). Writing a few years later, George Brecht expanded on these ideas in his influential essay Chance Imagery, referencing the work of Cage extensively (Brecht 1966, 18-25) in his discussion of generative systems. Brecht’s essay is just one example of the reciprocal influence between the visual, sonic, and performative domains that characterised progressive art of the time and formed part of a broader cultural movement questioning the structures of media and authorship. This tendency is vividly captured in Notations, a 1969 collection of single-page manuscripts curated by John Cage and Alison Knowles (Cage and Knowles 1969). 1 Though never reprinted, it remains a unique reference point in the study of graphic notation, offering a snapshot of a transformative period in music history when the boundaries between composition, interpretation, and visual language were being reimagined.
This section does not aim to provide an exhaustive analysis of graphical notation in the mid-twentieth century. Instead, it draws on selected works from Notations to illustrate key aspects of graphic representation in relation to traditional Western notation and the interpretive possibilities these deviations open up. These examples will serve as a foundation for the following section, which explores how such scores might be codified and interpreted algorithmically within a digital framework.
A note on terminology: The notion of the musical note is used interchangeably with that of the event. I feel this nomenclature is more appropriate given the experimental nature of the scores and the music produced by reading them, as it consciously opens a space for unpitched or otherwise inharmonic sonic material. Likewise, the terms graphic score and visual score are used interchangeably to describe notational systems where visual form provides the basis for open-coded musical interpretation, the former being more common in historical contexts and the latter in discussions of digital or algorithmic notation.
Building on the idea of the score as a space of possibilities, the performer’s role as an agent or particle navigating this space mirrors the operation of generative systems, described by Boden and Edmonds as those “in which work [is] produced by the activation of a set of rules and where the artist lets a computer system take over at least some of the decision-making” (Boden 2009, 24). Prior to the performance, and prior even to practice, a performer must decide how to “read” or interpret the symbology of the score/image – meaning is at once premeditated and procedural.
In both vernacular and academic forms of notation we encounter conventions that continually evolve in response to the struggle between the imagination of the composer and the limits of notational language. These conventions delineate the interpretive framework: tonal-metric relationships expressed on a staff line, tablatures specific to a particular string instrument, oral-mnemonic traditions surrounding raga, graphical interfaces and symbolic protocols in digital notation... What, though, of the information not contained or communicable by the given notational system? This is the negative space of the score, in which the as-yet unthought or uncommunicated aspects of a piece invite the performer to fill them by herself and with herself and so become an active participant in the unfolding of the piece.
Similarly, a generative algorithmic interpretation relies on rule-based processes to 'read' and manipulate the score, through which a kind of (pseudo-)creativity emerges. These systems, while essentially deterministic, 2 can produce complex and unexpected results, particularly when designed with flexibility and nuance (Boden 2009). Particularly interesting about algorithmic interpretation of a score’s symbology is the necessity of explicitly and universally defining any reading and its inherent rules. Unlike a human performer, in whose performance we invariably encounter not only a certain serendipitous indeterminacy, but also culturally mediated performative preconceptions, an algorithmic interpretation adheres strictly to predefined parameters unless exceptions are explicitly stated. Perhaps somewhat paradoxically, this rigidity allows algorithms to uncover implicit or emergent properties of the musical-performative space defined by the score. By shrinking the number of interpretive possibilities, algorithmic approaches create a more narrowly defined yet potentially more productive range of outcomes, more explicit in their readings and unburdened by years of conventional training. Algorithmic interpretation also prompts a re-examination of the act of interpretation itself, offering a heterarchical model in which the superimposition of an interpretive method on the visual score forms an integral part in the realisation of the work.
This at once technically constrained and conceptually extended framework highlights the advantages of computational algorithms in exploring and interpreting experimental musical notation, using them not just to reproduce a piece of music, but as a tool in investigating the plane of interpretation/appropriation of the musical-performative space.
Let us return to the metaphor of the performer as a particle navigating the territory of the score. A strategy for algorithmic interpretation may be defined as follows:
We wish to create a system in which a score can be defined digitally and interpreted — played, in essence, by an algorithmic performer. To begin, we must provide the algorithmic system with the drawings that constitute the score, translating the graphically defined territory into terms that a computer can understand. This translation requires expressing the score mathematically, relying on Boolean data types to denote its presence or absence at any given point. This binary representation forms the foundational step, as subsequent processes hinge on a smooth and accessible implementation. The challenge lies in creating precise instructions to define a score within this binary framework, necessitating a branching network of sub-rules. These rules must account for a range of expressive parameters, including boundary, position, shape, size, opacity, fill, and regularity.
Once rendered in this way, we can program a set of arguments to guide our digital performer — the particle — through the score. For the composer, these arguments may take the form of text fragments that convey interpretive information, such as duration, metre, number of voices, instrumentation, potential paths, decision points, velocity, and starting and ending points. While designed to be human-readable, these text fragments correspond to underlying rulesets that interlock to determine the particle’s path through the score.
Such a system would enable rapid iteration of multiple interpretive approaches to the same score, illuminating its implicit properties for the composer. These algorithmically generated interpretations could serve as standalone musical pieces or provide a basis for collaboration with human performers. Moreover, it opens the possibility for human and algorithmic performers to collaborate in real-time, interpreting the same image- and text-based instruction set while producing distinct sonic outcomes. One can imagine a resulting convergent interpretive space in which algorithmic and human prehension navigate the boundary between the discrete and the continuous.
A Speculative Study: Modelling Attentive Simultaneity
To situate these concepts more concretely, I include here a short audiovisual study illustrating attentive simultaneity — the capacity of an algorithmic reader to distribute its attention dynamically across multiple scales of a score, lingering, accelerating, or reframing its perceptual window according to internally accumulated parameters. Rather than recognising discrete symbols or notational events, the system treats the score as a navigable landscape whose textures guide its movement.
The example uses local binary pattern analysis to extract textural features from the score across micro, meso, and macro scales, contending with the image at different visual resolutions. The dynamic scope of each scale forms attentive windows encompassing the algorithmic field of view. Their outputs combine to modulate sound synthesis: textural features influence the generation of stochastic waveforms at the sample level, and accumulated measures trigger resonators and alter the temporal weighting of successive frames. 3 The audiovisual excerpt serves as a representative demonstration of these principles rather than a direct render from the system, which is presently in development as part of this research.
Metre emerges adaptively from evolving local conditions, much like in Kasemets’ Timepiece for a Solo Performer. Decision and inflection points arise not from fixed temporal coordinates or event-triggers but from thresholds of entropy and cumulative change within the analysed region.
Rather than imitating human interpretation, the system develops an interpretive agency grounded in computer-vision operations, using their idiosyncratic modes of prehension as creative constraints. By allowing the algorithm to “linger” on a region, to widen or contract its attentive window, or to treat the score simultaneously as detail and totality, the system begins to function less as a deterministic executor and more as a co-composer. The dynamic modulation of synthesis parameters and their arrangement provide a glimpse of a subjectivity that is congruent with the logic of the machine.
The preceding example invites a reconsideration of what constitutes interpretive autonomy when computation is an active component of performance. As Luciana Parisi points out in her book Contagious Architecture: Computation, Aesthetics and Space, “… digital algorithms are not simply representations of data, but are occasions of experience insofar as they prehend information in their own way, which neither strictly coincides with the binary or fuzzy logic of computation nor with the agency of external physical inputs.” (Parisi 2013, xii-xiii) From this perspective, the algorithmically read score and the aesthetic experience that proceeds from it represent an idiomatic form of knowledge in which the algorithm asserts interpretive autonomy, as both actor and record of action. Yet while it is clear that a cellist is not an automaton-extension of the composer, it is not immediately clear that a musical reading based on a strictly procedural algorithmic mode of perception reflects a similar interpretive autonomy.
When considering the algorithmic performance of an open-form score, the question arises whether it is possible to reintroduce some of the indeterminacy inherent to human musicianship into the performance of the algorithm. This indeterminacy — nuance, unpredictability, variation — is the very reason why established musical works continue to be performed, each iteration offering a unique experience rather than a definitive reproduction. One intriguing possibility is to embed controlled unpredictability within the computational system itself, drawing inspiration from the principles of the glitch aesthetic. (Cascone 2000)
"Glitch displays the fragility and vulnerability of technology. It points to the dialectic moments of irrationality embedded in rational computers. (...) And rationality is not just about following rules. There is something irrational about following rules slavishly".
As Torben Sangild outlines in Glitch: The Beauty of Malfunction, glitch art celebrates the unintended and unexpected, the malfunction as a creative force. Emerging in the 1990s, the glitch movement in digital music, with earlier roots in tape-based phase experiments by artists like Steve Reich and Brian Eno — himself a proponent of graphic notation — offers a paradigm for embracing error as an aesthetic choice. Sangild highlights how glitches, by disrupting the smooth functioning of a system, reveal its inner workings, its underlying mechanics, and its inherent fragility. "Glitch displays the fragility and vulnerability of technology”, he observes, adding that “It points to the dialectic moments of irrationality embedded in rational computers. (...) And rationality is not just about following rules. There is something irrational about following rules slavishly" (Sangild 2004, 268-269). Indeed, while creativity arises largely through structured recombination, true novelty requires some form of interruption. When applied to algorithmic performance, this approach could involve introducing intentional disruptions or errors into the deterministic flow of the algorithm, thereby creating a space for spontaneity. Just as graphic musical notation seeks to expand the boundaries of musical expression by shifting the interpretive agency from prescriptive symbols to open-ended structures, so can glitch introduce an aesthetics of indeterminacy into algorithmic interpretation. Rather than a subversion of the algorithm, error represents a natural extension of it.
A view contemporary with many of the composers whose works we examined previously may be found in the work of Dick Higgins, whose early reflections on computational art emphasize the potential of machines not only to execute but also to inspire creativity (Higgins 1968). Rethinking the role of error and system constraints allows algorithms to function as co-creators rather than mere executors. In this sense, glitch allows for a kind of artificial agency that can enrich the deterministic models of algorithmic interpretation outlined in the preceding section. By introducing allowances for error — through pseudo-randomisation, disruptions in predefined paths, intentional breaches of the rules, or other avenues of creative vandalism — the algorithmic process may be modified, repurposed or wholly broken, allowing it to transcend execution to explore new creative possibilities. Rather than simulating human indeterminacy, this reframes the exercise as a dialogue between composer, system, and performer. The mistake becomes a means of exploration in its own right, one which introduces “fuzzy states … involving new processes of quantification” inherent to the algorithmic mode of prehension. (Parisi 2013, 97)
We have seen that glitch — error, malfunction, regression, unintentionality, recursion — can prove to be a productive instrument in conceptualising an interpretive algorithmic agency. Yet an unresolved paradox remains: is glitch something we can will into being, or does it preclude premeditation by its very nature? In seeking to instrumentalise glitch, do we risk irrevocably distorting the very conditions that make it meaningful? The ontological purist may argue that its essence lies in the unpredictability of its emergence from malfunction. To retain its authenticity, it must remain outside of our control. In phenomenological terms, however, even the pseudo-random approaches we may readily employ in implementing glitch principles within a digital system could be perceptually indistinguishable from "true" glitch.
At the level of perception, the distinction loses significance; what matters is the effect, the disruption, and the creative potential that the idea of glitch unlocks. Even deliberate, calculated systemic sabotage has the potential to result in wholly unexpected outcomes. Still, glitch is only glitch insofar as it is unexpected to both listener and performer. By its very nature, it excludes the possibility of acclimatisation, remaining outside any resolved forms or structures.
Creating the necessary liberty to instigate the unintended is to an extent an aestheticisation of glitch, to be differentiated from its imitation, which concentrates on reproducing its sonic effects. The skip of a scratched disk, an overloaded CPU, a sound driver crashing, a dispersion of pixels — all of these produce distinctive artefacts, and can be sampled, combined, twisted and composed with, yet are rarely taken for the real thing when heard in a musical context. It is important not to conflate the two; the essence of glitch as a productive device that confers systemic agency lies in its emergence from the logic of the very same system, whether intentionally placed in it or not, not in the sonic effects normally associated with it, which can comfortably exist independently of it.
This leads to another critical question: where does glitch reside? Does it exist solely in the malfunctioning system, or is it brought into being by the perception of the listener? In other words, we are confronted with glitch as a phenomenon in and of itself in the world/system, as well as something that belongs inherently to the sphere of interpretation. In the latter, the listener assumes the role of a performer in their own right, actively engaging with, interpreting and ordering the artefacts of glitch. This expands the conceptual space of interpretation and appropriation, positioning glitch not as an isolated event but as a dynamic interaction between composer, performer and audience, a distributed form of creativity both within the musical-performative space of the score, and as a prehensile mode in its own right.
This understanding relates closely to the embodied nature of listening itself. Listening is not a passive act but a layered and multifaceted process, deeply tied to the body and the environment. By introducing perceptual irregularities, glitch challenges the listener’s expectations; it creates a space for new meanings to emerge, a space where interpretation is no longer fixed but continually evolving. This evolution is decidedly circular: a momentary destabilisation leading to acclimatisation, whose establishment is then subject to inversion, re-establishment, and so on. In this, glitch goes between being there and then not, designated always by that which preceded and followed it, never by itself. It is ultimately a non-thing that nonetheless exists within and emerges from the logic of the system, rendered substantial by our contextual perception. In this it is quite like the score in which it is embedded; meaning arises not from a single event but from a set of continuously evolving relationships.
In the tension between determinism and malfunction, every interpretation is provisional.
The graphic score already gestures towards this openness: an attempt to codify sound while highlighting and expanding the interstices of the composition, formally articulating the performer’s participation in the compositional process. By translating the visual score into an algorithmic reading, however, we introduce a different constraint — one that demands every interpretive rule be made explicit. Glitch intervenes at this threshold, reinterpreting from within the logic of computation what is lost in the process of codification. In this light, the algorithm moves beyond its role as a mere translation layer between graphic score and sound, beginning to act as a performer in its own right.
In reflecting on these systems, we are also reflecting on the shifting boundaries between composer, performer, instrument, and listener; on how we negotiate authorship and agency with the tools that increasingly co-create our music. If algorithmic systems (and the points of failure embedded in them) can complement traditional performance practices, they also challenge and redefine them.
Benjamin, Walter. 1968. The Work of Art in the Age of Mechanical Reproduciton (1935) (in Illuminations). London: Fontana.
Boden, Margaret A., and Edmonds, Ernest A. 2009. “"What Is Generative Art?".” Digital Creativity, vol. 20, no. 1–221-46. https://doi.org/10.1080/14626260902867915
Brecht, George. 1966. Chance Imagery. Something Else Press.
Cage, John. 1961. Silence: Lectures and Writings - To Describe the Process of Composition Used in Music of Changes and Imaginary Landscape No. 4 (1952). Wesleyan University Press.
Cage, John, and Alisson Knowles. 1969. Notations.Something Else Press.
Cascone, Kim. 2000. “The Aesthetics of Failure: Post-Digital Tendencies in Contemporary Computer Music.” Computer Music Journal, vol. 24, no. 4 12-18. https://doi.org/10.1162/014892600559489
Hall, David. 2018. Graphic notation: a brief history of visualising music. Accessed January 10, 2025. https://davidhall.io/visualisi....
Higgins, Dick. 1968. “Computers for the Arts.” Computers and Automation, vol. 17, no. 11.
Kamb, Mason, and Surya Ganguli. 2025. An analytic theory of creativity in convolutional diffusion models. arXiv. https://doi.org/10.48550/arXiv.2412.20292
Parisi, Luciana. 2013. Contagious Architecture - Computation, Aesthetics, and Space. MIT Press Books .
Sangild, Torben. 2004. “Glitch - The Beauty of Malfunction.” In Bad Music: The Music We Love to Hate. Routledge.
{X * current_state + (1-X) * previous_states}. Accumulation occurs when the value of X remains low for an extended period; high values erase the cumulative history. This adjustable persistence allows the system to “carry” traces of its recent past, producing effects of (dis-)continuity analogous to perceptual afterimages or ghosting.