Synaesthetic Audio: Tracing the Digital Index Across the Senses
University of Amsterdam
(Graduate School of Humanities)
MA Heritage Studies: Preservation and Presentation of the Moving Image Thesis Supervisor: dhr. dr. F.J.J.W. (Floris) Paalman
Second Reader: mw. dr. E. L. (Eef) Masson Submitted: 26th June 2017
Jim Wraith (11316810) Jonge Roelensteeg 4D 1012PL Amsterdam
Synaesthetic Audio: Tracing the Digital Index Across the Senses Table of Contents
Introduction 3
1 Historical Background 9
1.1 Sonic Transcription 9
1.2 The Index and Other Locations of Sonic Reality 13
1.2.1 The Index 16
1.2.2 Other locations of the Index: Time, Traces, and Trajectories 17
1.3 Digital Indices and the Fungibility of Everything 22
1.4 Data Sonification and the Codification of the Index 26
1.4.1 Data Sonification 26
1.4.2 Algorithmic Synaesthesia 29
2 Case Studies in Optical Synthesis 32
2.1 The Direct Transposition of Sound and Image 34
2.1.1 The Sound of the Apparatus 35
2.1.2 “Intermediate” works 37
2.2 The Analogous and Symbolic Transposition of Sound and Image 39
2.2.1 German Graphical Sound 39
2.2.2 Soviet Graphical Sound 41
2.2.3 Norman McLaren 44
3 Proposals 46
3.1 Proposals for the Direct Transcoding of Digital Sound and Image 46 3.2 Proposals for the Analogous and Symbolic Transposition of Digital Sound and Image47
4 Conclusions 53
5 Works Cited 55
5.1 Bibliography 55
5.2 Audiovisual works referenced 58
Introduction
In 1922, Laszlo Moholy-Nagy published the essay Production-Reproduction1 where in the call is made in support of experiment as key to the furtherance of creative processes, constituting the better training of the senses. An explicit part of this appeal is to the investigation of the
apparatus, and the recasting of reproductive technologies as productive ones. Whilst Moholy-Nagy was writing in reference of the phonograph, this call seems apposite given the increasing
convergence of productive and reproductive capabilities within the self-same apparatus, and the increasing availability of audiovisual material upon which to work. With the ability to create, manipulate, and reproduce audiovisual material becoming available to large numbers of people, it would seem appropriate to consider the possibilities of creative activities that explore the unknown “rather than simply reproduce the familiar”.
The purpose of this work is to investigate conceptual frameworks and practical methods for the creation of generative sound on the basis of information drawn from digitized moving image material, as what might be considered as a prosthesis for images created without sound of their own. Central to this analysis is a discussion about indexicality as the source of an archival object’s connection to reality, and hence the source of the authority of the archive. Increasing amounts of born-digital are being created at the same time as more born-analogue material is being digitized. At the same time as the inherent transferability and fungibility of binary data open new avenues of creation and manipulation, prevalent notions of the indexical relation between audiovisual media and what can be considered reality face epistemological challenges2. For want of
iconico-indexicality, new locations of the reality effect of the cinema, and of the indexicality must be investigated. It is my contention that indexicality, rather than being unique to visual iconicity, can be found in a number of places, and exists as one element of processes of cognition and recognition dependent on sensory training and implicit knowledge of the apparatus of reproduction.
With reference to theory within film studies, art practice, and computer science, and taking symbolic and conceptual cues from practices involving the transposition of the visual and the auditory, this work seeks to further understanding of the nature of audiovisual material’s relationship to reality as a source of archival authority. Faced by the epistemological questions
1 Passuth, 289
posed by the digital turn, this requires a discussion of how media are affected by processes of digitization and remediation, and how this affects and can be of service to archival practice3 in
terms of both the development of data into knowledge, and improving provision of archival access. In this instance we may see what (if anything) is gained or lost in the tide of digitization, and how we assign truth value to audiovisual artefacts in the face of the increasing “contentization”4 of
cultural heritage.
Taking the digital turn as primarily an epistemological concern5, it is proposed an impetus
has arisen for the development of new methods and tools of analysis within the humanities. Whilst the use of digital processing has provided fecund in the fields of visualizations tools, or geographic imaging systems, sonification is a field where little to no work has been completed that is of use to media historiography. I propose that sonification can prove apposite to the analysis of moving image due to commonalities in the nature of the sonic and the visual, particularly in so far as they can be related to temporal processes. Similarly, in the spirit of contemporary trends in humanities historiography, sonification works could provide a new manner of accessing information and presenting research away in a non-linear, non-textual fashion, presenting instead the beginning of new ways of navigating and understanding large complex datasets the likes of which are becoming ever more prevalent.
To discuss the transference of data from one sensory mode to the other, I introduce the notion of fungibility. It is my argument that the indexical link to reality recognized in audiovisual material is fungible in that it can survive transposition and remediation between the visual and sonic fields. However, it will be argued that models of direct transference between sensory modalities is insufficient for the meaningful conveyance of information, and that the considered application of a symbolic logic is necessary for indexical links to be retained in a coherent manner. The value of this to archival practice is, initially, to provide a foundation for new modes of access to archived
audiovisual material, and new methods of garnering knowledge from and about same. Concurrent to the increasing pervasiveness of computing in the audiovisual world when it comes to processes of creation and archiving, the range of tools available to assist researchers within the digital
3 Olesen, 12-13 4 Meyer, 15
humanities remain limited6. However it is not sufficient just to call for new tools the case must also
be made for which tools are needed and why.
It will be asserted that this provision of a symbolic logic, and ways of transcoding and transposition that retain a meaningful and recognizable representation of the indexical trace is dependent on a broader understanding of the specificities of the auditory medium, and the ways in which processes of audition are internalized as a result of familiarization. By developing methods that better encourage the sensory training such familiarization represents, methods will be proposed that utilize audition to convey information in a manner better suited to the sonic field than the visual. The challenge specific to the digital media in this instance is the complication presented by thickening layers of abstraction and complexity.
As an element of best archival practice7, open-source software and open formats will be
recommended in so far as is possible. This is to serve the goal of longevity of accessibility, the availability of source code and documentation staving off the threat of obsolescence by enabling strategies of recreation, migration, or emulation at some point in the future.
In my first chapter, I will delineate a brief history of sonic transcription, and the
epistemological models that have been created to explain the recording of sound. Taking cues from the Media Archaeology theories of Thomas Elsaesser, a broad purview is taken in looking for the origins of the cinematic sound apparatus, with preceding technologies being taken from the fields of cinema, music, and the natural sciences. These developments are discussed in terms of their
allowing the conditions of possibility for purely synthesized sound (that is, sounds made without reference to a naturally-occurring sonic event). Once this historical background has been
established, different theories regarding the connection between transcribed sound and image and the events they purport to represent are considered. Iconico-indexicality, the recognition of the image as printed on the character will be discussed as representative of a cinematic orthodoxy that privileges the visual and is insufficient for discussion of the audio dimension. This orthodoxy is discussed in terms of the “naturalized” listening which encourages the cinematic reality effect within works produced in a realist mode. Beyond this, other locations of the index will be
discussed. In searching for an epistemological model that can accommodate both sound and vision
6 Casey & Williams, 3 7 Lacinak, 11
on an equal footing, attention will be given to theories that concentrate on the measurement of traces of events. In this fashion, the indexical link to an objective reality will be located in the ability of audiovisual media to record trajectories of motion through time and space. Subsequent to this, the digital turn is discussed as a moment of epistemological rupture leading to a reassessment of epistemological and ontological concerns. Specifically for the present debate, the question of the digital index is addressed: visual iconicity is discussed here as something of a red herring, drawing the attention from the still present material inscription of sound and image, albeit inscriptions that are obscured from view by the increasing complexity of the apparatus and its concomitant
encoding, and the corresponding loss of easily recognizable visual iconicity. It is here I introduce the concept of fungibility in digital media as a means of theorizing around the easy reproduction, transposition, mutability of digital media. In discussing the fungibility of the index itself, theoretical and practical cues for the better symbolic encoding of the indexical link are taken from the fields of data sonification within computer studies, and Roger Dean’s theories of algorithmic synaesthesia drawn from practices of multimedia performance. In correspondence with epistemologies that emphasize the interrelationship of time and sound, the specific strength of audio in precisely judging temporal qualities will be discussed as potential strengths in sonification works.
In my second chapter, in order to place this work in the appropriate contextstudy will be made of graphical sound works that challenge notions of indexicality as found within realist sound reproduction, with attention also being paid to non-filmic works and practices that extend or abstract the optical sound technology of the analogue cinematic apparatus. In choosing to discuss works created at the margins of orthodox practice, testing the limits of the media, it is hoped that the role that the apparatus plays in the reproduction of the indexical site of the archival trace will be illuminated. Sources will be chosen across a range of practices comprising cinema, visual arts, sound arts, music and media art. This choice is made to so thoroughly as possible seek out and explore works that bring sound to the fore, going against the teleological or ahistorical paradoxical orthodoxy of turning a blind eye to the technological mediations of the apparatus that results from realist sound (allied with non-diegetic music) as the “natural”(-ized) state of affairs. This obviates looking at the prior-stated needed to approach this work with a purview that encompass works created outside of a realist mode of practice.
What unites these works, broad as they are in inspiration, ideological background, and aims, is the extent to which they challenge the presuppositions of mainstream film theory as regards the primacy of a specifically photographic conception of indexicality. If we take Zinman’s definition of synthetic sound as “the set of noises and tones produced by the optical soundtrack that result neither from recordings of natural sound events nor pieces of music capable of being performed by human musicians on conventional instruments”8, we can see where challenges to photographic
referentiality can begin. Works engaging with montage methods on the sound track already loosened the grip of the referential in a manner which does not threaten the indexicality of the recorded sound: the cut does not diminish indexicality, which governs its (re-)arrangement, but the emergence of optical synthesis allowing for tones to appear that make no reference to a sonic event, instead relying on the photographing of a symbolically-encoded set of instructions for the acoustic apparatus, applies further pressure to the question of indexicality. The degree of symbolic encoding applied in the transposition of sound and image is discussed in terms of their being either direct, created with the minimum of remediation or encoding, or existing on a continuum of
analogous/symbolic encoding, a distinction operating on the degree to which a structure is applied during the course of transposition.
The works presently under discussion support such definition of the index as
non-photographic/iconic by the transcription and representation of a spectrum of interactions with the carrier, which can be categorized along a number of (possibly arbitrary) axes relating to the
directness and method of inscription, methods and materials applied, or the application of a further apparatus (as per the photographing of rotating discs onto the optical track). This serves to reassert the location of the index in a number of locations beyond the photographic image. Whilst such works do not necessarily pertain to the reality effect of the cinema, they must be considered to bear indexical traces that link them to the traces of their creator’s actions. The matter at hand is to investigate how these traces are made manifest, and retain their presence because or in spite f the further technological processes involved in their production and reproduction.
In my third chapter I will offer concrete proposals for methods of sonifying the digital image that take into account the prior discussions of epistemology and works of optical synthesis.
Beginning with the direct transcoding of image into sound, it will be shown that the complexity of digital media encoding necessitates a nuanced approach to the construction of algorithms that
perform these transpositions. Methods of working with open source software to replicate the sonification of optical flow in the moving image will be taken up as a relative simple exemplar, based on the works of Jean-Marc Pelletier. Building on this, other approaches to the construction of sonified data will be proposed taking into account the specific strengths of the audio dimension discussed in the latter half of my first chapter. In discussing the technical requirements of creating such works, lacunae in the tools and theories used by the digital humanities will be identified.
In the final chapter, as well as recounting all that has come before, further speculative proposals for practices of sonification for the digital humanities will be taken up. Given the pieced-together nature of the tools used in the previous chapter, the call will be made for the import of expertise from a multidisciplinary perspective, so as best to fill knowledge gaps within the humanities. In addition to this, the need for further investigation into the best application of sonification techniques as a base for developing knowledge within the field will be addressed.
1 Historical Background
1.1 Sonic Transcription
“Cinema has never been one thing. It has always been a point of intersection, a braiding together of diverse strands”9
To understand the background of optical sound necessitates taking into consideration a number of these disparate strands as referred to by Gunning. This consideration will be undertaken with the Media Archaeology theories of Thomas Elsaesser in mind, in so far as that technological developments are to be understood as encompassing a broad range of different technologies and ideologies, operating at different points in time with very different agendas, with developments of new media not simply replacing or superseding that which has gone before10. Hence in the
development of the technology required to produce a sound at will, with no reference to a naturally-occurring sonic event, a number of technological developments and conditions of possibility arising from disparate fields of human endeavour can be identified. These have been theorized by Levin11
as comprising: initial experimentation to visually transcribe sonic events, the development of a method of acoustic transcription that allows for reproduction, the accessibility of said transcriptions and the ability for them to be manipulated in the same state as they are transcribed, and the
completion of a systematic analysis of said transcriptions so as to allow for the production of any sound at will. In recounting these developments, it is to be made clear that audio (re-)production technologies have always comprised transformative processes of transcription and encoding.
The development of the first of the necessary conditions can be attributed, as Levin has it, to Ernst Florens Chladni, who in his Entdeckungen über die Theorie des Klanges (1787) produced a method of visually transcribing sound by placing powdered quartz on a glass plate which was then vibrated by use of a violin bow. Finding that the quartz formed distinct patterns, changing as the tone of the bow stroke changed, we find definite visual traces of sonic qualities, traces with an “indexical” relationship to the tone itself. These developments encouraged the development of further methods of transcribing sonic events through analysis of their visual transcriptions (cf.
9 Gunning 2007, 35 10 Elsaesser 2016, 73 11 Levin 2003, 38
Tonschreibeskunst, Phonography, Vibrography) as well as the invention of specific apparatus for the completion of this task. Hence in 1857 Edouard Léon Scott revealed his phon-autograph, an
oscillograph designed specifically for the study of the human voice, to be followed in 1859 by the Scott-Koenig Phonautograph which introduced the linear transcription of sound as waves, the visual representation with which we would become the most familiar. In the matter of sound reproduction, it is also during this period that Wolfgang von Kempelen was perfecting his Sprachsmaschine, a rudimentary modelling of the vocal tract intended to synthesize human speech12.
It is with Edison’s development of the phonograph (initially revealed in 1877) that the second of the conditions of possibility are reached: in allowing for the transcription of sound in a format that allows for its later reproduction. Attempts were made to analyse and render recognizable the transcriptions provided: the Edison Company itself being involved in a number of lawsuits that attempted to define the status of phonograph recordings in so far as they could be considered “writing” in a legalistic sense13. Fifty years later, Moholy-Nagy would attempt to use phonograph
technology as the basis for investigation into whether these transcriptions could used as the foundation for a script for “sound-writing”14. What we can identify here is the continuation of an
epistemological characteristic rooted in the specifically scientific endeavours of Edison’s
12 Levin 2003, 34 13 Ibid. 41-2. 14 Zinman, 51
predecessors: such sonic transcriptions embodying a link to an objective reality, with said link representing a (contestable, at least) source of authority in so far as the information transcribed accurately represents reality. That the traces within which this information is contained are illegible to the naked eye precludes their immediate editing, contributing to the device a magical aura. Through subsequent technologies of sound reproduction, not least those found within the cinematic apparatus, this characteristic will remain present: that a “reality effect”, to be discussed later herein, will be experienced as the result of audition trained to blackbox the technological processes
occurring within the apparatus.
The emergence of sound reproduction technologies introduced philosophical questions. Most pertinent to the present work is the oft-cited15 Ur-Geräusch (‘Primal Sound’) of 1909, wherein
Rainer Wilke expressed the desire to use the new phonograph technology to “find” new (or ancient) sound by tracing the seams of a human skull with the phonograph needle, supposing that the
grooves in a skull may serve as “the locus of some sort of a signal (i.e. an inscription that, while not produced by a subject, might nevertheless be a trace of some other signifying agency)”16. This
notion that sound reproduction technology could serve to expose previously unknown qualities or attributes of a given object can be understood as an alternative attempt to identify something of the nature of the relationship with reality investigated by the apparatus. Whilst not central to the developments within optical synthesis and its application in film, the spirit of such attempts can be identified in different formations in a number of works under scrutiny in the present project.
The third condition of possibility for sound synthesis, the provision of a method of
reproducible sound inscription accessible and manipulable, was first reached with the development of sound-on-film systems. From what is known about the many diverse sonic technologies and practices extant in the days of the (non-)silent cinema17, two issues pertinent to the present project
can be identified that sound-on-film systems overcame. Firstly, they provided a consistency of performance superior to that of live accompaniment. Secondly, in aligning the sound and image in the same carrier, they provided for the synchronization of naturally-recorded sound to a degree not possible with competing innovations such as Edison’s Kinetoscope. Henceforth, from the 1906 UK patent of Eugene Lauste, through the developments of DeForest, the Tri-Ergon and
Tobis-15 cf. Levin 2003, Gunning 2007, James. 16 Levin 2003. 44
Klangfilms systems and so forth, the construction of an orthodox dispositif for the emerging sound cinema can be identified.
With the optical soundtrack, the difficulties of deciphering and manipulating the inscriptions of the phonograph are overcome, with the size of the optical film track allowing for its better inspection, and plasticity of the carrier allowing for a far wider range of interventions and
manipulations to be made directly onto the transcriptions. The optical sound apparatus served as the grounding for a diverse range of experimental works to be discussed herein that sought to
investigate the potential for the creation of wholly synthetic sound - sound from out of nowhere, sound emerging without any reference to a “natural” sonic event - and the boundaries of the apparatus that made them possible. These investigations, as embodied in the creations of those working in the field of the graphical sound film, can be understood as attempts to bring the completion the final condition of possibility for optical sound synthesis: the analysis and bringing into being a script or lexicon of sorts, allowing for the creation of sound at will.
It is of course worthy of note here that the notion of an orthodoxy in the dispositif is used advisedly: just as there has never been one true cinema, only corresponding practices within broadly similar parameters, parallel developments in optical sound can here be identified that provide necessary context. Just as the alliance of sound and image in the printed soundtrack was by no means a foregone conclusion - rather the result of technological, economic, and cultural forces – the cinema was not the sole beneficiary of its values. Shortly following the introduction of De Forest’s system, the pallophotophone was invented by Charles Hoxie18, an engineer at General Electrics. The
pallophotophone was a photographic sound recording/reproduction system allowing for 12 tracks of audio, that later served as a direct progenitor of RCA’s short-lived Photophone sound-on-film system.
Thus when referring to the optical sound apparatus, it is helpful to take a broad definition. The central and essential factor is the use of a light-dependent resistor (or ‘photo-cell’) as the central mediating component in the conversion of light into sound, be it modulated by the relative light/darkness imbued by the optically-printed soundtrack of a naturalistically recorded sound, or occluded by the presence of ink manually applied to the film, or the presence of any other matter
filling the slit between the lamp and the photo-cell receiver. Technologies and practices utilizing variations on this minimal conception of the apparatus can be found occurring in a number of fields, highlighting both the fuzzy boundaries of a given medium, and the difficulties of placing practices within neatly-bound definitions. Aside from the aforementioned pallophotophone, optical synthesis remained a going concern within musical practice until the development of logic-based synthesizer offered greater stability and convenience to the average user.
Writing at a time when the boundaries between different media and modes of practice continue to prompt epistemological questions regarding the nature of the moving image and the tools used to create and manipulate same. As technologies of production, manipulation, and reproduction of sound and image are increasingly becoming embodied within the same device (cf. the ubiquitous smartphone able to capture, edit, and distribute images with a speed and convenience previously unknown) and boundaries between previously distinct creative practices are becoming blurred19, it would seem apposite to define the scope of this work’s focus on the optical soundtrack:
as the predominant mode of sound reproduction in the cinema it has become the naturalized mode of audition in the cinema. The pertinence of this to the present research is two-fold: firstly, as a dominant technological mode it is of interest to think about how these practices have influence on technological developments post-digital turn as a result of practical/cultural continuity (as opposed to any technological strictures). Secondly, it is central to this research to investigate the location of the cinema’s reality effect as it can be seen to connect reproduced sound and image to what might be considered “real” events, and how this effect is contributed to by our internalization of processes of technological deployment and audition.
1.2 The Index and Other Locations of Sonic Reality
Early theorizing of the new sync sound technology foregrounded the visual, discussing the audio dimension in terms of the ways it could be put in service of the image, or complement existing visual modes of editing. Thus it was argued by Eisenstein, Pudovkin, and Alexandrov in their Statement on Sound20 about the new “talking films”, arguing for a ‘contrapuntal’ sound
montage as a means to most fully embracing the creative potential of the new technologies.
19 Elsaesser 2012, 109
However, questions soon arose about the epistemological implications of sound and sound
reproduction, and the implications for this in terms of the perceived “reality effect” of the cinema. Central to such discussions are the ways in which the apparatus and trained processes of cognition combine to provide a sense of immersion in the viewer/auditor. As per Elsaesser21, cinematic
perception can be understood as internalized, such that this reality effect is most present when the machinations of apparatus and technologies are least perceptible. In this context, notions of cinematic realism are to be discussed, with reference made to an orthodoxy that developed in the film-based media relating the iconicity of the photograph to an indexical link to reality, though bearing in mind the epistemological rupture of the digital turn22, other locations of the indexical link
can be reassessed against the requirements of new modes of audiovisual production. The prevalence of this to archival audiovisual media lies firstly in the source of the authority of archival materials, but also forces the question of the manner in which technological developments prompt broader reassessment of the ways in which historical representations shape collective memory.
Writing in the context of 1920s Modernist music and its slew of developments of new instruments and tonalities (the aetherophon, theremin, rythmicon etc.), Moholy-Nagy’s Problems in
Modern Film (1928-32) foresaw the development of the sound film as adjunct to contemporary
developments in abstract film, making the call for a radical application of sound reproduction technology to create works that would “enrich the sphere of our aural experience by giving us entirely new sound values, just as the silent film has already begun to enrich our vision” (p. 48 in Levin – Tones From out of Nowhere). It was only with the development of the optical film sound track that the required degree of control was found to allow for direct manual manipulation of the recorded trace (a result of the size and visibility of the optical soundtrack combined with the easier malleability of the carrier). This malleability and unification of the optical track on the same carrier as the visual, made the application of the range of editing techniques (splicing, touch-up etc.) already developed and applied for the visual applicable to the sonic. Hence, concurrent to the emergence of the feature-length sound-on-film film as discussed by Moholy-Nagy, works began to emerge that engaged with the call of Eisenstein et al. for explorations of how the recombination and alteration of naturalistic/referential sound recordings could be utilized to further cinematic creation. As early as 1930, Walter Ruttmann produced Wochenende23, where in the montage techniques of his
21 Elsaesser 2016, 17 22 Ibid., 73
prior abstract film works are put to use constructing a fantasy soundscape of the (non-visual!) city that could be considered a counterpart to Berlin: Symphony of a Metropolis (1927) which Ruttmann directed. Ruttmann himself positioned his work as a continuation of broader combinatory artistic strategies encountered in other spheres of artistic endeavour, yet also saw the sound film as a radical break from the silent film, and not just its enhancement24
A similar commitment to sonic experimentation is found in the early sound films of Vsevolod Pudovkin (himself a signatory of the Statement on Sound), such as Deserter (1933) wherein a non-synchronized, non-realist sound montage is placed at the service of the visual narrative in a manner which aligns closer to an ornamental, non-diegetic counterpart to the image than a realist reproduction of the chain of events presented visually. Also worthy of note is Dimitri Kirsanoff’s Rapt (1934)25, scored by the Belgian composer Arthur Hoérée. This work expanded on
the use of editing techniques to include the use of touching-up ink in a direct and creative manner to create a highly modulated and expressionistic sound montage. Whilst considered by its creator more in line with primitive tape music26 in the manner of Halim El-Dabh, Stockhausen et al., it is of
interest here as a combination of movements in engagement with the optical track through montage techniques, and concurrent practices aiming towards the development of wholly synthetic sound to be discussed as case studies later.
Theo van Doesburg, despite his own lack of either patience or aptitude in film-making27, was
a firm proponent of the cinema as a new ground for explorations of movement in time and space, writing in De Stijl to make the case for projected light as an autonomous medium and pleaded for film to be freed from “it’s relation to the subjective”, allowing expressions of an abstracted universality; in van Doesburg’s writing on film we see the emphasis on motion, and particularly immediacy of motion, commonly espoused by theoreticians of the early 20th-century avant-garde28.
Whilst largely subsumed in the later theories of the cinema’s reality effect, it will be shown herein that ideas of time, motion, and space (and the interrelation of the three) reoccur throughout later theorizing about the epistemology of cinematic sound and sound reproduction, human audition, and theories from data sonification to be discussed later in this chapter.
24 Elsaesser 2016. 177
25 https://youtu.be/fqNzJ3lrebQ [excerpt]. Accessed 10th March 2017.
26 James, 79 27 Van Beusekom, 63 28 Levin 1984, 57-8
1.2.1 The Index
As much as the sound film developed to take its own place in the corpus of its Modernist era, receptive and beneficiary of theories taken from the broad span of artistic practice, a
perspective within cinema studies soon calcified into an orthodoxy that, reasserting the presumed primacy of the visual, located the indexical link between reality and its reproduction in the iconico-indexicality of the photographed image as printed on film stock. Defined in Charles Sanders Peirce’s theory of signs as one element of an inter-related triad (comprising sign, index, and
symbol), the index is a sign that functions through an existential connection to its referent “by being really and in its individual existence connected with the individual object”29. Predominant
understandings of the index in film studies site the indexicality of the image in its visual
referentiality, the ability to recognize the image printed on the carrier. This understanding is most commonly associated with the writings of André Bazin, who saw the ontology of the photographic image as a theoretical transference of reality from subject to object, the directness and easy
recognition of this link serving as the guarantor of the image’s relation to an objective reality. This, though, is a usage too restrictive to apply comfortably to a number of non-naturalistic cinematic practices, and is based on an unnecessarily abstracted understanding of Peirce’s system of signs. Said system (as Wollen notes30) is an attempt to build a philosophy of mind, whilst Bazin was
writing to describe a specifically realist approach to the reality effect of the cinema. Hence the recognition of the iconicity of the printed image, the assertion of its indexicality by reference to a perceived visual recognition to its subject, serves as guarantee of the image’s representation of a “true” event. Gunning traces the source of film theory’s emphasis on photographic indexicality to Bazin’s emphasis on realist aesthetics and the concurrent assertion that the photograph and its referent “shar[e] a single being”31, though emphasizes the epistemological distance between Bazin’s
location of the cinema’s essential realism (or “realism effect”), and Peirce’s rigorous system of logic. Returning to Peirce’s definition of the index as one aspect of a semiotic triad, it is clearly a conception that does not depend on iconicity or photographic representation, no matter how far these are easily accessible and recognizable manifestations of same.
29 Gunning 2007, 30 30 Gunning 2004, 46 31 Gunning 2007, 32
Such a limiting understanding of the index proves problematic for a number of themes to be discussed where experimental works existing at the boundaries of creative practice serve to cast theoretical concepts into relief by introducing methods that draw into question the relationship between audiovisual material and the objective reality. In the instance of graphical sound film, the majority of which were completed in a purely animated, or at least non-naturalistic style, the relationship between the photographic image, sound and a given “event” is brought into question: what “action” is actually being represented on film, and how is it to be reconciled with the
objective? Optically-printed sound lacks any such visual iconicity yet, not least when recorded with realist intentions, retains the authority of the indexical recording. Such cases must be seen as
allowing a scintilla of doubt as to the universal applicability of Bazin’s iconico-indexicality, as suitable as it may be to understanding the reality effect found in a naturalistic, realist style.
However, as originally propounded by Peirce there is no specific requirement for the indexical trace to be visual, and a number of not-exclusively-visual traces can be imagined (e.g. glacial striation as a trace of the movement of glaciers across rock). Gunning squares the circle of photographic indexicality by siting the indexicality of a traditional photograph in “the effect of light on chemicals, not in the picture it produces”32, the prior focus on the image itself being then a
sleight of hand trick of misdirection: iconicity is not the sole defining factor of indexicality, merely one that supports an already-present indexicality by means of our recognition of the image, a negotiated truth claim dependent on our understanding the rules of discourse (that is, how to watch or hear film correctly). That such recognition is possible is dependent on a wide range of mediating processes largely rendered invisible by a cinematic apparatus operating in a realist mode that seeks to divert the audience’s attention from choices of lens, film stock, microphone choice and placement and so forth; here we can understand Bazin’s connection of visual indexicality and the reality effect as a mode of practice representing the shortest possible route to the internalized cinematic
perception referred to previously.
1.2.2. Other Locations of the Index: Time, Traces, and Trajectories
Whilst the theories discussed previously in this section were predominantly concerned with identifying the reality connection of the image, a later generation of theoreticians emerged who
placed the sonic dimension on an equal footing to the visual, leading to a series of new
epistemological constructs. In this tradition we have Michel Chion, who in Audio-Vision proposes an interrelationship between the visual and the aural wherein both elements are mutually supportive, as measurable in terms of the added value, defined as:
expressive and informative value with which a sound enriches a given image so as to create the definitive impression, in the immediate or remembered experience one has of it, that this information or expression “naturally” comes from what is seen, and is already contained in the image itself. Added value is what gives the (eminently
incorrect) impression that sound is unnecessary, that sound merely duplicates a meaning which in reality it brings about, either all on its own or by discrepancies between it and the image.33
In the language employed by Chion - “definitive impression”, “naturally” - we are provided the conceptual link that allows us to discuss the ways in which filmic sound can be connected to objective events. Levin does this by associating sounds (bar “fixed sounds”, those lacking in variation, that he contends can only be created “artificially”) as the “trace of a movement or
trajectory”34, a definition that explicitly links sound to time and motion. The image is provided with
temporality by its anchoring to points of synchronization within the audio track, sound playing a role analogous to punctuation, providing a trajectory in support of the montage, creating a sense of succession and “vectorizing” shots, orienting them towards a goal. It was sound that shackled the image to fixed tempos in a very literal sense when synchronized sound imposed fixed framerates on projection (Chion’s positing that it may be the case that “Sound Cinema is Chronography”35
represents the logical extension of this relationship), and the development of epistemologies of film sound elucidate and deepen the fundamental relationship between sound and time. Herein can be found an alternative route to the indexical relation of reality and the audio/visual, avoiding a
dependence on the iconico-indexicality of the photographic image: sound becomes the guarantor of time and motion within film, lending an authority commensurate to that of the moving image’s reality effect comprising in part deriving from the “tense structure” in which it holds the viewer, a “here and now” that is at the same time a “there and then”36
33 Chion, 5 34 Levin 1984, 57 35 Chion, 16-17 36 Elsaesser, 2016 p239
Branigan provides a nuanced model of the relationship between sound and time that seeks to investigate what sort of object is dealt with when we discuss sound. The vital difference here is the manner in which sound and image move through space: the apparently material image object shows itself to us by reflected light, propagated (broadly) along straight lines, resolving shapes and edges, whilst sound is perceived as both emanating from an object, and being diffuse in nature, filling the medium that transmits it: “our heightened sense of the movement of sound waves, accomplished through the stress and relaxation of a medium, accounts for our impression that sound is created and contingent – mediated – while light is directly possessed by distant objects and permanent.”37 The
connection of sound being perceived as emanating from the object and moving through a supporting medium provides the feeling of immersion that the two-dimensional image does not. Hence, sound is experienced in relation to its movement in time, through space. Branigan exemplifies this idea by noting the prevalence of spatial metaphors in discussing time38, temporal duration being compared
to the material (“it’s been a long day”).
Additionally, Branigan introduces a two-level model of perception applicable to both sound and image that allows for the coexistence of what Chion terms “reduced listening” (a phenomenological attention to sound in and of itself) and “semantic listening” (listening to a code which is to be interpreted, such as music or narrative). Hence bottom-up perceptive processes work on data directly, organizing what is perceived into basic categories such as pitch or timbre. Concurrently, it is possible to apply top-down perceptive processes, working within the framework of the spectator’s expectations and broader cultural understandings, to govern the appearance of more complex information structures. This model will serve us well in discussing the transposition of sound and image, corresponding as they do with analogue and symbolic models of data sonification which shall serve as means to understand the creation of a sense of recognition in the sounds created.
Returning to Levin, his central accomplishment in The Acoustic Dimension39 is bringing to the fore the important role played by the technological apparatus in the
reproduction of sound. This is in contrast to more purely phenomenological approaches to filmic sound wherein the lived experience of audition is given a central position (cf. Christian
37 Branigan, 313 38 Branigan, 1989. 39 Levin, 1984.
Metz40). Such approaches, in failing to fully separate the sound reproduced from its referent
object, render the apparatus and its supporting technological and social practices invisible. Rather than accept the ‘transparency’ or ‘neutrality’ of the audio dimension, Levin exposes it to the same degree of scrutiny as applied to the visual. Here, then, sound is not
(unproblematically) recorded, but reproduced. This reproduction is the terminal stage of a chain of complex and transformative processes of remediation (recording, montage, playback) each of which provides a further degree of abstraction from “the real”. The dimensionality of sound becomes a binding agent providing an “A-effect” a term of Levin’s deriving from Benjamin’s conception of the auratic, which works to obscure the apparatus: “the direct relationship to the collective which is so intrinsic to the phenomenon [of sound] itself is probably related to the spatial depth, to the feeling of being encompassed that envelops the individual, which is common to all music.”41
The ramifications of Levin’s model pertinent to the present discussion are two-fold: firstly, and most simply, the inextricable interrelationship of sound and motion (motion through both time and space) are asserted. Additionally, the directness of sound as experienced as a directness, or lack of remediation is brought into question. Henceforth, the dimensionality of sound is brought under scrutiny as an illusion to be understood in terms of its essential difference to whatever has been recorded, and its essential difference when reproduced by different equipment in different
environments at different times. The importance placed on the apparatus in this model is not in itself revolutionary – Levin himself takes inspiration from the critical stance of Adorno & Eisler’s 1940’s work on the cinematic apparatus42 - but becomes of importance to the present project when
considered in conjunction with Levin’s discussion of the nature of sound in itself. In discussing experimental works completed during the sound cinema’s nascent period, when apparatuses and modes of practice were sought both to explore the limits of the emergent medium whilst attempting to define new orthodoxies, one can see a parallel in Levin’s emphasis on time and motion and the similar adopted by the avant-garde of the time as evinced in the works of Von Doesburg as previously mentioned: sound as the bearer of time, and by extension of motion through space. I posit that such a stance is worthy of reassessment in light of the digital turn, and can be of value in providing for an understanding of the reality effect and indexicality based on cognitive recognition.
40 Metz, 1980 41 Levin 1984, 62 42 Adorno & Eisler 2005
Once the full scope of technological intervention is taken into mind when contemplating both analogue and digital works, the necessity of understanding questions of both indexicality and the relationship of audiovisual material to an objective reality become clear. The shift towards digital media necessitates new attention to the location of the index, not least in the case of the sonic field. In our present moment of increasingly brief technological lifespans and continuous crises of obsolescence, the digital (taken as a broad whole) constitutes a terrain of ever-renewing
possibilities. As a rapidly increasing amount of audiovisual material is converted to binary data, ever-more powerful equipment with which to manipulate said data becomes more accessible. This suggests that conditions of existence for new modes of practice should become apparent, yet parameters and strictures must be observed. Much like the technological and practical convergences that allowed for the emergence of the optical soundtrack, a lack of standardization and orthodoxies necessitates the exploration of the boundaries of the medium. Yet concurrently technical restrictions are enforced from the top-down by industries that define the ways in which these data can be manipulated and deployed. An example would be the inability of contemporary projection systems to adequately project frame rates and aspect ratios common in early cinema but now considered “unorthodox”, for reasons unrelated to technical capacity. In the case of binary data, the apparatus processes a given piece of data on as sound or (moving image) on the basis of instructions
contained therein that describe the type of data in use, with the underlying “matter” being an at least apparently undifferentiated mass of ones and zeroes. The word apparently is used advisedly, here: what is at play here is perhaps the revelation in concrete terms of the increasing layers of
complexity and abstraction demanded by the digital apparatus, a revelation that brings to the forefront of our consciousness the need for systems of encoding that allow us to access and process and audiovisual material.
1.3 Digital Indices and the Fungibility of Everything
At the same time as our sounds and images are being converted to the same mass that is data, and the amount of digital-born media increases exponentially, ideas of indexicality and materiality have faced a radical epistemological challenge. Digitized audiovisual material loses not just its iconicity, but its apparent materiality, the vast apparatus of data centres and fibre optics tending to remain below the threshold of public consciousness. This apparent dematerialization presents the risk of decontextualization and defamiliarization of our audiovisual heritage, and demands new concepts and strategies to understand and navigate this new terrain. In discussing this state of affairs I will borrow from the Media Archaeology theories of Thomas Elsaesser, before introducing the term “fungibility” as a central concept to digital data-handling. Subsequent to this, concepts from the broad field of data sonification, and the intermedia theories of Dean et al termed “algorithmic synaesthesia”43 will be discussed as potential avenues of inquiry allowing us to build
information and then knowledge from the corpora of data increasingly being made available in the digital humanities.
As the digital turn encompasses continuities in both technological developments and creative practices in the broad sweep of audiovisual media, it is preferable to consider it per Elsaesser as an epistemological rupture, a metaphorical staging point at which to pause and both assess how history has been conceived, and how the future is to be conceived44. Speaking
specifically to questions of indexicality, the problem has been couched by Gunning as one of iconicity45, analogue film gaining a large degree of its authority as the representation of truth from
the naked eye resemblance between object and image. Thinking back, it is perhaps worthwhile taking into account Altman’s discussion of same as “half a fallacy working on the other half”46; the
increasingly constructed nature of the modern film image belies the claim to a sacred authenticity inherited from early thinking about photography. Hence the centrality of transformative
technological processes, and the lengths gone to avoid leaving traces of these interventions: the recording, printing, and reproduction of the image exists as a long chain of transformative actions, analogous in many ways to the recording and production of the image. Photochemical film printing is the result of electro-chemical actions to much the same degree as the transcription of patterns of
43 Dean et al
44 Elsaesser 2016, 72-3 45 Gunning, 2007 46 Altman, 2012
light as digital data by way of Charged-couple device. Similarly, before socioeconomic factors greatly increased accessibility and use of the by now ubiquitous hard disk and solid state storage, similar questions could have been asked about the location of the index in magnetic tape. If our understanding of the cinematic apparatus takes into account the complicated collection of historical, mechanical, and phenomenological factors at play on either side of the digital turn, the search for materiality and material traces rapidly becomes one of degrees of abstraction. This has ramifications for the archive in so far as archival documents that pertain to a representation of reality have the nature of this representation brought into doubt as the search for the indexicality of the trace in the iconicity of the likeness comes under question. From such a perspective the notion of the digital as metaphor for how audiovisual material is understood and worked with comes into clearer focus. Taking the question of archival authenticity/proof, it is sufficient to look again to Elsaesser’s discussion of this being a social function, the result of a negotiated process of trust being placed in the institutions responsible for the archive (the example given is that of news photography: our expectations of the publisher outweighs any doubt our knowledge of photo manipulation techniques may raise47)
Again, sound here appears somewhat of an outlier, having never been such a “transparent” rendering: not recorded, reproduced. Much like Moholy-Nagy failed to decipher the tracks of a phonograph record, optically-printed sound cannot be read by the naked eye. One might even suggest that the non-iconic digital image taking on the same obscured visual recognizability represents an opportunity for a level-playing field: was the optical soundtrack a subjugation of the sonic to a carrier designed for the visual, by technical necessity encoded and remediated using optics? What are the implications of this in relation to digitization? Having dismissed the notion of indexicality being “lost”, the impetus is rather to investigate and understand other places in which it can be identified, other ways in which one can discern reference to objective reality and recognize the transformative human creative process.
To explain the presence of the index in digital media I introduce the concept of fungibility. Borrowed from economics, fungibility refers to a characteristic of a given object being
interchangeable or mutually substitutable for another. I propose this term in the context of digital media to acknowledge that whilst the digital turn does not represent a clean break in practices or
aesthetics, but does represent a massively increased availability of strategies of collection, collation, mutation, and transformation of audiovisual media. More specifically, units of data contain exactly this degree of interchangeability: on one level it is possible to open a video file, for example, in a text editor. On a more involved level it is possible to extract numerical data from stock prices and use it as a determining factor in musical composition. More prevalently to the current exercise, numerous applications of digital data can be found in the increasing number of visualization analyses being completed within the digital humanities. The index, then, has become loosed from its attachment to a specific carrier or mode of access, being afforded a transferability that means its location is to be defined by cognitive processes, with a concomitant development of new methods to symbolically encode indexical relationships in a manner amenable to the training of said cognition.
To do this requires new tools of exploring and working with digital audiovisual material, and new methods of reading the media that are clear about the transpositions of data and encourage an understanding of the processes of its production and reproduction. Rather than revert to the ahistoricizing thought that removed the (analogue) apparatus from our view, it makes sense in a world of pervasive CGI and synthesized sound to consider processes of transcription and the nature of the reality effect they can claim. As Manovich places cinema as one event within a wider history of animation48, a contention highlighted by the digital turn (see also Elsaesser’s making reference to
Zielinski’s conception of the era of analogue film as an “intermezzo” in the history of
audiovisions49) , and Levin proffers an explicit link between the epistemological “contamination” of
works such in Pfenninger’s and a post-digitization shift from optic to graphic representation (cf. Increasing prevalence of CGI and greater degrees of post-production image manipulation in the cinema), it seems arguments regarding visual realism risk obsolescence outside of a supporting role when applied to negotiating truth claims to the reality effect.
In the interest of placing the audio dimension on the same plane of importance as the visual, the present exercise emphasizes epistemological constructs that find the locate of the index in the representation of the trace of a trajectory (as per Chion). In this conception the reality effect in cinema is the result of a sense of present engagement in the image (understood in Bergsonian terms as the movement of the projected image, not the static images on the film strip) motivated by its motion. Accepting such a conception seems congruent with both the avant-garde/futurist emphasis
48 Elsaesser 2016, 72-3 49 Elsaesser 2016, 76
on motion that influenced the creators of graphic sound film, as well as the theories from film studies that places sound as the source of movement within the cinema. Additionally, it serves as a convenient and useful base for the transposition of the sonic and visual, as to be discussed in reference to practices of data sonification.
A slightly tangential note is to be made here on the ontology of digitized audiovisual
material as it relates to time. Whilst the image, as discussed, was always recorded as a collection of stills which would be sewn into the whole cloth by a cognitive illusion engendered in the projection process, the optical soundtrack is continuous and hence can be considered analogue technology in the true sense. With digitization, however, sound recording ceases to be such a flow, and is instead sliced into thousands of discrete samples per second (the exact number of which relying on the sample rate applied during the recording or digitization process). In making reference to sound as existing in slices of one twenty-fourth of a second that Norman McLaren50 gives the lie to a subtle
shift by those working in the field of graphical sound film: well before the ubiquity of digital audio, sound was already being subdivided into such discrete pieces by those working with methods drawn from animation. Whilst nominally a matter of convenience (and by no means a universal practice, with some works eschewing the frame-by-frame photographing of the sound track), this represents an interesting semi-arbitrary choice to transfer methods and understandings from visual editing to the sonic. In the perception of sound, diffused as it is, the difference may be unnoticeable, but from the perspective of our understanding of sound creation the ramifications are clear given the
aforementioned importance of sound to the sense of time within the cinema
50 “the sound track has a mosaic nature; in other words it builds up out of small units one twenty-fourth of a second long” (McLaren and Jordan, 224).
1.4 Data Sonification and the Codification of the Index
“Western science has decided it is desirable to isolate the senses in order to study them, but much of much of my work has been aimed at putting it back together” - Bill Viola [date]51
As the aforementioned epistemological rupture of digitization provides a time for both the reassessment of preceding historiographical practices and the contemplation of the new, this section will comprise a review of current practices of data sonification as can be of potential benefit for the auditory display of data culled from the digitized image. In addition to this, reference will be made to models of human audition as a means to understand somewhat processes of cognition at play in the practice of listening, as well as the intermedia works of Dean et al as a source of guidance as to how data can be encoded so as to better facilitate cognizance of its content.
A further contributory factor is the present paucity of theorization and tools for sonic data analysis or the sonic presentation of data specific to the digital humanities. As the digital humanities experiences a growth in the availability and convenience of tools for visualizations in stylistic film theory and others applicable to socioeconomic cinema history (such as GIS)52, making reference to
either the ARCLIGHT Guidebook or the Digital Research Tools Directory53 shows no evidence of
tools relevant to the sonification of (meta)data drawn from the increasing number of corpora being made openly available in the digital humanities (cf. Cinemetrics, ACTION toolkit datasets). Such a situation then necessitates drawing on disparate fields in looking for ways that this paucity may be addressed, with a multidisciplinary approach having the additional benefit of encouraging a reflexive approach to historiography; by paying due attention to the underlying methodological procedures, some insight may be gained into the ways in which historical discourses are
established; this serves to highlight the ways in which practices within the digital humanities both embed themselves in, and are shaped by, archival practice54.
1.4.1 Data Sonification
The “Sonification Report” paper prepared in 1997 for the National Science Foundation by Gregory Kramer and members of the International Community for Auditory Display, is broadly
51 Sagiv, Dean, and Bailes, 295 52 Olesen, 7
53 http://dirtdirectory.org/ 54 Olesen, 10-12.
considered the starting point of sonification becoming a discipline in its own right5556, beginning
from the standpoint that research into data sonification was (at the time of writing) ten years behind that of visualization. Aside from giving an overview of the field as was, a working definition is provided that defines sonification as “the use of non-speech audio to convey information. More specifically, sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation.” (3)57. Whilst
competing definitions have been proffered58, and disagreements exist within the field over the
position of purely creative/aesthetic works created using sonification methods59, the definition of
Kramer et al will suffice for the present exercise, centralizing as it does the transformation of (fungible) data relations, whilst leaving a broad scope for different manners of intervention.
The work of researchers into data sonification comprises a search for the best means by which information can be communicated sonically, and as such provides a good source of information as to processes of audition and cognition pertinent to the present work. This project manifests itself in investigating attributes of the auditory system that lend themselves to data processing, whilst concurrently identifying strategies for the sonic conveyance of specific data relations. Scaletti and Craig formulate these inter-related tasks as such:
55 Worrall, 313-4 56 Dubus and Bresin, 2 57 Scaletti and Craig, 207
58 Cf Worrall 313-4, Dubus and Bresin, 1-3 59 Dubus & Bresin, 2
Illustration 2: Algorithmic representation of data sonification
modelling and analysis programs on the electronic computer are capable of generating vast amounts of data, though not necessarily in an optimal input form for the biological
computer. Our job is to structure this data in ways that will improve the performance of the human observer in interpretation and analysis.60
The primary strength of the auditory system in data processing when compared to the other senses lies in its ability to integrate large amounts of data into a cognitive schema, a process termed Auditory Scene Analysis61; this ability results in the suitability of the audio dimension for the
presentation of high-dimensional data in a manner that does not create an “information overload” for users; this has been theorized as the being the result of the ears always being “open”, and functioning in a multi-directional manner when compared to vision62. Recalling discussions
previous in this chapter regarding sound’s diffuse nature, herein we see perhaps some origin of the brain’s auditory processing abilities: the need to receive and decode a continuous flow of data.
As speculative as this suggestion may sound, a number of researches have been completed in support of its premise that are of note to present discussions about the role of audition in creating a sense of time. Schutz and Lipscomb63 present a number of experiments supporting two aspects of
this role: firstly involving the perceived duration and rate of light flickers when accompanied by audio tones as measured against light alone; secondly, identifying the onset time of flickering lights, again tested with audio accompaniment and without. In both instances audition is proven to
improve the completion of tasks with such a temporal aspect. Further researches support the hypothesis of audition proving remarkably adept at processing of multidimensional data64,
identifying finely-varying and logarithmic data65, as well as a broad general sensitivity to temporal
characteristics such as differentiation between periodic and episodic events, detections in small changes in the frequency of continuous signals, and the identification of patterns embedded in noisy datasets66. Of particular note to the previous epistemological discussion regarding the relationship
between time and motion through space is the assertion by Kramer et al67 that the ability to register
60 Scaletti & Craig, 207 61 Bregman
62 Scaletti and Craig, 207-8 63 Schutz and Lipscomb 889
64 Bly (1985) cited in Scaletti and Craig p. 208 65 Ibid.
66 Kramer et al, 4 67 Ibid.
small differences in timing in concurrent signals is a byproduct of binaural hearing, which results in such differences being subjectively registered as changes in the apparent location of the sound source.
1.4.2 Algorithmic Synaesthesia
Whilst the foregoing research serves well to introduce the strengths of sonification as communicative or expressive practice (or range of practices), further thought must be given to the nature of encoding most useful for the research at hand. If we follow the logic (per Scaletti & Craig) that the primary function of sonification processes is the mapping of data from a source range to an object, the nature of the system of relations that hold between the source must be investigated as the location of the indexical link (210)68. In this instance, provided a recognition of said system of
relations between the two, then the link becomes apparent. However, this recognition can be made manifest in a number of ways, existing on a continuum between the analogous and the symbolic. Scaletti & Craig posit all such relationships as analogous, but herein I introduce a model of
analogous and symbolic as existing on a continuum. Whereas an analogous relationship represents a more direct mapping of a given parameter (or parameters) into the audio field, symbolic
relationships depend on the creation of a more abstracted symbolic logic; a further distinction can be identified in the generally continuous nature of analogous sonification, as compared to those made manifest in discontinuous signals69.
As data sonification represents a broad range of practices not entirely pertinent to the present research, I will draw inspiration for sonification strategies from the intermedia works created by Roger Dean in a manner termed by the creator “algorithmic synaesthesia”. Working specifically with the transposition of sound and image, Dean defines algorithmic synaesthesia as practices “interrelating different media by real-time computer-mediated generation or manipulation of sound and image, in which algorithmic processing in both media has extensively shared features” (Dean et al, 312). The idea of not entirely new: one can propose that the algorithmic basis of a piece of music be considered to its score, or a series of instructions recorded in MIDI format. However, it is on the presentation as a mode of operation for real-time multimedia performance70 that a number of ideas
68 Scaletti and Craig, 210 69 Worrall, 314-15
70 It is worthy of note that Dean makes clear a technological genealogy of his intermedia performance works quite separate from the cinematic focus in this work, encompassing as it does the development of the Macintosh
from Dean’s work resonate with the present research, and can be of greater use in discussing the transposition of the sonic and the visual. Drawing metaphorically from the experiences of
individuals experiencing synaesthesia, an attempt is made to draw out similarities and differences in sensory experiences, and to delineate ways of “enriching our experience of the symbolic codes of specific art forms by engaging with subsidiary meaning through other modes of perception71”.
In sensory terms, Dean emphasizes the cognitive accessibility of rapidly-changing data, which is seen as a function of the plasticity of auditory perception, the ability of the sensory organs to be trained to better process what they are perceiving. In support of this, transparency of
algorithmic strategies is encouraged, with the basing of these strategies in synaesthetic experience presented as a factor in favour of their being easily forming learned associations within the broader cognitive schema.
The proposed outcome of such exercises is to explore the empathetic and anempathetic relations of sound and vision, terms taken from Chion’s Audio-vision to describe cinematic music that either aligns with, or works against, the prevailing emotions depicted on the screen (8). The fungibility of sensory data here plays a key role in the epistemology of these explorations, where the semiotic exchange between media “can be very fertile because each medium ‘means’ in a different way, so that the merging of different media can extend, multiply, and superimpose
meanings”72. Whilst a number of potential strategies based on scientific researches of synaesthesia –
the mapping of synchrony with onset time in images, association of colours with tones or timbres, f for example – the primary matter is the construction of a symbolic logic by which these algorithmic transformations can be expressed and understood. Hence a distinction is made between “zero-order” transcoding, where no data manipulation is made between modes, no structure added to the data signal. Such works lead to an immediate relationship between sound and image in conceptual terms at least: as will be shown in later discussion of practices of databending. An issue highlighted with this strategy is the differences in data structure between different digital audiovisual formats: a video clip will contain a great degree of data not easily applicable to the audio dimension such as the definition of colour spaces. With this in mind I will show that such transcodings are broadly insufficient for the expression of meaningful information, though some interesting facets of the
computer, and subsequent hard- and software developments contributing to the possibilities of multimedia performance (e.g. MIDI, MAX/MSP, Jitter.) (Dean et al, 314).
71 Sagiv, Dean, and Bailes, 305-6 72 Dean et al, 313
apparatus under use can be discerned. Building upon this, then, the suggestion is made that
algorithmic control is grounded in the mapping of parameters across media in such a way that that visual data is used to modulate an external audio signal. The transformative encoding will be used as the basis for further proposals as to how the digital image can be sonified in a manner along the analogous/symbolic continuum whilst retaining the fungible indexical link to reality.
