Mapping Mixed Reality – On Virtuality, Cognition and the Interface

Master’s Thesis

MA New Media and Digital Culture

University of Amsterdam

Mapping Mixed Reality

On Virtuality, Cognition and the Interface

Frederic Vosseberg

Student ID 12324019

Address f.vosseberg@gmail.com Supervisor Dr. Marc Tuters

Second Reader Dr. Gavin Mueller

Date 25 June 2019

1

Table of Contents

2

1. Introduction

1.1. Mixed Reality

“Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts... A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding...” (Gibson 108)

The well-known quote from William Gibson's 1984 science-fiction novel Neuromancer is credited with coining the term cyberspace as a metaphor for both the internet and virtual reality (VR) before either of them existed (3). While this metaphor for a transcendental world of information has played itself out today, a cultural condition can be observed in societies that have advanced the digital revolution up to a point which the above quote might describe just as well. Emanating from the omnipresent screens of smart phones, tablets, computers, and a plethora of other digital media are the light cones of virtual space, appearing as maps, platforms, feeds, timelines, diagrams, wikis, and other interfaces that organise and reduce the complexity of the material world’s data and information. These digital media offer representations of the physical world and orientation therein. This condition of cognition through computation deploys virtuality very differently: While Gibson's trope of cyberspace envisioned how information technology would construct a world of beyond, today's digital media build interfaces to actual dynamics in the immediate earthy environment. This condition is elevated further by new media technologies like augmented reality (AR) that quite literally suggest cognitive extensions through projecting virtual information into the perceptual stream of users.

This situation is the outcome of technological and social transformations over the last 20 years: Personal computing has gone from fixed desktops to mobile devices that unify the phone, camera, map, GPS, internet browser, music and video player, and other media in a single object. Computing has physically moved into body implants,

3 cars, clothes, buildings, cities, homes, and other "smart" objects. This prompts new media researchers to speak of "ubiquitous media" which "pervade our bodies, cultures and societies" and create new fundamental questions about the conditions of knowledge generation (Featherstone 2-3). New media researchers thus look at technologies, software, platforms, interfaces, algorithms, and science and engineering cultures to understand the effects of new media on their own terms. These are less found in the content, stories or opinions they serve to society, but in the logistical efficacy of digital media (Peters 7).

The “new media” of today come with the momentum of the internet's ongoing expansion and transform social, political, economic, and cognitive systems. This has created a cultural condition of global connectivity and interactivity (Featherstone 4), democratised knowledge (Sanger), but also of algorithmic governance redefining knowledge access, subjectivity, or social relations (Barreneche 331, Cheney-Lippold 165, Bucher 30), and of an exploitative economisation of cognitive faculties such as attention (Terranova 1), which some scholars situate in a greater political economy of

cognitive capitalism (Boutang 9) or neurocapitalism (Sampson 10). Others anticipate

the utopian potential of new media technologies like the Internet of Things (IoT), smart energy grids, and artificial intelligence (AI) to realise a new industrial revolution that shifts late capitalism’s paradigms towards a sharing economy, such as American economist Jeremy Rifkin who sees therein the chance to avert climate change (38). Why should these potentials of digital media be observed in terms of virtuality again? I want to argue in this thesis that digital media’s representations of planetary and social environments can dispose their users' cognition of these environments. In other words, virtual environments become representative and referential of actual environments. Thus, this cognitive interposition should be understood, functionally and epistemologically, to discuss how it effects the coevolutionary development of humans, technology, and the planetary ecology. For this, an understanding of virtuality, cognition, digital media’s interfaces, and their mutual interrelations is required which this thesis will provide. Although it will remain theoretical, it leads to various more particular questions about the potential of technologies like VR and AR.

4 As media philosopher John Durham Peters points out, the climate threat caused by the extensive human influence on nature–alienating that concept in the process– encourages to rethink the relationship between media and the environment up to a point where environments should be conceived of as media too (1, 3). Other than Durham, this thesis will not follow an approach that conceives nature as inherently meaningful and media as meaning-transmitters (4), and also not a hyper-reality discourse in the sense of Jean Baudrillard. Instead, it researches constructivist epistemologies that support an understanding of contemporary digital media and their worldly representations as significant epistemological tools for human and nonhuman users.

The thesis will largely follow the media theories of American literature and science scholar N. Katherine Hayles who has prominently historicised and theorised virtuality, cybernetics, and information theory in the 1990s and today ranges among the thought leaders on computational media and posthumanist philosophy in the humanities. Having a background in the natural sciences and an exceptional approach to the interplay between science and literature, Hayles is a vocal proponent of transdisciplinary exchange between academic fields and cultures. The media studies have profited greatly thereof as her take on virtuality has been widely received in new media research during the 1990s (Rogers 19).

On the emergence of ubiquitous media, Hayles remarked in 2010 that a new social condition can be observed that may be characterised as mixed reality ("Cybernetics" 148). She describes how contemporary digital media have moved on from constructing virtual realities as worlds of beyond and instead pervade the physical world with information networks, creating "environments in which physical and virtual realms merge in fluid and seamless ways" (148). This fluency is governed by their interfaces: While the virtuality of earlier media was constructed through the constraints of VR helmets or desktop computers, the new media project through their graphical user interfaces (GUI) virtual information right onto physical objects and environments (148). She points to a tight integration of ubiquitous media with human environments and the cognitive processes therein (148, 151). Mixed reality as a term of art is borrowed from the genre of media technologies that include AR. For Hayles, it seems

5 to describe very well how an abundance of mobile devices, geolocation technologies, extensive broadband networks, and pervasive computing environments converge in a cultural condition of virtual spaces superimposed on the physical world (148).

Her assessment follows from a historiography of the transdisciplinary scientific field cybernetics which over several decades has to varying degree influenced how academic theories regarded the interrelations between humans and machines. In Hayles' view, cybernetics now goes into a fourth phase as its views become important again to theorise new media and digital societies. Moreover, the history of cybernetics itself is an important resource for understanding the present state of information technology as its intellectual heritage is deeply ingrained into computing, engineering, media, and culture (Hayles, “Cybernetics” 155). Among its great achievements are a transdisciplinary language of communication and control based on information theory and the scientific comparability of human and animal organisms with computers, machines, and social systems (152). In the cybernetic view, these very different phenomena, organic life and machinery, become very similar in functional terms. Hayles ultimately sees the internet, VR, and computational media themselves as outcomes of cybernetic discourses (152).

Hayles’ mixed reality diagnosis grasps the cultural condition described above and frames it as a historical consequence of cybernetic thought and culture. However, her descriptive accounts of the condition are quite limited. Thus, this thesis will reconstruct how she arrived at her assessment by researching her academic works of the past and present. The first part traces the history of cybernetics, VR, and Hayles’ theorisation thereof from the 1990s. Involved in this alternate history of computing are touchpoints to philosophical discourses about the constructions of reality and the human condition.

Building on that, the second part reviews Hayles' current theory of nonconscious cognition in human bodies, environments, and computer systems. This may not only demonstrate a fourth-phase-cybernetic theory by herself, but also establish an understanding of digital media’s involvement with cognition. Cognition as a classical object of psychology and philosophy is therein reconceptualised in line with intelligent

6 technologies based on paradigms and findings from the cognitive and neurosciences that will prove valuable to the new media studies. Hayles has presented a revised framework of cognition and consciousness in her book Unthought: The Power of the

Cognitive Nonconscious that will be researched in regards to the mixed reality

condition.

The third part will take a leap towards interface theory to work out how virtuality and cognition are constructed and disposed by digital media’s interfaces. Interfaces are elemental in human-machine-communication and subject of the software studies approach to new media. A comprehensive account of interfaces is given by American design theorist Benjamin H. Bratton whose research connects architectural theory to computer science, geopolitics, and philosophy. Bratton's 2015 book The Stack: On

Software and Sovereignty presents his monumental "geopolitical theory for the age of

global computation and algorithmic governance" (“Benjamin H. Bratton”). Therein, he proposes a view on the global amalgamation of information technology as a converging computational megastructure connecting several phenomenal layers that govern global communications, power, and subjectivity. Therein, he summarises and abstracts digital media's interfaces as logistical communication tools but also as epistemological technologies that present their own interpretations of reality (Bratton 220). In his theory, interfaces are systematic gatekeepers inside a global digital information system. Thus, it offers itself to understand the exchange between physical and virtual worlds.

By connecting Hayles and Bratton, this thesis attempts to illustrate a correlation between virtuality, cognition, and interfaces in the contemporary cultural condition of ubiquitous media and intelligent machines. This helps mapping how digital media can be epistemological and transformative forces on this planet. The significance lies in providing a theoretical frame that shows how the design of media, software, and information systems can cause problematic cognitive programs or support ethical ones for a sustainable human future on the planet. In other words: This is to understand how digital media produce consensual hallucinations and inscribe them into the world.

7

1.2. Research Question

This research takes Hayles’ suggestion of a mixed reality condition as starting point to illustrate how various academic and technological developments have led to the contemporary condition. Her descriptive accounts of this are limited and explicitly stated only in her historiography "Cybernetics" (148). Thus, a part of this thesis will reconstruct this her theories on new media, cybernetics, VR, and virtuality. Central to her supposed fourth phase of cybernetics is an interpenetration of physical and virtual environments. Interfaces are vaguely pointed out as “constraining frame” (148). Italic

mixed reality refers from here on to Hayles' cultural condition to distinguish it from the

mixed reality (MR) class of technologies.

Since her assessment primarily refers to cybernetic theories about new media, this research looks at current new media theories to find resonances with the concept. This pertains to her own 2017 publication Unthought and Bratton’s 2015 book The

Stack. One specific analytical model is presented from both. Hayles’ cognitive assemblage describes interconnections of differently embodied cognitive systems.

Bratton’s interfacial regimes describe how interfaces project cognitive mappings onto the world and indirectly guide its change over time (229, 245). Identifying resonances and contrasts between the two authors and their concepts not only links virtuality, cognition, and interfaces into a synthetic perspective on new media but also contributes to the field a compatibility check of these two important theories.

On one hand, this thesis disseminates and questions notions like cognition or virtual, augmented, and mixed reality by researching their constructions in scientific culture and epistemologies that underlie the involvement of “reality”. On the other hand, it compares and synthesises theories on digital media’s construction of virtual spaces, their embodiment of cognitive activity, and their mediation of physical territory through representation in interfaces. This supposes a technical and epistemological interpenetration of physical world and virtual representation that becomes a social reference for human and nonhuman users of new media. Unfolding this argument follows the guiding research question:

8

How can the interpenetration of physical and virtual environments through digital media interfaces be constructed as a cultural condition?

This phrasing can be read and answered in different ways. Firstly, as an inquiry sparked by Hayles' remark, this thesis traces the intellectual and historic contexts that inspired her assessment. This leads to research into the emergence of virtuality as cultural construction based on cybernetics. Secondly, it analyses Hayles' current work on cognition in regards to how it supports her assessment and illustrates the cognitive potential of technology. Thirdly, this question can be read as research on the cognitive and cultural influence of interfaces. These three perspectives are handled in three parts of the thesis. The research question asks how these perspectives understand and construct these digital media effects as cultural condition.

A cultural condition is understood here as contingent manifestation of culture resulting from a co-evolution of humans and media. This builds on two definitions of media: For Hayles, media store and transmit information (“Cybernetics” 153). Media can be material artefacts or social organisations, semiotic codes or recorded signals. Media encompass computers, networks, interfaces, or the human body, including further media such as DNA, eyes or the brain. I connect this definition to the notion of media

ecology: Classic media theorist Neil Postman distinguished the human environment

into a natural environment made up of air, plants, or animals, and a media environment which consists of symbolic techniques such as language, numbers, images and machines (11). He stressed the fundamental principle of this ecological perspective on media by defining a medium as "a technology within which a culture grows" from interactions between media and humans (10). He believed that media give form to "a culture’s politics, social organization [sic], and habitual ways of thinking" (10). Postman built his definition upon an analogy with the Petri dish wherein cell cultures grow in a substance (10). This establishes a meaningful bridge from the media studies to biology and an understanding of culture as result of interactivity between humans and media.

9

1.3. Methodology

To answer the research question, this thesis follows three conceptual threads from which I will illustrate how the contemporary cultural condition can be understood as

mixed reality. This triangle constellation presents perspectives on new media in a

historical context, in a neuroscientific context, and in a technical (or design) context. Each first frames the perspective, presents literature reviews, and recursively reflects thereupon in partly genealogical, partly comparative analyses. The reflections of later parts take recourse to previous ones and compare the presented theories and authors in regards to resonances and contrasts of concepts and views. This converges in a synthetic conclusion. The first part will consist of literature reviews and mainly genealogical analysis. The second part will combine genealogy and comparative analysis of literature reviews on Hayles. The third part will present a literature review and a comparative analysis between the two main authors, synthesising some of their concepts.

Genealogy is understood in the sense of Michel Foucault as a specific type of history that researches how conditions and notions that appear natural in the present evolved from constituting forces and developments in the past (Prado 33). Foucauldian Genealogy rejects linear histories, origins, and totalising grand themes (33). Instead, it asks for the historic dynamics, powers, and disciplinary authorities that establish the apparent "natural state". Events and developments in this view emerge from the interplay of disparate forces which are then continuously re-interpreted into fabricated narratives by traditional history (34-35). Genealogy looks instead into "effective history" as a series of relational changes that have no constants and totality (40). The genealogic method looks to history from a situated perspective in the present, conceiving knowledge as inevitably perspective rather than objective (41). Genealogy also researches subjectivity as emerging from knowledge production about the subject, historical practices and discourses the individual is involved in rather than natural preconditions (80). To Foucault, the subject's experience of itself and the world is delimited even in before cognition by social categories and behaviours imposed on it by power (80). In this regard, the subject is never universal but the results from conditions that themselves must become object of observation. This research thus

10 includes cultural and scientific constructions and discourses of Western academic tradition in its scope.

I also invoke the concept of a problematic elaborated by French philosopher Gaston Bachelard that has been widely received and implemented in French structuralist and materialist philosophy (Maniglier 21). To unfold a problematic means to not state an already elaborated problem but to establish a perspective from which a set of problems can be formulated (21). This approach inspects a phenomenon by putting it in a defined view that sheds light on a whole range of questions pertaining to it. Knowledge generation then works by interrogating the phenomenon from different angles and working up to more precise questions rather than fixating on strict problems and strict solutions. A problematic questions the questions as much as their possible answers (22). The research question at hand thus asks for how certain constructions of the phenomenon can emerge from different perspectives. Each perspective will thus start out from the research question and frame a different angle, complementing the previous. The choice of perspectives results firstly from Hayles' description of mixed

reality as succeeding the virtuality phase of the 1990s ("Cybernetics" 148), secondly

from searching evidence for the current phase in her recent work which also provides cognitive understanding of new media, and thirdly from her indication that interfaces play a central role in this condition for which contemporary interface theory is imported to this research.

The problematic is self-reflexive as it does not assume or ask for the objective quality of the phenomena, but for their contingent representation inside cultural knowledge systems. Thus, this thesis is foremost analytical new media theories research. It invokes a transdisciplinary approach that understands media studies as a field related to history, philosophy, informatics, design, sociology, cognitive and neuroscience. Theories of these disciplinary knowledge systems then provide the excavation sites for uncovering paradigm shifts, constructed definitions, historical conditions, and technical concepts that explain the research question. The epistemology of the

problematic requires to dissolve the subject-object relation into a recursive question of

knowledge production: Instead of ontological essences of things, there are only representations of representations inside disciplines to be questioned (22). This leads

11 to reframing the view on things and relations that appear natural today, making the

problematic inevitably critical (22). In the Bachelardian view, knowledge pertains less

to conformance with a static reality than to the production of scientific truths (23). Understanding a cultural condition of digital media as mixed reality then needs to understand the scientific constructions of virtuality and reality as well as cognition and their mediation through interfaces.

12

2. Virtuality

2.1. Perspective

This part will establish a genealogical perspective on the cultural construction of virtuality. Virtuality is associated with the condition of computer simulations and emerged from the literary trope and media technology of virtual reality (VR). Discourses around VR have during the 1990s dominated a general understanding of the internet as a placeless virtual realm (Rogers 19). This understanding has given way to the contemporary idea of digital media as it became clear that the virtual “supplement[s] rather than substitute[s] for the 'real'” (Rogers 20). Reflected in this is a historic change in the meaning of virtual following the resolution of misconceptions hailing from much earlier scientific paradigms established in cybernetics and information theory. But virtual spaces still characterise the contemporary social condition of digital media not although but because they supplement the "real". As implied by the research question which asks for the interpenetration of virtual and physical environments, the virtuality that used to be understood as metaphysical quality of computer simulations submits to a different understanding today, describing digitalised representations of physical or social environments in platforms such as digital cartographies, social media, or AR systems. How did the cultural perception evolve that the world can be “digitalised”, i.e. rendered into digital information? While this question would point to the mechanics of digital technology itself, the epistemological interest is better served by asking how a reality could be conceived as virtual. It thus asks how virtuality emerged, or to problematise further, how a condition such as virtuality could be culturally constructed from scientific discourses and literary tropes.

To understand, this part looks at the historical, scientific, and cultural predecessors of virtual reality. It presents historiographies and genealogical accounts of cybernetics and information theory, fictional and technological instantiations of VR, and 1990s media theory. This mainly follows the historic and historiographic works of Hayles from the 1990s, complemented with other authors. This chapter will also lead to the

13 philosophies of constructivism and posthumanism. The goal is to understand how virtuality continues to be important for today’s digital media ecology.

2.2. Cybernetics

To understand virtuality, research has to look back in history to the foundations of cybernetics. This chapter asks for the theoretical and historical grounds of cybernetics, which will echo throughout later parts of the thesis. Cybernetics as an interdisciplinary scientific exchange has a fascinating history that is deeply ingrained in computer technology and culture and provided many analytical instruments to the contemporary sciences and the media studies. The basic concepts, historical phases and disciplinary touch points will thus be presented in this chapter. Hayles herself has given extensive historiographic accounts of cybernetics which will be the main resource for this review ("Cybernetics" 145-56; "Boundary Disputes" 441-67). Additionally, the works of Austrian-American physicist and philosopher Heinz von Foerster will be presented who was central in the paradigm shift of second-order cybernetics that elevated cybernetics into a reflexive epistemology.

The history of cybernetics can with Hayles be delineated into four distinct phases that revolved around different central concerns ("Cybernetics" 147-48): Emerging as a controversial scientific field of control systems at the end of the second world war, it established the validity of man-machine comparisons. Between the 1960s and 1985, cybernetics evolved into a second-order paradigm, shifting its focus to the reflexivity of observation. After a quiet while, it would re-emerge during the 1990s revolving around virtuality. After vanishing again at the millennium, cybernetics finds novel importance today for understanding the relationships between humans and digital media.

Cybernetics has been prominently presented as the science of "Control and Communication in the Animal and the Machine" by its pioneer mathematician Norbert Wiener's famous book (Hayles, "Cybernetics" 145). As this definition implies, cybernetics sees a structural equivalency between biological organisms and

14 technological systems. Before Wiener coined the term "cybernetics" when referring to the study of control mechanisms in all systems, the early conferences held in the 1940s carried the name “Circular-Causal and Feedback Mechanisms in Biological and Social Systems” (von Foerster, "Responsibilities of Competence" 192). Later they would become known widely as the Macy conferences, the central stage of the early cyberneticians. Another definition of cybernetics was given by psychologist Gordon Pask as "the field concerned with information flows in all media", a definition that expresses the essential role of cybernetics for the media studies (qtd. in Hayles, "Cybernetics" 145). The cybernetic view unlocked systems from their material instantiation and abstracted them to their functional structures. This allowed them to observe how information flowed inside the system and how control mechanisms could be observed or introduced. This view came to Wiener from his work on anti-aircraft weapon systems during the war and would later inspire thought about other types of symbiotic man-machine organisms (Rid 49). The early Macy conferences discussed the scientific validity of such views by means of simple robots that mimicked thinking or living in their view (Hayles, "Cybernetics" 146).

From the beginning, cybernetics unravelled the functional mechanisms of control and communication in all systems and how these would be translated between systems which prominently implied that man and machine could be regarded as interconnected information systems, an idea popularised by Wiener as "servo-mechanism" (Rid 49). From this, cybernetics anticipated the replacement of lost limbs with artificial prostheses that would substitute sensory nerve impulses with electrical signals. Psychiatrist Ross Ashby noted that the materiality of the limb was not the defining feature of its function (Rid 19). Thus, cybernetics questioned whether the body was not just as much the environment of the brain as the external world. The boundary between system and environment became a matter of observation. In this tendency of displacing boundaries Hayles sees a characteristic feature of all phases of cybernetics ("Cybernetics" 149).

The central cybernetic instruments for mapping information systems are the feedback loop and information theory. Feedback loops describe how the output of a system is fed back into the system's activity, amplifying or opposing the activity of the system

15 (Rid 49). One tangible example for this is the shriek noise loudspeakers give when their output signal is captured by a microphone that is routed back into them. Early on, cybernetics adapted Claude Shannon's mathematical model of communication which defined information through the probabilities of its successful transmission but excluded the meaning and context of the message entirely (Hayles, "Boundary Disputes" 448). Information in this formal construction was diverging from the conventional conception that information is meaningful, Hayles points out, but it allowed widespread usage of the theory in physics, electrical engineering, computer science, and beyond (448). In the combination of this quantifiable model of information and the idea of the feedback loop, Hayles sees the foundation of cybernetics ("Cybernetics" 145).

The problem of observation also involved the unpredicted complexity that feedback loops would generate in recursively networked systems and two paradigms competed during early cybernetics for how to handle it: reflexivity and homeostasis (Hayles, "Boundary Disputes" 446). Highly complex systems would react unpredictably to slightest changes of parameters through feedback loops and cyberneticians disagreed on whether to observe this as problem for stability or as pointer toward openness. Hayles argues that the homeostasis paradigm privileged predictability while the

reflexivity paradigm anticipated complexity as open horizon ("Boundary Disputes"

446). In her view, this competition also reflected a rivalry of conservative and progressive values. While the homeostatic view dominated early cybernetics, it was largely given up in light of the paradigm shift towards second-order cybernetics summoned by the insight that systems exhibited different behaviour depending on how or by whom they were observed (Hayles, "Boundary Disputes" 446). The observation of the system thus had to observe the observer as well.

Reviewing the second-order foundations will help understand how cybernetics became epistemological and thus decisive in the emergence of virtuality as an epistemic quality. Hayles marks its begin in 1960 with cybernetics redrawing the boundary of systems to include their observers ("Cybernetics" 147). A key figure in the new and more complex paradigm was Heinz von Foerster who published on cognition, logic, and physics, often with punning titles like Observing Systems or Understanding

16

Understanding. In "Cybernetics of Cybernetics", he remarks: "we do not see that we

do not see" (284), drawing an analogy from the blind spot in the human eye to the cognitive bias he saw rooted in the (particularly Western) delusion of objectivity (285). Given the imperative that "properties of the observer shall not enter the description of his observations”, von Foerster asked how making descriptions is possible if the observer did not have properties that allow for descriptions in the first place (285). This led him to postulate a self-referential theory of the observer built on infinite recursion (285). Von Foerster understood epistemology as theory of knowledge acquisition and cybernetics as its appropriate conceptual framework ("Epistemology" 229). The "problem of cognition" was for him the search for an understanding of the cognitive process, a need arising from the observation that "the physical nature of the stimulus [...] is not encoded into nervous activity"; that the fabulous experience of a colourful reality was apparently produced in the nervous system and not in the world ("Constructing Reality" 215).

“‘out there’ there is no light and no colo[u]r, there are only electromagnetic waves; ‘out there’ there is no sound and no music, there are only periodic variations of the air pressure; ‘out there’ there is no heat and no cold, there are only moving molecules with more or less mean kinetic energy, and so on. Finally, for sure, ‘out there’ there is no pain.” (Von Foerster, "Constructing Reality" 215)

Von Foerster conceptualised the internal construction of a reality as a "computational" process (“Constructing Reality” 215-16). He used the term for any cognitive operation on observed objects or their representations, pointing to the lack of numerical reference in the Latin origin "computare": "contemplating" or "putting things together" (216). He saw the process of recognizing a visual image as a sequence of neurophysiological transformations beginning with a projected image on the retina and continuing with subsequent reiterations on each new stage of neural activity (216). By semantically reducing the process, he concluded that reality "appears only implicit as the operation of recursive descriptions" and proposed to understand cognition as infinite recursive computation processes ("Constructing Reality" 216-17).

17 Von Foerster's view fell into the philosophy of radical constructivism that frames cognition as inherently independent of the reality. He thus emphasised the distinction of referring to a reality or the reality ("Epistemology", 232): The reality could only be spoken of by assuming that different sensory perceptions confirm each other in representing the environment as it is. In his constructivist epistemology, however, sensory perceptions could only correlate and cognitively compute and model a reality internally. He contested that the reality is at best a special case of a reality (232). "The environment as we see it is our invention", he claimed ("Constructing Reality" 212). His philosophy did not question the existence of the reality as in solipsism, however (226). Rather, internal representations of the outside world inevitably emerged in the brain from recursive computations of sensory information in the system's own language ("Epistemology" 232).

Other theorists of second-order cybernetics that developed disruptive epistemologies were Chilean biologists Humberto Maturana and Francisco Varela who theorised about autopoietic and allopoietic systems. While allopoietic systems served an external purpose, such as a car that serves the transport of a driver, autopoietic systems served foremost their self-organisation and self-reproduction, such as all living things (Hayles, "Boundary Disputes" 462). In their view too, organisms were formally closed systems that only connected to their environment through structural

couplings with other formally closed systems (Hayles, "Boundary Disputes" 442).

Epistemologically, this meant that recognition of outside objects worked because these objects had to be other systems that happened to irritate the sensory and cognitive apparatus of the autopoietic system, triggering representation in the system's own language inside. In simple terms, an observer sees a tree not because it is a tree but because it sheds a particular light on the eyes which triggers in the observer's brain the learned concept of a "tree".

This view built the foundation for theories of virtuality in which cognitive systems were immersed into virtual environments, i.e. structurally coupled to a computer (Hayles, "Boundary Disputes" 465). From the constructivist notions of reality also follows why this may be conceived of as a virtual reality. Reality in this cybernetic view is understood as internal construction of a system. In sociological cybernetic theories

18 such as German sociologist Niklas Luhmann's systems theory which picked up the

autopoietic view, social systems such as society and its subsystems reproduce

themselves through recursive communications and construct their own internal realities as self-referential narratives through correlation of events with probabilities stored in social memory (Massenmedien 15). Social reality then becomes a projection of society’s media as a functional self-observation (118-19). Luhmann's modernist theory aspires to universality as it postulates a world society composed of the totality of autopoietic communications that determine any observations inside this system (Intransparenz 18). It is perhaps his failure to recognise computers as game changers of communication that make his theory less well applicable to new media even though it still has an enormous standing in German media theory.

This chapter has presented the foundations of cybernetics and its essential tools as scientific grounds for virtuality to arise from: control, feedback loops, information theory, reflexivity, functional closure, structural couplings. It has also introduced the

observer as a reflexive epistemological subject and distinguished notions of reality: an

ontological one (the reality) and a constructivist one (a reality; cognitive or social). These will help understanding virtuality and cognition later on. To get there, a historical perspective on VR will be presented next which conjoined with cybernetics explains the discourse on virtuality in 2.4.

2.3. Virtual Reality

This chapter provides a brief historicisation of VR to serve as a frame in which theories on virtuality can be understood. This history touches upon the general history of computers that will not be covered here. Instead, this chapter inspects selected cultural factors and asks what precursors and preconditions led to historic images of VR. Throughout the history of VR, visions usually outperformed realities by far. The revival of VR in the last decade however comes in an entirely different scale of computing power and immersion that needs to be distinguished from the historical image of VR elaborated in 20th century academia and science-fiction.

19 Early precursors of VR could be located at various points in time, some predating the 20th century. Leighton Evans has connected it to Robert Baker's Panorama in Georgian Britain of 1787, a spectacular 360° painting of Edinburgh that would immerse visitors into a different place by obscuring the limits of the painting and deceiving the senses (17). In 1838, Sir Charles Wheatstone's invention of the Stereoscope, a small device that would project the illusion of a three-dimensional image inside by showing the eyes two slightly displaced angles of the image would bring total immersion from the public to the person (Evans 19). A 1935 science-fiction story by Stanley Weinbaum featured a pair of goggles that would transport the main character's senses into a holographic world, a story closely resembling the contemporary vision of VR (Barnard). However, it was not until the 1950s that computational systems such as Morton Heilig's Sensorama were invented that showed stereoscopic full-colour videos with audio, haptic and even smell stimuli (Barnard).

When the 1960s brought forth computers and cybernetics, visions of fully immersive and interactive virtual spaces accessed through head-mounted displays (HMDs) would be formulated by computer scientists like Ivan Sutherland (Barnard). Sutherland, a computer graphics pioneer, dreamt of The Ultimate Display framing access to a simulated world where concepts beyond physical laws could be studied– "a looking glass into a mathematical wonderland". He developed the first HMD for VR entitled The Sword of Damocles in 1968 which adjusted perspective to a wire-frame space according to the user's head movements (Barnard). The rather primitive device was too heavy to carry for long, and so it took a few more decades before VR escaped these experimental conditions (Barnard). The first VR devices on the consumer market were produced in the 1980s: The iconic DataGlove and the EyePhone were developed by pioneering firm VPL Research whose founder Jaron Lanier coined the term virtual reality (Barnard). In the VR hype of the 1980s and 1990s that followed, the flourishing video game industry repeatedly experimented with VR. However, the technical gear was expensive and inconvenient. The costs of VPL's EyePhone HMD around 1990 was about 10.000 USD and it could barely generate five or six frames per second, whereas fluid television imagery operated with 30 (Evans 3; Ellison).

20 But the futuristic vision of cyberspace as a virtual realm outside of the real world was already well elaborated in science-fiction. Iconic author William Gibson introduced the term cyberspace in his 1984 novel Neuromancer (Naughton). In his vision of cyberspace, humans could link in via human-machine-interfaces connecting the neural signals of their sensorium to a network of computational systems that would simulate a vast virtual environment appearing to billions of collective users as "consensual hallucination" (Gibson 108). Hayles points out that Gibson's vision of

cyberspace was influential in the development of actual VR imaging software (Posthuman 21). This idea of cyberspace was frequently reiterated in the emerging

literary genre of cyberpunk. As the prefix "cyber" indicates, cyberspace, cyberpunk and cyberculture originally referred to cybernetics but this meaning gave way to vague associations with computers in the course of its popularisation (Naughton).

During the 1990s, cyberpunk and VR became deeply enmeshed in pop culture with Japanese cult movies such as Ghost in the Shell (1995) and Hollywood movies such as Jonny Mnemonic (1995), Hackers (1995) and The Matrix (1999) taking it to worldwide cinemas. But just as the latter topped the box offices, VPL Research filed for bankruptcy as the turn of the millennia saw interest in VR quickly dwindling (Evans 3). VR technology could not live up to the expectations popular culture had established and investors shifted their focus to the rise of the more promising mobile technology (Ellison). The internet also lost its placeless appeal: New media researcher Richard Rogers dates the death of cyberspace to the year 2000 when French lawsuits against Yahoo! led internet providers to detect and sort web traffic after geolocation (39). The 2000s saw mobile phones gradually become ubiquitous and smart and the internet expanding in coverage, connectivity and users. VR remained silent for a while but re-emerged in the 2010s following the development of Oculus Rift, then its $2.3 billion acquisition by Facebook, and similar developments by competitors Sony, HTC, Samsung and Google (Evans 3). Especially the extremely affordable paper-made Google Cardboard priced at $10 generated mainstream interest in VR applications for mobiles. The comeback of VR also revived its existential component in pop culture. The trope of cyberspace reappears in movies such as Ready Player One (2018) connecting it to mildly served social critique.

21 In light of massively increased computing powers, today's VR devices come much closer to the visions proposed by pop culture. Other than in the classic cyberspace trope, VR now comes to many applications for communications of the "real" world such as in news reporting. The New York Times published digital journalist Ben Salomon's VR film The Fight for Falluja which documents actual war zone happenings in the Iraq war by following the Iraqi army taking back the town of Falluja from ISIS. VR enables "immersive journalism" to take viewers into conflict zones from a first-person perspective. Other applications see VR used in trauma therapy such as in Skip Rizzo and Arno Hartholt's Bravemind project that treats PTSD in war veterans with immersive technology (USC). Because a VR experience can disembody subjects and change their perspectives to any situation, VR can significantly improve empathy towards other individuals or social groups, studies find to varying degrees (Herrera et al. 1). The effects of embodiment and disembodiment are also being researched for subject's relationships to their own bodies, making use of the body transfer illusion that emerges when subjects treat their virtual avatar like their own material body (Markowitz and Bailenson 6). Research has thus found that embodying a different skin colour in VR can reduce racial bias (Peck et al. 2) or eating disorders (Clus et al. e157). The future of VR yields promising social value, it seems.

This chapter has presented how the historical image of VR was prefigured by 19th century immersive art installations, early science-fiction, and 1960s interface theory in coincidence with cybernetics and computer technology. 1980s cyberpunk literature, in particular William Gibson, have painted a picture of VR as collectively disembodied otherworld that inscribed itself into early VR technologies and VR receptions in 20th century pop culture. A different picture forms from contemporary research that applies VR in medical, journalistic, and educative contexts outside the entertainment industry and points to VR's potential as supplement for social causes. Even though VR experiences remain isolated, the cultural picture is less defined through otherworldliness.

22

2.4. Constructing Virtuality

The historical correlations of cybernetics and VR converged in a theoretical discourse about virtuality itself that will be covered in this part. Understanding the development of this meaning builds the foundation for understanding how the contemporary condition of new media echoes this in a way that can be constructed as mixed reality. This chapter gets through to the question of how virtuality was culturally constructed from scientific discourses and literary tropes.

VR offered fascinating challenges to intellectual thinking across the academic landscape. In the humanities, the particular idea of linking the human mind to a virtual environment was picked up to investigate how the relationship of mind and body revealed itself as cultural construction in this experimental condition. So did Hayles in her numerous publications in the 1990s that linked VR to cybernetics and information theory. She summarised her theory of virtuality in her book How We Became

Posthuman, one of the most important works next to Donna Harraway's A Cyborg Manifesto that developed a posthumanist theory based on cybernetics.

The discourse around virtuality was so influential during the 1990s that Hayles propagated a third wave in the history of cybernetic thought following second-order cybernetics ("Cybernetics" 148). This was not understood as third-order cybernetics but as a revived interest in second-order cybernetic theories. She saw the cybernetic impulse continued in the pertinent questions on boundaries between mind and body that VR brought up and in the implicit conception of the organic body as a medium that processes information in connection with other information media such as computers, VR systems, or the internet. The construction of the body as information medium then became the basis for her much of her subsequent work.

Virtuality as a cultural condition emerged in Hayles' view when the cybernetic idea of the feedback loop was combined with the increasing power of microcomputers and resulted in immersive interfaces on the consumer market ("Boundary Disputes" 442). The interconnection of the human cognitive apparatus with computationally simulated worlds made particular sense in the cybernetic view: Through recursive visual and spatial information flows between these materially different yet functionally similar

23 systems, a virtual environment composed entirely of information could be projected into the sensorium. Following this paradigm, it seemed like the human mind could be disembodied from the material body and translated into information. This resonates in the literary idea of cyberspace where disembodied subjects may take on any desired body. Hayles points out that this classic idea suggests virtual reality as a world distinct from the "real world" that is accessed through the constraints of human-machine-interfaces like VR goggles ("Cybernetics" 148). This differs from the contemporary condition of new media where virtual spaces (such as social media platforms) are often linked to actual locations, persons, or products. Hayles disseminated this paradigm in 1994 and traced it back to the conceptual construction of information during the early days of cybernetics. Ever since the 1948 Shannon-Wiener model of information theory was adopted, information was per definition unlocked from meaning and matter and instead quantified as a probability function of successful transmission (Hayles, "Boundary Disputes" 464). Hayles assumes that Shannon's model strategically excluded changes in a receiver's mind state from the definition of information to make it more flexible and efficient, whereas other historically competing models framed information in a reflexive relation to the embodied receiver (448-50). Still, much of computer science, engineering, or psychology today uses the Shannon-Wiener model and its strategic disembodiment of information from material context. This disembodiment of information had far-reaching implications. It allowed information to pass through changing material contexts and enabled the human mind to be conceived of as a recursive network of information flows that might be represented in a sufficiently sophisticated computer (464). In this regard, virtual reality seemed to be a step toward transcendence of the mind from the body. Hayles explains with this construction that reduces consciousness to an information pattern how scientists like physicist Hans Moravec suggest that human consciousness could at some point be downloaded into a computer (Posthuman 1). Hayles sees this scientific construction as a remnant of the ancient distinction of form and matter in Western philosophy which is found in Descartes' mind-body dualism and even earlier in Plato's allegory of the cave ("Boundary Disputes" 464).

24 The fundamental distinction of information/materiality is, however, foremost a scientific construction that Hayles explained with scientists' tendency to privilege the abstract over the instance, and the human desire to find transcendence from the mortal body (Posthuman 13). She objected that information could only ever exist instantiated in a medium and that a virtual world could never be self-sufficient as it is rooted in the materiality that it conceals ("Boundary Disputes" 443). The simulation only comes accessible through the material cognitive system that perceives it through the cybernetic circuit. The simulation can also only run on materially assembled computers. Therefore, the implicit disembodiment of VR is a misconception (464). No one can really be disembodied as the very bodily features are the only way of perceiving and interacting with virtual environments in the first place. To the cultural moment of the 1990s, however, the displacement of the material world through a virtual world seemed possible (443).

Following these circumstances, Hayles redefined virtuality as "the cultural perception

that material objects are interpenetrated by information patterns" (Posthuman 13). In

this definition, she strategically frames the duality of information and materiality in a way that credits its viability while simultaneously accounting its construction in scientific discourse. “Virtual reality” in this view implies that a world of information might be superimposed onto a material reality. But the definition is also applicable to a wider cultural perception of objects or environments as pervaded by information such as biological bodies by DNA or cities by broadband network signals (14). Virtuality in this view does not only exist in a simulated environment. Thus, it becomes a new question whether a reality is not inherently virtual in the constructivist sense as it is observed, described and represented through cognitive processes computing information on various levels. If cognition works through representative levels of information, where could the line even be drawn between the reality and its reception as information pattern?

Hayles saw in VR a perfect exemplification of the structural couplings between human brains and their environments wherein a computational feedback loop was interposed (465). The VR headset would make it obvious to the user that their perceptions were mediated by interfacial chains and their sensorium attached to a "prosthesis" (465).

25 Hayles posited that this view could be taken to the extreme so as to regard the body itself as prosthesis that creates mediated perceptions (466). Even though she argued that such views disregard the evolutionary significance of the material body, she adopted a similar view for her theory of the body and embodiment ("Flesh and Metal" 297). Being able to temporarily disregard the biological and social limitations of the body in VR exposed the extent to which the experience of the body was constructed by these.

She suggested the conceptual pair of body and embodiment as a spectrum between the outside perspective and the experience from the inside in an attempt to offer an understanding of the mind as inevitably embodied ("Flesh and Metal" 297). The body here is a changing but always culturally constructed ideal while embodiment is experienced through emotions and sensations emerging from "complex interactions between conscious mind and the physiological structures that are the result of millennia of biological evolution" (297). Both are subject to change relative to culture, Hayles concluded from the adaptability of the human neural system to different environments and the immediate representation of external objects used as tools or interfaces in the brain ("Flesh and Metal" 300). Virtuality in this view can significantly alter the body and embodiment which leads to the question of whether ubiquitous media construct ubiquitous virtuality that significantly changes how digital media users experience their bodies.

In Hayles view, this may follow as she resolves the mind-body dualism into a single

mindbody entity that emerges as a biologically evolved actor from bidirectional

feedback loops with its technological environment (303). She also assumes that the increasing ubiquity of media and technology raises the intensity of this co-evolutionary loop (303). She exemplifies this with the evolution of language which changes the human brain structure but in return evolves with changes in that same brain structure (303). Similarly, computer interfaces evolve with their users: While early DOS interfaces were strictly catering to the cognitive apparatus of scientists and engineers, the later, more intuitive interfaces of personal computers were better adapted to the processes of the human brain (303). These thoughts fall into Hayles' evolutionary

26 perspective on humans and technology that she explains as technogenesis ("Cybernetics" 154).

For Hayles, the human condition changes with its relation to technology. On these grounds she then built her philosophy of posthumanism that proposes a fundamental critique of enlightenment humanism. Philosophical humanism propagated a persistent worldview which privileged individual human autonomy, secular freedom, and mastery over the world and the body along with dominance of rationalism (Hayles, Posthuman 3). Posthumanist and postmodernist theorists see this liberal humanist subject as universalised construction of human subjectivity complicit in the emergence of imperialism, colonialism, and capitalism (4). For Hayles, this subjectivity became untenable in the face of contemporary science, the technogenetic condition of humans and computers, and the implications of virtuality. Especially the scientific findings of evolutionary psychology and neuroscience revealed that autonomy, rationality, individual agency, and free will are questionable concepts (319). Posthumanism thus dismisses the autonomy of humans and focusses on human symbiosis with biological and technological environments through evolutionary feedback loops (2). Hayles' particular articulation of posthumanism builds on the disembodiment of systems (and information as such) from material form, explicitly privileging information pattern over material instantiation and therefore regarding the body as "original prosthesis" and its biological embodiment as an "accident of history" rather than an "inevitability of life" (2).

Therein then also lies the ground for a critique. As a cultural construction offered as replacement for a previous one that was problematic in its justification of dominance over the body, nature, and other humans, it refers to mutual interdependencies of humans and machines, decentring the human autonomy. However, emphasizing the human co-evolution with technological environments over the evolution inside biological environments paradoxically makes human agency towards the environment prevalent again. How does the posthuman subject transcend the culturally constructed disconnection from its biological environment by constructing itself as situated foremost in coevolution with machines–which humans could only invent due to their biologically evolved cognitive capabilities from interactions with and observations of

27 their biophysical ecological environments? How does this posthuman subject reconfigure human consciousness in a way that builds awareness of the complex environmental interrelations when it proposes a machine-defined subjectivity? These questions suggest the view that the human body is deeper integrated into symbiotic relationships with biological ecologies than with technological media and that this relation should be made more conscious than it is in order to maintain a planetary ecological balance. Space and topic do not permit a comprehensive response to posthumanism here and Hayles does certainly not disregard these things, but it cannot be stressed enough today how significant human involvement in the planetary ecology is.

To conclude this chapter, posthumanism is to be situated as a philosophical deduction from the 1990s discourse around virtuality that moved on from understanding it as an alternative to the reality to a supplementary construction of a reality. The VR discourse revived interest in second-order cybernetic theories as it implied that the human brain and body were information media that could be structurally coupled with computers, disembodying the mind and situating in computer-generated worlds. This disembodiment made the material body and world seem replaceable; a misconception grounded in the axiom of Shannon’s information theory framing information as not specified by its material vehicle. The cultural construction of this axiom became problematic as no cognitive exchange between minds and virtual worlds could really work without including the biologically evolved and embodied body or material components of computers in the cybernetic circuit. Nevertheless, its perceived replacement exposed cultural constructions of the body. Virtuality thus had to be defined by Hayles as a cultural perception seeing the material world as pervaded by information patterns. On these grounds, she developed a posthumanist epistemology articulated to fit in a coevolutionary setting she calls technogenesis wherein humans evolve in relation to their technologies.

28

2.5. Reflection

This part problematised and genealogised the condition of virtuality. It demonstrated through historical perspectives on VR, cybernetics and Hayles’ works of the 1990s how virtuality was understood in different ways throughout history and how the paradigms of information theory and cybernetics have constructed these understandings. In summary, the vision of VR was from the outset led by science-fiction literature and ancient Western transcendental phantasies. This resulted in a discourse orientated by the trope of cyberspace as alternative universe composed of pure information wherein disembodied minds could enter through human-machine-interfaces constraining their environmental and bodily perceptions. This view fell in line with axioms of first and second-order cybernetics that reduced man and machine to information processing systems, allowing functional interconnections through feedback loops of information between cognitive and computer systems.

The recurring boundary disputes of cybernetics led to displacing the boundary between a system and its observer at some point. Thinkers like von Foerster questioned ontological notions of the reality and suggesting instead constructivist notions of a reality that only correlated with others rather than detecting the objective reality. Virtual reality then became a discourse of its own as it proposed an alternative reality and carried over the transcendental appeal that science-fiction and early computer science and interface theory imagined. This coincided with early developments of the internet in the 1990s that seemed to affirm this transcendence. The discourse around the substitution of the material world and body through virtual ones was consequently reflected in cinema and pop culture.

It had to become clear though that virtuality must be rooted in materiality and could therefore only be understood as a cultural perception. This perception was implicit through the disciplinary authority of Shannon’s information theory that had disembodied information to make it functional. As Hayles has prominently pointed out, this construction sufficed to put other cultural constructions into question such as the social ideals of the body based on liberal humanist subjectivity. Virtuality therefore

29 became its own cultural condition of perceiving the world with media technology as pervaded by information patterns.

While she took this outset to develop a posthumanist philosophy for the technogenesis of humans and intelligent computers and defined a posthuman subjectivity, it may also be taken into a radical constructivist perspective to ask whether cognition is not inherently virtual as it processes environmental information through several layers of embodied representation. The VR experience is then only one more feedback loop of cognition routed through a computer-generated environment. This would frame all realities (as plural of a reality) inherently virtual and displace the reality beyond the horizon. This view may follow from von Foerster’s computational definition of cognition. When setting today's digital media into this perspective, they introduce more cognitive feedback loops between observer and environment–if technical information processing is accepted as cognitive activity. This problem will thus be handled in the next part. In other words, they would introduce more virtual layers of a reality. Since accurate representation of the reality could not be possible, any totalising representation must be regarded as virtual construction. Representations then follow from other representations, and the question of construction must turn to the interfaces that represent them. As Hayles noted earlier, the construction of virtuality is constrained by the interface of the medium and today's media construct it through GUIs rather than VR helmets ("Cybernetics" 148). This points to integrating contemporary interface theory which will be done in the third part of this thesis. From the perspective established in this chapter, the thesis' research question could be answered so far by pointing out how virtuality was constructed from scientific and literary discourses as cultural perception that seemed to naturally allow disembodiment of the mind. This reflected itself in phenomena like cyberculture. Sorting out misconceptions in these discourses, Hayles redefined virtuality and laid the grounds to understand a contemporary cultural perception of virtual space pervading the actual, physically embodied world. This new concept also makes understandable how digital media today seem natural in their representation of the world through illustrating the information patterns therein.

30 However, these representations disembody information patterns from their original physical embodiment and simulate it inside a differently embodied computational medium. It is thus a virtual environment composed of data captured from the physical world, a virtual reality that correlates with others and thus appears as the reality. Do digital media thus interpose a virtual envelope between the physical world and human cognition? This will be illuminated in the Interface part. It already gives an indication of why Hayles would observe the contemporary cultural condition as mixed reality.

31

3. Cognition

3.1. Perspective

Having understood virtuality as a cultural construction and condition illuminates how today’s digital media go beyond the concept and frame their perspectives as representations of the actual world. Cyberspace was the product of a utopian vision of information-based simulated worlds, but also of a misconception rooted in the functional definition of information given by Shannon’s information theory and propelled by cybernetics. The problem therein was disembodiment. Information really could only be disembodied in theory, and so virtuality could only be a metaphysical quality in cultural perception. Today’s digital media, however, mediate knowledge of the physical world and organise interactions therein by representing it in virtual projections.

This convolution complements each other: The physical world is mapped as information structure into virtual models and these virtual representations reference and designate the physical world. The result is a media ecology of information platforms that rationalise and accelerate communications and redistribute cognitive activity: memory, orientation, navigation, attention, and other conscious or unconscious cognitive faculties are externalised to digital media’s databases, algorithms, and interfaces. This part will illustrate that in a perspective on cognition, which requires to question the notion of cognition as such. Psychology and neuroscience have traditionally researched cognition in the organic brain and nervous system. This view seems increasingly questionable in an age of intelligent machines. The problematic of the research question thus shifts to how cognition can be conceived of in mixed reality environments. If digital media support human cognition through additional feedback loops of information between system and environment, do they not become part of the cognitive system? The expansion of AI amplifies and complicates this question: As autonomous agents such as self-driving cars and virtual assistants emerge, encouraging a posthuman perspective on this problem, it becomes ambiguous which cognitive system these media become part of. How can the