Drama, Performance and Digital Multimedia

Drama, Performance and Digital Multimedia

Introduction

Few months ago I was asked to agree to an Arabic translation of my essay on Theatre and digital multimedia (Marsilio, 2003). The request came from the 19th Cairo International Festival for Experimental Theatre that translates a foreign essay for each edition of the Festival. I was honoured, and immediately started to update the essay, sending the chapters to the translator. After the first chapter was sent, I received an email enquiring if I was interested in organizing a round table during the Festival, and to discuss the topics of my essay. The idea of confronting the scholars and practitioners from the Arab world and other countries was very challenging, and I went ahead without delay.

That is why on the 5th of September I held a conference session on Drama, Performance and Digital Multimedia, followed the next day by a workshop on the same topics.
I have to thank Prof. Fawzy Fahmy for inviting me to organize the round table, and overall Josephine Anstey, Christina Finger, Vincenzo Lombardo, and Emanuele Quinz, for their essential contribution. Here the reader will find all the talks.

The round table has had a great attention both form scholars and media. It has been very intriguing to understand how the experimental theatre is a sort of international language, a kind of open source code shared by intellectuals and artists coming from all over the worlds and from all different cultures.
The fascination – together with the fear – for the use of digital language in performing arts has been the common ground on which the discussion as developed. Besides the different point of view or the single aspirations, it has to be underlined that the Ministry of Culture has shown an interest in discussing the topic.
We are still waiting for the blooming of such interdisciplinary discussion in Italian academia and theatre institutions.

Antonio Pizzo

Introduction to Drama and Artificial Intelligence

Antonio Pizzo, University of Turin, CIRMA

This round table has been designed to provide a broad approach to the use of digital technology in theatre and performing arts. The issue about the relation between theatre, performance and new media, has been arisen since the Eighties, when contemporary dance started to introduce robots and electronics on stage, or when fringe theatre investigates the relation between live event and TV camera and monitors.

Since then, much has been written about it, so that it is now possible to follow different approaches to the question.

Here the panellists will talk about different topics: new dance and new media, the theatre and virtual reality, the simulation of space and time in virtual representation, the interactive stage and costumes. All of these are topics that deserve a full length discussion, and certainly this is just a start, and I invite whoever is interested to come and join us at tomorrow morning workshop, at 11 am, here at Supreme Council for Culture, for further details.

The whole text [PDF, 380 KB]

[This text was presented at the conference session I held in Cairo on Drama, Performance and Digital Multimedia, on the 5th of September 2007, followed the next day by a workshop on the same topics. The other conference material is here]

1.
This round table has been designed to provide a broad approach to the use of digital technology in theatre and performing arts. The issue about the relation between theatre, performance and new media, has been arisen since the Eighties, when contemporary dance started to introduce robots and electronics on stage, or when fringe theatre investigates the relation between live event and TV camera and monitors.

Since then, much has been written about it, so that it is now possible to follow different approaches to the question.
Here the panellists will talk about different topics: new dance and new media, the theatre and virtual reality, the simulation of space and time in virtual representation, the interactive stage and costumes. All of these are topics that deserve a full length discussion, and certainly this is just a start, and I invite whoever is interested to come and join us at tomorrow morning workshop, at 11 am, here at Supreme Council for Culture, for further details.

2.
Now, let me spent few minutes to discuss how theoretical links can be drawn between drama and artificial intelligence, so that more effective creative combinations of the two can be developed. The points I will try to make are:

1) Digital performance isn’t about technology but about language – it is aesthetic rather than prosthetic.
2) Theatre and Digital Multimedia are both live events, happening in the ‘here and now’.
3) Drama and Artificial Intelligence share much in common because the basic element of both disciplines is “actions” over, for example, ‘thoughts’, ‘words’, ‘descriptions’.

Often digital media has been seen from a too specific technological point of view. We should all agree that the question about digital theatre isn’t about increasing the quantity of gimmickry on stage. And we should because it is well known that theatre, although too often associated with poetry and literature, has always confronted new technology. As mere technology, digital media don’t represent any significant innovation in theatre.

This leads us to consider the clash -or the relation – between digital media and theatre from a more abstract point of view. Critical theory on the subject – particularly Brenda Laurel and Janet Murray – has clearly stated that in both cases (theatre and digital multimedia), the core is an event: a live event. Digital multimedia is closer to the theatre based notion of hic et nunc, – the here and now – than to the way we show images on screens.

Whereas some literature on digital media, for example Lev Manovich, grew out of the field of film studies and therefore tended to link digital media with cinema, later studies linked digital media more closely to computer studies and science. I would further suggest that the primary element in both computer science and drama is the notion of procedure and algorithm. Both are ‘procedural’ disciplines, based on a kind of algorithm.

The definition of Algorithm “is a finite list of well-defined instructions for accomplishing some task that, given an initial state, will terminate in a defined end-state”.

In other words, it is a set of input-output behaviours. When we run software, any software, on a computer, we are running a procedure that will make an event happen, a specific event, at that time in that space. Could we see drama as something similar? That the software is run by an operating system on a computer as the playscript is run by a director and actors on stage?

In semiotics it is said that digital representation increases the separation that already exists in video or filmic representation between the real thing and the represented one; up to the point where the digital image is quasi totally auto-referential. The mathematical model, situated within the representation process, between the signs and their references, is a diaphragm that moves the two terms away from each other. The images generated by the digital procedure will be inevitably auto-referential. Beyond a mere simulation of the real world, these signs refer to the binary code and to the software that interprets it. What we observe on the computer screen may not have any relation with the real world (if not a simulation, where – we know – the interpretation is more important than the signification). When we see a world in CG 3D, no referential object exists; only the idea of the creator (and again, i.e. his interpretation of the real thing). We cannot even suppose a distant relation between what we see and what really exists.
For example, watching a tree on the TV screen, there is a representation of the distance. The tree doesn’t share our time or/and it doesn’t share our space.

With the computer, the image of the tree is an event, because it takes place in the very moment in which we watch it. That specific string of data will be read on the fly – instantaneously – by an apposite plug in, thereby generating what we see, far from any referentiality. That tree “is” in that very moment and in that very place, as we are. The computer screen doesn’t show object but generates our own sight, the act of vision, because it happen only within the polarity of object and subject. It is, in its very nature, an interpretative act: the thing takes apparent form only in front of us. Such auto-referentiality is common to theatre too. The happening hic et nunc of the theatre performance prevents – or minimises – the representation of distance we have mentioned above. The performance, although refers to ancient stories, faraway worlds, happens always in the very moment we are watching it. The actors and the stage generate the event hic et nunc. Hamlet “is” every time we see him on stage. However, when you watch Laurence Olivier playing Hamlet in the film, you are watching a narrative, in the past, from a distance. Unlike cinema, «Drama is primary. It isn’t the representation (secondary) of something (primary); it represents itself, it is itself». This is the sequence of presents, that Peter Szondi identifies as the root of the drama.

Within this polarity of media (cinema, TV, theatre, digital), divided according the representation of the distance from the pre-existent object (then and elsewhere of the reproduced image) and the complete absence of that distance (here and now) – the virtual reality, the computer graphic and the other digital representations belongs to the latter side of the polarity, together with – as we said – the theatre, and – in a wider sense – the performance.

3.
Hence, there are – even if on a preliminary investigation – some interesting contacts, from a theoretical perspective, between the live event which is theatre and digital multimedia. So far the relation between digital multimedia and theatre doesn’t appear as a clash of opposites but rather a delicate attraction between equals

Both are live events, but this doesn’t mean that the actual appearance of the event has to be the focus of our interest. Too often the performance draws too much attention to the event itself, and you may end up talking about technology, rather than aesthetic. If on one hand it is worth illustrating the phenomenology of how the theatre, the stage, has been invaded by digital technology; on the other hand this could lead to a dead end road, to a long list of names and applications. Soon or later we discover that – as I said – the theatre event, the spectacle, IS, and has always been, all about technology, since its birth.

Aristotle wrote that the sixth element of the tragedy, «Spectacle, [opsis – the look, the appearance] while highly effective, is yet quite foreign to the art and has nothing to do with poetry. Indeed the effect of tragedy does not depend on its performance by actors, and, moreover, for achieving the spectacular effects the art of the costumier is more authoritative than that of the poet».

Although Aristotle is no longer the only authority, as he was in the Renaissance, and the contemporary aesthetic has moved a long way forward to investigate mass culture, collective creation, disputing the importance of ‘the author’, etc., let’s agree for our purposes here that the author is important. That it is an author (the ‘programmer’) who designs the theatre performance. So, behind theatre and behind digital performance there is a specific language, a code that the author uses to write the procedure that will be run in a certain time and space.
Drama, even from an Aristotelian point of view, is a set of rules to be run on stage. Here emerges a more profound connection between drama and algorithm. Drama is, by nature, procedural.

4.
In the last ten years, as the multimedia boom has started to settle down, a few scholars have started to focus on what has been called interactive drama. The aim was to formalise the rules of a dramatic event so that it could be encoded in an algorithm (to be performed interactively by a computer). Although it may sound like the scientists are trying to teach the computer how to write a play, that is not at all how we perceive it. We see it as a question of language.

As we said, drama isn’t only something that happens in a certain moment. It is also, a way to shape actions in a specific way. We may say that drama is the design of an event that has specific dramatic qualities. Here I don’t want to sketch out the notion of drama, I want only to stress that the language of drama has some specific features. Naturally, by ‘language’ I am not referring to the words written in the lines of the text, but to the language of actions.

Drama theory has largely discussed whether drama is about plot or characters. The argument is centuries-long and has changed according to the contemporary socio-psychological model, cultural/religious beliefs, and aesthetic theories. Briefly, scholars and artists, in the last two centuries, have basically agreed that the most effective outcome of drama is the fictional character. The character has been – and still is – considered the medium the author uses to bind us to his story, to his message, to his soul. Character has become the interface between the audience and the artist.

Approaching the question from a more plot- or action- based perspective, in 1895, George Polti defined Les trente-six situations dramatiques as a set of situations, or a way to achieve a certain goal . Here, the most important thing, rather then the accuracy of the number of definitions, is the idea of a finite pool of elements: Supplication (Implorer), Deliverance (Le Sauveur), Crime pursued by Vengeance (La Vengeance poursuivant le crime), Vengeance taken for Kindred upon Kindred (Venger proche sur proche), etc. From an “actional” point of view those situation can be described as an interaction between agents endowed with goals and emotional state. Indeed the Polti approach was influenced by the success of the pièce bien faite (as defined by Sarcey) and by the relevance of natural science and positivism in those years. Nevertheless, this approach defines a proto-structuralism through which it is possible to discuss the true language of drama. Along this road to the semiotics of drama, we encounter the effort made, in recent years, by artificial intelligence theorists to formalize human cognitive and physical behaviours.

It is well known that the first famous attempt to simulate a human being using a computer procedure was Eliza, in 1966. Joseph Weizebaum wrote a little software program to simulate a dialogical interaction between a human and a computer. The software was conceptually based on few simple assumptions about natural language and on a specific therapeutic dialogue (in fact, Eliza was mimicking a therapist). This was the birth of a generation of chatterbots (chatting robots), who are still evolving – especially in the last three years – for example as virtual assistants on the web.

Although the aim of Eliza wasn’t drama at all, it was clearly stated by Weizebaum that the interaction had to be based upon the dramatic paradigm of character. Eliza had to be designed as a specific fictional character in order to be believable within that specific interaction.

In more recent years artificial intelligence research has started to look beyond the level of natural speech, focussing on behaviours and, therefore, on actions. Agent theory analyses speech as just one part of the higher level of ‘action’ Via a rational assumption of human behaviour, the character’s actions are described as a ‘goals and plans system’: from a cognitive point of view, the action is the will of reaching a goal, the design of a plan to get this goal, in a specific environment and situation. The character is designed to interact with the world: therefore he is provided with a formal model of the world where he lives. In other words, within the artificial intelligence framework, the character has a local knowledge about where he is and what kind of rules there are; a goal to achieve; and he adopts plans accordingly.

More technically this is called the belief, desire, intention model (BDI). Naturally a large goal requires a large and complicated plan. This large plan would be made of smaller plans that fulfil smaller goals… and so on, following a fractal structure.
The BDI model conceptualizes the character as a highly interactive medium, as an entity always on the move to adapt, to relate with someone or something. Character is more and more becoming the paradigm for human computer interaction.

5.
A deeper analogy between drama and artificial intelligence is emerging here. It means that in order to achieve a better human-computer interaction, computer science has to assume the rules that drama has developed to design characters that seem alive.

If artificial character is about the representation of a ‘live’ being, then it can be said that the first artificial character has been created on stage – in the theatre. Hamlet is alive and is the outcome of a process designed by Shakespeare. Synthetic artificial characters have to compare themselves to this cultural background. On this basis, the relationship between theatre and digital multimedia appears to be more profound. Here we can abandon phenomenology and start to discuss the ontological level.

Critical theory has long discussed issues and problems around the essence of drama. For the last two centuries, manly thanks to Anglo-Saxon pragmatism and the French pièce bien faite, it has been clearly established that drama it is not – at least, not only – written poetry. The defining feature of drama is action. In other words, the dramatic author faces the challenge of organising actions to be performed on stage. Drama is the language of actions. Representing actions via a written text can be a difficult job. Over the decades, the more technologically developed the stage became, the more complex the job became for the director. Stanislavsky basically re-wrote Chekhov plays adding hundreds of descriptions and directions; Piscator adopted prompt-scripts divided into columns to match the spoken words with the other elements of the action; even as purist an author as Pirandello made complex use of stage directions and soon decided to run his own theatre company to better control his plays.

The individual line of text in the play (‘to be or not to be’) may not be the best means of representing action…While dance has developed a number of notation systems (for example the Labanotation), and semioticians have tried a few complex systems to annotate drama, there still isn’t a proper formal system to represent – as data – the complex inner and outer contents that we can usually see – on stage – in a single written line of a play’s text. Agent theory may be able to contribute here. Agent theories have endeavoured to formalise a proper language for representing actions, providing scholars and practitioners with a fairly reliable – although complex – system. Because it is a new way of describing actions, it can also be a new way to describe or produce drama.

As we said, drama uses the character as a medium and shapes his actions according to a set of rules. Drama is a live action performed at the present time by agents. Usually the performance follows a script more or less precisely designed by the author.
In the last decade, computer science and the videogame industry have been trying to encode the rules of drama to be able to design dramatic events that can be performed interactively.

One of the best known attempts has been Facade, the interactive drama designed by Michel Mateas and Andrew Stern. There the user can freely interact with two characters (husband and wife, Trip and Grace), within a given situation, via text and gestures. The aim is to simulate a dramatic event as in an actor improvisation. The chief achievement of the experiment is to have made clear that, in order create believable characters, the interaction must follows the rules of drama. In fact behind ‘Facade’ there is so-called “drama manager” that constantly shapes, redirects, and controls the two characters. The “drama manager” performs as a live author, or as a dramaturg on stage, who has a formal knowledge of how the drama has to be handled.

Mateas claims a sort of neo-Aristotelian theory of drama, where the user takes part in the six elements of tragedy together with the character. Although facade has been the first and most effective experiment up until now, it has to be said that the theory behind it is still developing.
The software has been designed to match that specific dramatic situation (husband and wife, who host a friend – the user – for a drink in their flat and have an argument). So there is a lack of general assumptions about drama. The character doesn’t have the ability to develop enough to maintain the audience’s emotional engagement for longer that 10 or 15 minutes. Moreover, the audience coincides with the user and vice versa.

At the head of Aristotle’s six formal causes there is the author. Reconsidering the six elements within an interactive environment necessitates – I believe – adding a new element called ‘procedure to the line, directly above ‘plot’. Hence characters are the set of attributes (of different kinds) shared both by the virtual and human agents (the user). The procedural quality of the drama comes above the character’s interactivity and the latter is an outcome of the former. ‘Procedure’ is a new element available to the author to shape the drama, in addition to character and the other elements, and it can be perceived by the audience as one of the materials of which the drama is made.

In the last few years we – at the CIRMA research group in Turin – have developed a theoretical system called Drammar that accounts for the character’s emotional state (and its changes), and for the building of a plot using drama units. In our system the character is designed upon the BDI paradigm to which we added an emotion paradigm (using a pre existent formal model of emotions). The character is a set of beliefs and emotions. He also has drama goals and plans to reach them. The behaviour is shaped according to the drama unit direction, i.e. a higher goal that aims to change the emotional state of the character. Drammar, actually, doesn’t aim to any specific application. It rather wants to lay the theoretical foundation for a wider paradigm for interactive drama.

Nevertheless, we are using this background in more practical applications, such as a virtual character on a mobile device who guides visitors around historical sites. I will discuss these applications deeper in tomorrow’s session if anyone is interested.

Drama and theatre are bound together so that any change in the former will affect the latter. The new paradigm of interactive drama challenges the performance and its components: actor and space. The theoretical framework I have tried to draw, allows us to figure out a new kind of performing space, which is able to respond automatically, intelligently to the performer’s actions. Stage and actors are no longer in opposition, but in symbiosis. Here the dream of the Avant-gardes (from Symbolism to Futurism) became a starting point, a lens through which the scholars and the artists may see new experiments. On the other hand, the body of the actor itself can be augmented thanks to technology. This augmentation coincides with the “augment” of the stage. The stage has to review its own limits, both practical and conceptual. The space of drama can be sensible and intelligent, but also dislocated, as has already happened in pervasive gaming.

6.

On the higher level of narration, there is another point of view to investigate. To focus on plot and direction, means to overcome the fashion of the hypertext as something totally open, undetermined, and to move toward a particular form of procedural narrative. From this perspective, the notion of digital theatre sketched above, rather than stressing the kaleidoscopic nature of the pixels, highlights a so called procedural attitude. So, a ‘procedural’ drama produces a set of situations that, even if they can be customized by the user, are driven by a given structure that tends to shape the user’s emotional engagement. Moreover, behind this structure there is an author. After the last twenty years, during which digital arts, following the post-modern culture, have dealt largely with the loss of meaning, where interaction has mainly been interpreted as the fortuitous and peculiar relation between the user and the artwork, the time could now be right for investigating the field of the meaning, the possibilities of a well structured narration. To the random function raised at an aesthetic level, it is possible to oppose the strong notion of plot as a production of meaning.

Theater and Virtual Reality

Josephine Anstey, University at Buffalo

1 Introduction

I started working with VR in 1995, and have always considered it a theatrical medium. I studied at the Electronic Visualization Laboratory, at the University of Illinois at Chicago, which developed the CAVE VR system. In the earliest papers describing this system, it was referred to as a Virtual Theater [4]. Virtual Reality is now a term used to describe many virtual spaces, such as on-line worlds and multi-user games. But today I am using the term to describe a more elaborate system that is designed to support the user’s senses more fully. The CAVE and Head Mounted VR systems typically support stereo vision and are designed to completely immerse the user in 3D graphics. Tracking technology is necessary to allow the system to draw the graphics from the user’s POV, and also to allow the user to impact the world more directly. The user can reach out and virtually touch, grab, pull, or hit objects in the world.

I was initially interested in Virtual Reality as a site for dramatic first-person experience. I envisioned an intimate theater for one person, who would act as a protagonist in her own drama. Later I became involved with creating and staging networked VR events, and I became interested in the potential for creating performances in shared virtual environments. More recently I have also become interested in mixing the virtual and the real, in mixed reality and mixed media environments, and in playing with the relationship between live and virtual performers and performances. Today I will talk about my work with networked VR, mixed reality and mixed media.

The whole text [PDF, 64 KB]

Dance et nouveau medias

Emanuele Quinz, Anomos, France

Introduction

Dans sa célèbre analyse de l’impact des technologies de reproduction sur le système des arts, Walter Benjamin constate la distance, ou mieux l’opposition entre les nouvelles oeuvres techniquement reproductibles (cinéma, photographie, etc) et les arts de la scène, comme le théâtre et la danse¹. Car, si ces derniers proposent une expérience de l’immédiateté, où l’interprète et le spectateur sont présents corporellement l’un à l’autre dans un même lieu et au même moment (hic et nunc), pour le cinéma – par exemple – cette présence passe à travers la médiation d’un appareillage technique.
Cela implique, selon Benjamin deux conséquences: d’un côté, la prestation de l’acteur n’est plus présentée entièrement, mais ses limites spatio-temporelles sont données par le montage, de l’autre l’acteur perd la possibilité d’adapter son interprétation au public.

Irréversibilité de l’objet technique : dans ce cas, la perte de l’aura ne se caractérise pas seulement comme une transformation de l’interprète qui devient un élément parmi les autres de la machine de représentation, mais surtout comme clivage de la co-présence entre acteur et spectateur, comme explosion du hic et nunc. Le rapport au dispositif technologique (la caméra, l’écran) remplace le rapport direct entre acteur et spectateur. A tel point que la distinction entre eux n’est plus substantielle mais fonctionnelle. Comme conclut Benjamin, la médiation technique s’oppose et neutralise l’immédiateté du corps.

L’apparition d’une nouvelle technique ou d’une nouvelle technologie (en tant que “pensée de la technique”) impulse dans le système des arts des secousses et des mouvements de tassement. La mutation du paysage n’advient pas, comme trop souvent les critiques et les apologètes de la dite “art numérique” proclament, par une suite de remplacements et de catastrophes, mais par un processus de stratification et de complexifications.

The whole text [PDF, 56 KB]

Interactive stage and dynamic costume design for the opera. “Marlowe: The Jew of Malta”

Christina Finger, büro+staubach, Germany

In 1996 André Werner was commissioned by the Munich Biennale Festival, a festival for modern music theatre, to compose a contemporary opera. The opera premiered in 2002. It’s based on the play “The Famous tragedy of the rich Jew of Malta”, written by Christopher Marlowe in 1596. The Jew of Malta unfolds as a story of expropriation, revenge and death of the Jewish merchant Barabas set in a power struggle between representatives of three world religions.

Very early a team was formed taking care of the direction, the stage design, the media concept and the programming. The task was to create an interactive stage and dynamic costume design in close co-operation with André Werner during the process of composing. It was clear from the outset, that the aim was not to create a high-tech spectacular on stage but to extend the options of the traditional stage and costume design.

The composer rewrote the drama for the libretto. He broke the chronological storyline and used only fragments of the original text. Although in Marlowe’s play Machiavel only appears in the prologue, André Werner made him the central figure of his opera.

Machiavel considers himself a creator of worlds. In the opera he sets up experiments like improvised plays-within-a play. He directs his experiments, which mainly deal with the acqui- sition of power. Machiavel controls the scenery. He interferes with or stops his experiments, to initiate a new setting, like starting a new program. His power is also expressed by distributing the 15 roles among the other four actors on stage – though technically this process is pre- defined. In the course of the opera he ultimately loses control and fails in his own game.

The whole text [PDF, 572 KB]

Representation in Space and Time: Control Score and Responsive Interface to a Virtual Performance

Vincenzo Lombardo, CIRMA, Università di Torino and Virtual Reality & Multi Media Park

Abstract

This lecture is about the application of the simulative methodology brought over by the digital culture to the design, portability and fruition of a performance. The simulative methodology is a computer-based approach that organizes and delivers the aural and visual elements of a performance according to the temporal sequence defined by a control score, to provide a realistic experience through a responsive interface to a virtual reality environment (Brooks 1999). The result is a virtual performance. The simulative methodology provides a novel approach to the representation and control of a performance, bringing together the descriptive power of storyboard drawings (that here become more or less realistic 3D objects and scenes), the algorithmic and interaction capabilities of the computing machine, and the attractive mediation of a virtual reality environment (though adequate interface design is necessary to realize a satisfactory simulation).

With more and more sophistication in computer graphics representations (with high mimetic quality) and the simulation power provided by the programming environments, the virtual performances become a powerful tool to design and pre-visualize a complex event. Both static (representation) and dynamic (algorithmic) aspects must obey the lexicon and the rules of a formal language that become the symbols manipulated by the control system. Contemporary shows are complex events with contributions from several sources (each requiring specific competencies) and the existence of a common paradigm for representation and control is a viable possibility for providing a connection over operational attitudes that hardly communicate (consider, e.g., video and programming practices) and that conversely require neat and unambiguous responses.

The whole text [PDF, 252 KB]

Documents

Material produced during the festival

The Experimental, n. 2 [PDF, 624 KB], n. 4 [PDF, 420 KB], n. 6 [PDF, 800 KB].

Note

1. W.BENJAMIN, Das Kunstwerk im Zeitalter seiner technischen Reproduzierbarkeit, in Ausgwählte Schriften 1, Frankfurt a.M., Suhrkamp Verlag, 1977, p.150-162 [↩]