Multisensory interactive installation

External collaboration
– Software development and design for the first demo presentation in flash: Manuel Viñas Limonchi, University of Granada.
– Music for the first demo presentation: Umberto D’Auria, Mozart Studio Salerno, Daniela Voto.

Abstract
This paper presents the “Multisensory Interactive Installation,” (MII) a musical interactive installation based on Kandinsky’s paintings, through which a user can become a painter and a composer simultaneously. The painting’s visual elements are synesthetically linked to musical elements: each color is linked to a specific instrument and chord while each shape is linked to a specific rhythm. Through “Multisensory Interactive Installation,”users can explore and manipulate visual elements, generating different musical outputs, constructing and deconstructing the art piece and/or creating a new work, and musical composition. The selection and organization of given visual-acoustic elements, determines the different musical atmosphere created by each user. The methodology implemented in the system/installation, explores the correspondence between color and sound, visual and acustic elements, and is based on the neuroscientific research of synesthesia, an involuntary physical experience of a cross-modal association.

Keywords: musical interactive installation, virtual reality, Kandinsky’s paintings, synesthesia, multisensory processes.

1. Goals
In “Multisensory Interactive Installation,” the authors aim to explore the interaction between music and visual perception, using advanced technology, to create an interactive synesthetic experience in which the user can feel him/her self, as a painter and a composer, allowing the user to create his/her musical composition through the manipulation of visual elements, and the power to implement a cross modal interaction between different sensory channels, based in the latest research of multisensory processes.

This system can be applied and implemented in the fields of art, music, therapy and educational projects through the complementary use of various sensory channels.

2. About synesthesia
Synesthesia (Greek, syn = together + aisthesis = perception) is the involuntary physical experience of a cross-modal association. It occurs when the stimulation of one sensory modality evokes a perception in one or more different senses. The additional perception is real for the synestheta, perceived not only by the mind’s eye, but also by the body.

Two or more senses are automatically and involuntarily coupled so that a voice is not only heard, but also felt; seen or tasted Synesthesia is a physical experience of the brain, not the product of imagination or learning. It differs from metaphor or artistic expressions in that it deliberately links sensations. It has physiological features and is an intriguing violation of the conventional perceptions of the world. A specific form of synesthesia occurs more often than others: sound- sight (colour hearing) is common whereas unions involving taste and smell are rare.

The most common subtype is color-graphemic, in which letters or numbers and occasionally shapes, produce color patterns. It is also possible for more than two sensory modalities to be perceived simultaneously. Colour-Sound is one of the most common subtypes of synesthesia. It is possible to find evidence of this type of experience in the biographies of many artists, musicians and inventors.

3. Background and research
Synesthesia is a natural phenomenon in people. Recent neurological research has shown that it is crucial to understand how the human brain works in order to reveal how the senses are interrelated in the brain. Cytowic, the best-known neurologist for rediscovering synesthesia in 1980, affirms, in fact, synesthesia is a normal brain process that is prematurely displayed in the consciousness of a minority of individuals [¹]

One of the main interests of both neuroscientists and philosophers of science concerns how we perceive the external world. Much of the history of perceptual research had focused on the functional properties of individual senses, setting aside that our perception of the world is fundamentally a multisensory phenomenon. The recognition that a multisensory perception is necessary to enhance our understanding of sensory perception has led, as a result, to the emergence of a distinct field of scientific endeavour designated as multisensory integration or multisensory processing [²]

Our senses are designed to function in harmony. Our brain is organized to elaborate information, coming from the different sensory channels, cooperatively, in order to have a complete vision of reality.

Modern neuroimaging techniques, such as MEG, fMRI, PET, has enabled researchers to examine the neural consequences of multisensory integration in the human brain.“Multisensory Interactive Installation” focuses on the cross modal interaction between visual and auditory stimulus.

Subjectivity is a peculiar trait of synesthetic perception. As stated in the history of synesthesia, individuals perceive the synesthetic correspondence as sound-colour, color-graphem, etc., in different ways. In Newton’s circle of colors, the musical note “do” was red; in the Clavecin oculaire by Castel, the same sound was blue; in the paint translations by Luigi Veronesi it was violet. It is more likely to find general agreement among people in relation to the perception of synesthetic correspondence. For example, for most of us, high sounds suggest images with clear colors, while low sounds suggest dark and round images, etc [³].

A painting in black and blue, for example, can easily give an impression of a dramatic atmosphere associated with dramatic music and low sounds of a contrabass, not the acute sounds of a flute.

The mechanism of learning is increased if we simultaneously stimulate different areas of the brain, visually, musically, linguistically and kinaesthetically. Using a cross-modal sensory stimulation, we are able to activate a multisensory perception. The cross-modal perception of reality can also change our sensory perception. Recent research has shown that a sound can radically change the phenomenological quality of the percept of a visual stimulus: when a single brief visual flash is accompanied by multiple auditory beeps, the single flash is perceived as multiple flashes. It is clear that the information processed by the different sensory modalities are integrated into a coherent multisensory percept by the brain [⁴].

During the nineteenth and twentieth centuries, many artists were working to achieve an integration between the arts. Wagner introduced the concept of “Gesamtkunstwerk”, or Total Art-work: he theorized an idealized union of all arts, from music, dance, and poetry, to visual arts and stagecraft.

The theories of Wagner had a strong influence on various artists of the XX century, such as Kandinsky and Scriabin. Kandinsky, an abstract art pioneer, affirms that the sensations coming from different sensory channels can resonate simultaneously, as the strings of a violin resonate when only one is touched. Scriabjin created the first musical poem, “Prometeo,” in which he mixes pictorial and musical elements with a parallel “andamento”.

Previous research has demonstrated that the cross modal interaction between two or more different sensory modalities increases the activity of the brain.

There are people who see keys or notes in colours and composers who favour keys for their sonic “colour” characteristics. The difficulty, from a scientific perspective, comes in trying to compare these sense perceptions between people. There has been no established pattern to the colors or keys based on their respective colors; it varies by individual. The study thus far has been based on arbitrary correspondence stemming from extensive investigative research of the phenomenon.

4. Conceptual methodology implemented in “MII”

MII implements a methodology based on the voluntary and rational activation of different sensory modalities to permit data acquisition. The authors hope to implement a new form of symbolic music representation.

The user is able to follow the melody of a music piece through the rotation of geometric shapes linked to a specific color in vr space.The user can live a synesthetic experience in which he/she can perceive the correspondence between sound and color frequencies: a painting with dark colors produces a more gloomy musical atmosphere, otherwise a painting with light colors produces music that is more relaxing and cheerful.

In the Kandinsky’s paintings of MII, each color is linked to a specific musical track; in total there are 8 musical tracks. The superposition of these tracks can create an orchestral composition. The user can also choose how to create an audio-visual composition: he/ she can use the synchronization or the succession of audio-visual elements and/or the repetition of the same element within the same composition. The choice of colors and order in which they are arranged permits the creation of unique musical compositions, allowing the user to be free to express his/her creativity.

In another one of Kandinsky’s paintings, “Picture’s Gnomus,” the visual elements, of point and line are connected to specific musical effects, in this case, using rhythmic instruments. The color and size of the point, and the color and shape of the line determine the type of rhythmic effect. The user can choose from several musical effects and to arrange in superposition or in succession creating different audio-visual compositions.

The color of the surface determines a harmony linked to a specific tonality. For example, a blue surface is connected with a harmony in minor and low sounds. A blue is mysterious and sad, while a green surface produces a harmony more relaxing and quiet.

The goals are to experiment with the creation of different musical compositions with given visual elements and the memorization of a specific tonality linked to a specific color.

Kandinsky Picture’s Gnomus 01

5. Interacting with “MII”

Multisensory interactive installation-longitudinal view 02

The user will find several paintings by Kandinsky in the installation. The shape and color of each painting is associated with a specific musical theme. The synesthec experience is determined by the correspondence between color and harmony and shape and rhythm.

For example, a painting with dark colors will generate a sad musical atmosphere. A painting with a combination of dark and light colors will generate a musical composition more peaceful and mysterious. Geometrical forms such as circles, triangles and squares are integrated in superposition or in temporal succession within the painting structure.

Kandinsky paintings in MII 03

The user can select the geometrical elements from the original painting and shift them to a white canvas. When this happens, they morph into three-dimensional forms generating music. The rotation of the elements in 3D will give the sense that the music flows in time.

While transferring music into images and visualizing music flow has been an historical dilemma, today this has been overcome with the emergence of digital technology and virtual reality.

The combination of colored-shapes on a white painting associated with musical themes generates new audio-visual compositions with the given elements.

“Multisensory Interactive Installation” focusses on the cross modal interaction between visual and auditory stimulus.

Kandinsky painting in MII 04

5.1. Colors and music in “Multisensory Interactive Installation”
In “Multisensory Interactive Installation” the correspondence between visual and music elements is based on the expressionist theories of colors, and on the teosofic theories of relationships between color and feelings. The correspondence also focuses on chromotherapy and music therapy research results. The use of simulateneous correspondence between sound and color is considered highly beneficial for people with mental illness.

Color therapy has a solid scientific foundation. A high number of researchers have demonstrated the biophysical changes in the human body caused by natural and artificial light. Each colour is used in the chromo therapy science for a specific treatment.

TABLE 1 Correspondence Colors-Sounds-Feeling

In “Multisensory Interactive Installation” each color is associated to a specific feeling and musical theme. The musical effects are created with electronic music and combined with other sounds in order to create a specific musical sense. The combination of audio-visual elements will create different musical compositions. The elements can be in superposition or in succession within a new painting.

Transformation in 3D of square and triangle 05-06

5.2. Shape and rhythm in “Multisensory Interactive Installation”

In this project, the emphasis was divided into two courts. On one court is the synethesia color, sound and rhythm.

On the other court are points and lines associated with the rhythms. This association is based on linking shape to the length of each sound.

Circle = static and short musical element
Triangle = long sound

In the 3D environment, the shapes rotations create visual effects generating music. Associations of different colors determine the musical atmosphere, while the choice of shape determines the rhythm of the musical element.

The final result is the creation of original paintings and musical compositions with given elements.

5.3. The first prototype
As a first phase development, a multisensory interactive environment was implemented, using Macromedia and Flash software.

The environment is a virtual museum where specific pictures by Kandinsky can be found.

Interacting with the paintings 07

The visitor can navigate in the virtual museum. Once he selects a painting, he can apply geometric transformations (translate, rotate, scale, deform…) in shapes to change the initial graphic composition. Every change in a particular shape, linked to a specific colour, emits a predetermined sound effect. He can move the shapes to another white painting. The organization of the shapes determines a new musical composition.

Interactive Phase 08

Interactive Phase 09

6. Music artistic motivations
In MII the authors want to explore new ways of interacting with the music composition using images.

History is full of artists and theorist who have created systems that bring colour and music together.

The challenge is to create images that elicit the same kinds of associations and emotional connections that listening to music does. The authors aim to find ways to compose, produce, and play music using a visual aesthetic.

In the first prototype of MII, from artistic perspective, the user can play with images obtained from Kandisnky’s paintings. Following the theories of Nils Hansen, creativity can be developed with different combinations of known elements. In a second phase the possibility to create original visual elements will be implemented. From the perspective of music, in the first prototype of MII, 8 sound elements (musical lines) are created based on the superposition of elements following the logic of orchestration.

It is the first demonstration to analyze the correspondence between sound and colour. The implementation of the concept in MII is based on several experimental music didactic theories. The composer intends to develop a new methodology of musical composition linked to images. It starts by rhythmic ear training to achieve melodic ear training. The methodology is based on rhythms repeated [ostinato = repetition of a same rhythmic element in the time] connecting to shapes and chords connected to a specific colour.MII wants to create a creative environment that allows the visitors to create music on various levels, individually as well as within an interactive team, through interaction of remote users networked together using the latest technologies.

The system wants to integrate movement, speech and tactile perception following the cutting edge research in the technologic field.The verbal induction produces a gesture. The gesture produces a sound in an interactive VR environment; gesture can be perceived as a tactile experience by a remote user. The sound gained produces correspondent image. The result is a production of sharing music through multi sensory channels.

We believe that modern technological age requires and allows new ways to approach art, music and composition. In the near future Audio-Visual Interfaces that are simple and functional can make Art and Music accessible to vastly more people and not only to the experts.

7. Technical requirements
In this block we analyze the logical and physical requirements (software and hardware equipment) necessary to develop an initial interactive application (a beta application, running in personal computer) and those matters related to the development of the MII final project based on a VR interactive application (a complete digital project running in an immersive platform).

7.1. Software
Next, we specify some computer programs required to develop MII.

7.1.1. 3D Entities: Modeling and Animation
Specific programs prepared to creation of volumes -vectorial structures- constituted by polygonal surfaces and meshes.

• Discreet 3DStudio MAX.
• Alias/Wavefront Maya.

7.1.2. Paint Images

Specific programs prepared to creation of -bitmaps-images, ideals to embed to those surfaces that compose the constituent base of the 3D environment.

• Adobe Photoshop.
• …

7.1.3. Virtual Worlds Modeling

Specific programs prepared to creation and modification of graphic entities disposed to be showed using VRML players.

7.1.4. Virtual Worlds Viewer (Players)

Specific programs prepared to visual representation of graphic entities created in VRML format.

• Cosmo Player.
• Cortona VRML Client.
• Sony Community Place.

7.1.5. Programming Languages

Indispensable programs to creation and interpretation of objects and actions in any VR and VRML application.

• C++.
• Java.
• VRML.
• HTML

7.2. Hardware

7.2.1. Conventional Digital Devices: Computer + Instruments to execute interactive experiences

The PC or workstation screen will be the graphic means of diffusion that -aided by visualization devices, type HMD and others disposed like sensors coupled to a data glove or a joystick/mouse- displays all the interaction specifically designed for this concret virtual environment.

Likewise, the sequence related to the creation of formal and sound entities that integrate our VR and/or VRML creation requires computer devices in accordance with the methods of design employees in CAD projects; that is to say: we need computers that include a potent processor and enough readiness as for devices in charge to develop graphic process tasks.

Beyond these traditional computational devices, next we expose the more efficient machine in which we want develop and show our project MII: the CAVE.

7.2.2. CAVE

The CAVE is a virtual immersion platform, like a small room (10x10x10 ft.). In their walls and roof are projected 3D images. Introduced in this reduced space the user-visitor experiments the sensation of being in an encircling environment where he can collect optic and sound sensations extracted from the purest sensorial reality belonging to the well-known or unknown nature.

MII in the CAVE 10

Today – possibly – the more perfect multiuser platform created to perform a virtual immersion. Inside we can interact in a fictitious environment.

8. Bibliography

• BERTIROTTI A., L’uomo, il suono e la musica, Firenze University Press, Firenze., 2003.
• BRODLIE, K.W. et al. Scientific Visualization, techniques and applications. Springer-Verlag. 1992.
• EARNSHAW, A.; GIGANTE, M.; JONES, H. Virtual Reality Systems. Academic Press. San Diego, 1993.
• FREEMAN, H. (Ed.). Machine Vision for Three-Dimensional Scenes. Edit. Academic Press. San Diego, 1990.
• GRAU, O. Virtual Art. From Illusion to Immersion. The MIT Press. Cambridge, 2003.
• HEIM, M. The Metaphysics of Virtual Reality. New York and Oxford University Press. 1993.
• HELSEL, S.; ROTH, J. Virtual Reality: Theory, Practice and Promise. 1991.
• KANDINSKY, V. Punkt und Linie zu Fläche. München 1926. Edizione italiana: Punto, linea, superficie. Milano 1968.
• KANDINSKY, V. Scritti intorno alla musica, a cura di N. Pucci, Fiesole 1979.
• KRISHNAMURTHY, B. Data Visualization Techniques. John Wiley & Sons. 1999.
• LÉVY, P. Becoming Virtual: Reality in the Digital Age. 1998.
• RAMACHANDRAN, V.S. and HUBBARD, E.M. (2001). Psycholophysical investigations in to the neural basis of synesthesia. Proceedings of the Royal Society, 268, 979-983.
• SCHOREDER, W; MARTIN, K.; LORENSEN, B. The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics. Prentice Hall Computer Books. 1997.
• VERDI L., Kandinsky e Skrjabin, Akademos & Lim, 1996.
• VINCE, J. Virtual Reality Systems. Addison-Wesley, 1995.
• VIÑAS, M. Técnicas de Infografía. Variables creativas metodológicas en el desarrollo de la imagen digital. McGraw-Hill Interamericana de España. Madrid, 2000.
• ZEKI SEMIR, Inner Vision: An Exploration of Art and the Brain, Oxford University Press, Oxford-New York 1999.

Pictures by Manuel Viñas Limonchi

PDF [268 KB]

9. References

1. CYTOWIC R. E., Synesthesia, a union of the senses, MIT press, 2002. [↩]
2. G. A. CALVERT, C. SPENCE, B.E.STEIN, The Handbook of Multisensory Processes, MIT press, 2004. [↩]
3. MARKS, L. E. (1975). “On coloured hearing Synesthesia: Cross-modal translations of sensory dimensions”. Psychological Bulletin, 82, 303-331. [↩]
4. SHAMS, L., KAMITANI, Y., & SHIMOJO, S. (2002). “Visual illusion induced by sound”. Cognitive Brain Research, Vol. 14, pp. 147-152. [↩]