Sound Seeker

In 1932, the film theorist Rudolf Arnheim published a little book entitled Film as Art. It offers some intriguing challenges for contemporary new media designers and artists. Arnheim believes : “a combination of media that has no unity will appear intolerable.” (p.201). By media he was referring to sound and image, the new talkie films. But the problem extrapolates easily to other aspects: how to unify text and image? text and sound? text, image and sound?

In  response to this challenge (and in response to the innate need to see things as being right or belonging in the same space) Sound Seeker arose. It incorporates what I call incestuous interactivity or cybernetic synaesthesia. It is incestuous because the computer interface interacts with itself: sound  (from an mp3 or a microphone) influences image or text change; it is synaesthesia because sensory modalities mingle. This technique is very simple and widespread from screen-savers to VJing apps; basically sounds change visuals. It also occurs throughout nature: a sharp noise makes an animal cringe, the prey flee, glass shatters. It has become normative for viewers to experience interactivity that crosses sensory modalities (from sight to sound, sound to touch, touch to sound) a whirlwind of data occurs.

Unity (or the perception of unity) arrives when events synchronize and seem responsiveness to each other.

The computer doesn’t think of data structures as being associated with specific senses. Data-structures are normally piped thru sense-specific devices: the sound data sent thru sound card, video data sent thru gpu. But in general this information is valence-free when it is stored. It easily leeches back and forth. Societies of data emerge. Relations between data evolve and emulate organic structures. This feature allows sound-image-text changes to form a micro- community, a colony of inter-relations that contributes to the perception of unity.

Another challenge is what Arnheim would term ‘profundity‘: choices about relationships that are meaningful. Arnheim was discussing film; his examples relate to camera angles. Was there a significance and thought provoked by the angles or position of camera or style of editing. Was this necessitated by the story? Did it contribute to the advancement of plot?

In multimedia this translates into a concern over the relation between interactivity and content. Jason Lewis states the problem this way: “If you are moving something: why? The why is connected to meaning…There is an intrinsic space for beauty, but I also believe that one of the approaches to take to digital media is to think very seriously about motion and interactivity as tools to create the meaning…” (source). In short, is the interactivity meaningful? Does it contribute to the content?

Each of the Sound Seeker demos tries to consider the content’s relevance to the synchronization and layout. In most cases the changes are subtle but distinct modulations in rhythm. In other cases, the shuffling font speed and style is mapped to music that attempts to emotively contribute to the text’s content (For an example, see Next, where the content is divided into 3 sections: the first is agitated, the second calm, the third rapid yet resigned to its rhythm.) By modulating text in synchrony with sound, computation can contribute to the evolution of a sense of unity in the art object and perhaps go some way toward answering the challenges Arnheim raised so many decades ago.