StarCAVE

The StarCAVE is a 15 sided virtual reality environment installed at CalIT2 at UC San Diego. It has a pentagonal floor-plan, and spans about 12 feet across. All 15 screens are back projected with dual 2k projectors, whose signals are resolved in full 3d with the aid of glasses. A head tracking system is implemented that allows perspective shifting in relation to a viewer, resulting in a realistic 3d immersive environment.

I was involved in the planning and integration of the audio system for the StarCAVE. The initial installation was a 15.1 configuration, set up as 5 - 5.1 systems of Meyer Sound MM4 4-inch speakers. The top system is a middle speaker cluster that is located as a first reflection off the middle of the top screens. The second row is placed at the vertices of the pentagonal floor-print, and is a direct source to the listener. And the third group of five is housed in the floor of the cave, and points straight upwards. The subwoofer is housed in a resonant enclosure under the floor of the cave.

I have also developed several spatialization software systems for use in the StarCAVE. Research continues for the best sounding audio methods to complement the immersive nature of the cave.

Screen

http://www.noahwf.com/screen/

Screen, by Noah Wardrip-Fruin, is an immersive literary work, redefining an audience's interaction with text through gameplay. Screen was created in the "Cave," a room-sized virtual reality display. It begins as a reading and listening experience. Memory texts appear on the Cave's walls, surrounding the reader. Then words begin to come loose. The reader finds she can knock them back with her hand, and the experience becomes a kind of play as well-known game mechanics are given new form through bodily interaction with text. At the same time, the language of the text, together with the uncanny experience of touching words, creates an experience that doesn't settle easily into the usual ways of thinking about game-play or VR.

I have been involved in two specific implementations of the audio environment for Screen. The first was developed for Siggraph 2007, specifically for the newly constructed StarCAVE system at UCSD. The cave is a 15 faced, fully immersive VR environment. The entire video display is stereoscopically projected, thus giving 3D video with glasses. A head tracking system provides locative information. A 15.1 channel speaker system augments the video component, providing a full spherical sound stage. As Screen plays, the audience interaction triggers sound events that correspond to text ripping, breaking, and being hit, as well as ambient sound events. Using ambisonic processing techniques extremely accurate spatialization was achieved even in this resonant structure.

A second installation of Screen was constructed at the Beale Center at UCIrvine. This installation was only for a single stereoscopic screen, and a four channel spatialization. However, the previous software version allowed an easy scalability to this reduced setup with minimal overhead.

Crosstalk

CrossTalk, by Jay Cloidt, is a “duet for pianist and piano", as the live pianist's performance is analyzed by another virtual pianist, and the virtual pianist responds to melodies it recognizes and replies with piano melodies of its own. I wrote software to analyze the score in realtime, and trigger more melodies based on the performance.

Recombinant Media Labs

http://www.recombinantmedialabs.org

Recombinant Media Labs (RML) is a unique arts incubator and performance space, exploring the emerging format of spatial media synthesis.

The heart of Recombinant Media Labs is a 360 degree, 10 screen surround cinema room with the purpose of exploring spatial media synthesis; the art of projecting image and sound in 3D space. The room also features a 16-channel sound system, augmented by 8 bass cabinets, 2 sub-bass LFE cabinets, and 16 floor transducers. In addition, the facility houses a wealth of analog and digital creative tools to enable artists to create works for the emerging surround cinema format.

The RML facility shares space with Asphodel Studios, a full 72 track recording studio with a comprehensive assortment of analog and digital equipment. We also maintain a full synthesis room of modern and vintage instruments, including Serge and SMS modular systems.

RML hosts residencies by cutting edge artists working with new media, curated by Naut Humon. Artists that have performed or developed work at RML include: Ryoji Ikeda, Pansonic, Richard Devine, Morton Subotnick, Skoltz-Kolgen, Max Mathews, Joe Colley, Scott Arford, Sue Constable, Les Stuck, John Bischoff, Jochem Paap, Scott Pagano, Biosphere, Zbigniew Karkowski and Daniel Menche. With such a diverse array of talent entering our one-of-a-kind space, each event is a unique mix of cutting edge technology and experimental art, pushing the boundaries of electronic media.

I serve RML as a main audio/visual engineer for the surround performance space, an audio engineer in the synthesis and recording studios, and as a programmer researching and developing tools and interactive media software to support visiting artists. I am also involved in the design and maintenance of new systems for surround audio and video aimed at festivals and new performance venues around the world.

(photos by R. Yau)

TrioMetrik

http://www.beamfoundation.org

TrioMetrik, house ensemble of the BEAM Foundation, is an string ensemble exploring rich interactivity in electronic music. Using enhanced instruments coupled through an intelligent computer network, we create new music that is reactive, introspective, and fluid; resulting in surprising complexity and intimacy.

At the heart of TrioMetrik is a computer system dubbed MAPPS. It serves as the hub of the musician network and collects, interprets, and acts on the musical information played by the ensemble. They in turn react to it; in this way it functions much like the fourth member of TrioMetrik. At BEAM I am involved in the development and maintenance of this system, as well as the research of other new methods for interactivity in the performing arts.

This system analyzes the notes, gestures and intentions of the musicians and uses it to generate new music, process the sounds of the ensemble, and affect the score. A formalized personal display system provides notation, directions, and feedback that guide the musicians through the score representing the composition while reacting to their individual performances. This sophisticated level of integration of live instruments, captured audio, synthesis, video and sound processing enables a form of interactive music that is engaging, immediate, inviting, and never before possible.

FleshLightMovement

FleshLightMovement is a piece by Kinesthetech Sense, an organization founded with the intent to collaborate with visual artists, dancers, and musicians creating ferociously interactive experiences for audiences throughout the world.

In the piece shadows of the dancer, cast on a screen, are interpreted to directly influence the sounds in the composition. This linking of dancer and music allows the composition to be evolved over time by the choreography, creating a dynamic work. I developed the software to translate these shadows into compositional states, and the time delayed video projection.

"The metaphors surround light and its absence are powerful, encompassing truth, revelation, discovery, memory and consciousness. FleshLightMovement is an activated space that creates new metaphores from movement, media and sound. Events, both sonic and graphic, are repeated, shaped, and manipulated over time. Someone casts a shadow onto a screen as the image of the shadow is then processed and projected onto a second screen. Thus, there exist three planes of movement: human, shadow, and manipulated shadow, mirroring the three planes of existence in Buddhist philosophy; the planes of desire, form, and the incorporeal. The density of the shadow in different areas of the screen transforms an ambient sonic texture and participants can trigger bells by reaching up to the heavens, interjecting pointillistic commentary into the wash of sound."

Respondr

Respondr is a revolutionary interactive video system that creates high-impact branding and immersive multimedia experiences through audience participation and physical interaction. This cutting edge technology engages consumers through responsive interactivity, opening the door for creative branding and immersive environments.

I have worked on the modularization of the realtime interactive video effects in Respondr to optimize and allow for future expansion.

Radio Baton

The Radio-Baton, developed by Max Mathews, is a controller for live computer music performance. It tracks the motions, in three dimensional space, of the ends of two batons which are held in the hands of a performer.

I work on the software side of the Radio Baton, and have updated it for cross platform compatibility and a more flexible MIDI implementation to allow communication to a variety of synthesizers.

Immersive Media Research

Immersive Media Research is dedicated to providing sound designers, musicians, and composers with professional, and powerful software products that address the entire production chain associated with the development of multichannel audio content.

As a programmer at IMR I have been involved with the development of a number of tools allowing users to build complex surround mixes using spatial motion paths, and incorporating features never before seen in commercial mixing applications; such as algorithmic mixing (chaotic, non-linear, flocking, etc.) and real-time control via an assortment of USB and MIDI controllers.