Maelstrom
Environment as a collective sensory memory
‘Maelstrom’ presents a journey through the symbiotic relationship between humans and the environment, with a groundbreaking real-time audio-visual interactive interface. This piece reflects on the prevailing overflow of data and technology that shapes our information ecosystem and the sensory overload imposed on humanity. It delves into the impact of “immersive” information environments and generative technologies on our cognition: within today’s data ecosystem, how do immersive experiences and generative technologies influence our capacity to process information? How can we perceive patterns in otherwise seemingly chaotic and unpredictable streams of activity as a means of not only surviving but also embracing the inevitable interdependence of an unpredictable future?
Marshall McLuhan recognized parallels of today to Edgar Allan Poe‘s novel “A Descent into the Maelström” In this short story, a sailor becomes so absorbed in thought that he is caught in a maelstrom. Realizing he cannot row his boat out of the situation, he must observe the objects within the maelstrom and grab onto the recurring ones as a means of escape. The lesson: only through pattern recognition can we cope with information overload. This vortex embodies the challenges of our information age today, and the exhibit replicates the process of “detachment” through information overload, yet it possesses cyclical patterns in time. Through such sensory patterns, collective memories and habits become an inheritable feature of the environment, offering a way out of the maelstrom.
Commissioned by SUSAS (Shanghai Urban Space Art Season), this audio-visual installation, eveloped in Unreal Engine and utilizing Dolby Atmos spatial audio technology, uses 7.1 surround sound and generative projection, empowering the audience to create their own unique responsive spatial auditory experience by developing the original map and soundscape of Shanghai with generative model. It explores the concept of “information ecology” and perceptive memories. The installation plunges visitors into a sensory vortex, providing an innovative guided
experience. It helps us embrace the overwhelming forces of data today to bridge the gap between evolution and technology, and between technology and humanity.
This project is a part of Acoustic Garden Program, focusing on decision making and cognition support through affective computing and spatial music system.
2023
Built with Dolby Atmos, Unreal Engine/MetaSound
Artist/Director – Yufan Xie
Co-Writer – Mel Lewis
Sound – Ziqian Yin
Visual – Wei Wu
From a historical overview of visualization, sonification, music, and architecture, this project harnesses the latest interactive technologies to innovate human-centered narratives. These narratives are driven by spatial interaction and acoustics, enhancing accessibility and providing affective feedback for general audiences, including those who are visually impaired. A new culture emerges, interpreting messages universally through time within a spatial musical system. This system aids humans in navigating an era of sensory overload, offering a critical perspective on prevailing ocularcentrism and sensory bias. It provides a multisensory method to augment the decision-making process and navigate the current maelstrom of messages with a musical framework.
Pattern recognition support through multi-sequential support with Distance-related Audio Effect Modulation, users learn their impact on targets and navigate faster and more accurate through scene iterations.
The visual aspect of “Maelstrom” utilizes a generative AI model, creating branching sequences directed by users. It places the audience at the epicenter of an information landscape, where imagery is generated not merely through the image itself but through the process of temporal evolution. The installation employs DALL·E 3 generative models to evolve urban satellite images into real-time memory maps, reflecting geological shifts and generational evolution. Characteristics and memories inherited across generations facilitate the audience’s acclimation to new environments, based on past experiences. The resemblance between previous and subsequent scenes, as well as the sensory memory of these scene attributes, accumulate as experience, guiding the audience’s navigation within the space. Concurrently, the audience’s choices shape the evolutionary path of the ensuing scene, establishing a mutually influential relationship with the environment.
Pattern recognition support through multi-sequential support with Distance-related Audio Effect Modulation, users learn their impact on targets and navigate faster and more accurate through scene iterations.
Auditorily, “Maelstrom” is an interactive sound installation that prompts audiences to learn and react to environmental shifts through an evolving sonic narrative. In this narrative, audio cues movement, and movement redefines the audio, delivering a multisensory experience. With our spatial DAW (Digital Audio Workstation), centered on Distance-related Audio Effect Modulation, this work transcends traditional audio-visual experiences and becomes an interactive musical instrument where audiences can engage through a minimal interface. It redefines musical conventions with spatial and non-linear narratives, where music progression unfolds as a multi-branched journey, advancing beyond mere sonification and visualization. Guided by spatial audio, the installation empowers all participants, including the visually impaired, to partake as independent performers. The narrative is crafted as an accessible learning journey, inclusive and shared by all, not exclusively reserved for professional musicians. Technically, with the latest technologies from Unreal Engine, Metasounds, and a distributed speaker array, it elevates the responsive environment of Dolby Atmos spatial audio to a new level of interactivity.
MAELSTROM at SUSAS 2023
Photographer: LeleTown Model: Xiaoxiao Zhao
From the perspective of Information Ecology, the “Maelstrom” team attempts to respond and supplement the theme of this year’s Shanghai Urban Art Season, “Metro-biosis”: “Metro-biosis, or co-existence, as an everlasting topic between humans and their environment, encompasses not just our relationship with the ecological and urban surroundings, but also our dynamic relationship with the surrounding information environment from an individual standpoint. How we process the information within our perceptual range and utilize new tools to harmonize our relationship with information flows is an emotional and cognitive challenge, a dual test posed by personal experiences and environmental scales.”
The exhibition is opening on 2023 September 24th in Shanghai. For more information, check https://www.susas.com.cn/
IMAGE
Maelstrom at Spectra Studio LA
The end of Oct 2023, Maelstrom was premiered as a unique experience combining dome projection and spatial audio, remotely connecting our memories from Shanghai to Los Angeles. This dome projection + 7.1 spatial audio presents the state of art of multi-media experience.
Hardware Collaborator: Harvey Moon
Audio Support: Brian Cantrell
