Maelstrom
With memory of patterns, we navigate through chaos towards an unpredictable future.
How does spatial medium convey non-textual languages through the depth of repetitive musical pattern? How do we survive in the chaotic environment of nowadays sensory overload and embrace the future of uncertainty? Maelstrom is a spatial instrument and audio-visual installation for everyone. It offers a means to navigate the whirlpool of overwhelming messages by providing an accessible audio-driven pattern recognition tool. Based on habits and memories of the past, the tool helps users to reorient themselves through recursive patterns of sound and visual in space, to further take decisive and mutual impacts on the ever-evolving environment. By challenging the capabilities and perceptive process of human brain, it critiques the politics of art and technology and sensory hierarchy of the prevailing ocular-centric culture.
Marshall McLuhan recognized parallels of today to Edgar Allan Poe’s novel “A Descent into the Maelström” – when a sailor gets caught in a maelstrom, by observing the pattern of objects in the vortex and grabbing onto the recurring ones, he finally escaped the chaotic situation. The instrument ‘Maelstrom’ is a journey through multi-sensory vortex, showing new perspectives upon the humanity and technologies. Guided by predictable musical patterns in turbulence, we navigate through a landscape of maps transformed by AI interaction, in which memories and habits allow us to adapt to the ever-evolving environment. Immersed in projection and spatial audio, audience tweaks a controller to move in the soundscape from chaotic center of the overwhelming vortex, to outer boundary of dim acoustic space. Audio informs movement and movement reshapes audio – with our innovative spatial music framework, the project delivers a user-performed multi-branch music structure, exploring beyond sonification, audio-spatialization and visualization. It interpretes musical progression as a spatial-temporal logic of all sensory elements, offering decision and cognition support.
This project is a part of Acoustic Garden Program, focusing on decision making and cognition support through affective computing and spatial music system.
2023
Built with Dolby Atmos, Unreal Engine/MetaSound
Artist/Director – Yufan Xie
Co-Writer – Mel Lewis
Sound – Ziqian Yin
Visual – Wei Wu
Commision by SUSAS 2023
2024 SXSW Innovation Awards – Audio Experience Category Finalist
2024 CHI Interactivity
Marshall McLuhan recognized parallels of today to Edgar Allan Poe‘s novel “A Descent into the Maelström” In this short story, a sailor becomes so absorbed in thought that he is caught in a maelstrom. Realizing he cannot row his boat out of the situation, he must observe the objects within the maelstrom and grab onto the recurring ones as a means of escape. The lesson: only through pattern recognition can we cope with information overload. This vortex embodies the challenges of our information age today, and the exhibit replicates the process of “detachment” through information overload, yet it possesses cyclical patterns in time. Through such sensory patterns, collective memories and habits become an inheritable feature of the environment, offering a way out of the maelstrom.
Commissioned by SUSAS (Shanghai Urban Space Art Season), this audio-visual installation, enveloped in Metasounds and Dolby Atmos as 7.1 channel generative audio, combining immersive projection, empowers the audience to perform their own responsive spatial auditory and visual experience by controlling their movement with a joystick. The installation plunges audiences into a vortex, offering an audio-guided experience of “escaping the chaos” – from the center to the edge of the vortex, the journey presents a transition from aurally-chaotic and visual guided narrative, to visually-limited and audio guided narrative.
Based on precedents of visualization, sonification, music, and architecture, this project focuses on human-centered narratives of acoustic-driven spatial interaction. It explores accessibility and affective feedback for general audiences, including visually impaired people, and resonates with the predictive and rewarding processes associated with music and the human brain. We introduced a new framework of spatial musical system that interprets messages via audio cues and real-time patterns influenced by users. It provides a critical perspective on linear narrative, graphical interface, and the prevailing sensory bias of ocular-centrism, offering a pattern recognition tool that augments individuals to navigate and cope with nowadays’ sensory overload, by putting non-visual sense as the core language to assist the decision-making process.
Visual Evolution
Generative visual iteration using DALL·E inpaint & variation. Interaction with the environment, including audio objects, take decisive impact on the devolution of maps. The AI generative process, becomes a spatial reference for, and defined by human. Habits in the past are inherited and changed through time.
The visual utilizes generative AI to create morphing maps. It places audiences at the center of a vortex-like landscape, where imagery is generated not merely through the image itself but through user’s movements. The installation employs DALL·E to evolve satellite images into real-time memory maps. Characteristics and memories inherited across generations facilitate the audience’s acclimation to new environments, based on past experiences. The similarity between previous and subsequent scenes forms sensory memories and habits, guiding audiences to faster familiarize and navigate scenes. Meanwhile, the audience’s choices on audio objects take decisive impact upon the morphosis of image variations, establishing a mutually influential relationship with the environment.
Music Progression
Audio guides movement, movement reshapes audio.
Auditorily, “Maelstrom” prompts audiences to learn and react to a self-similar recursive spatial-sonic narratives, where audio guides movement, and movement reshapes the audio, delivering a multi-sensory experience that can be accessed without visuals. With directional spatialization and our core technique of Distance-related Audio Effect Modulation (DRAEM), the installation extends traditional audio-spatialization and audio-visual experiences, into an interactive musical narrative accessed via simple movements and interactions. Inspired by one of typical progressive structures in electronic music, where musical progressions and affective feedback are created by modulating audio effects as linear variations applied to repetitive patterns – in DRAEM, the distance relationships between user and targets are reflected as audio cues of multiple audio effect modulation, by modulating parameters such as filter frequency, grain size of granular synths or pitch of sound. Informed by such relationships, users move in space with audio, and produce weaved musical progression in return. It defines music as spatial and non-linear narratives that unfold as multiple branches, beyond mere sonification and visualization. Accessible and shared by all, the instrument is not exclusively for professional musicians but for all general audiences.
MAELSTROM at SXSW 2024
MAELSTROM at SUSAS 2023
Photographer: LeleTown Model: Xiaoxiao Zhao
From the perspective of Information Ecology, the “Maelstrom” team attempts to respond and supplement the theme of this year’s Shanghai Urban Art Season, “Metro-biosis”: “Metro-biosis, or co-existence, as an everlasting topic between humans and their environment, encompasses not just our relationship with the ecological and urban surroundings, but also our dynamic relationship with the surrounding information environment from an individual standpoint. How we process the information within our perceptual range and utilize new tools to harmonize our relationship with information flows is an emotional and cognitive challenge, a dual test posed by personal experiences and environmental scales.”
The exhibition is opening on 2023 September 24th in Shanghai. For more information, check https://www.susas.com.cn/
IMAGE
Maelstrom at Spectra Studio LA
The end of Oct 2023, Maelstrom was premiered as a unique experience combining dome projection and spatial audio, remotely connecting our memories from Shanghai to Los Angeles. This dome projection + 7.1 spatial audio presents the state of art of multi-media experience.
Hardware Collaborator: Harvey Moon
Audio Support: Brian Cantrell
