SPATIAL INSTRUMENTS
GAP +
2021
Architectures of Sound
Upon the topics of post-colonial and communism identities of Kulangsu island, from a standpoint of invisible architecture, “GAP+” emerges as a comprehensive synthesis of my explorations in algorithms, sound/media art, and installation. It embodies an experimental spirit that seeks to dissolve the boundaries of traditional architectural design. I envisage architecture as a generative system, a crucible for new forms and possibilities, fueled by diverse theoretical frameworks.
Drawing inspiration from the “live set” music concept — where the mechanisms of sound mixing shape the aural space — GAP+ offers a critical lens on the conservation of “historical” urban spaces. It constructs an auditory landscape, a soundscape that engages with the commercial-dominated urban soundscape. GAP+ redefines architectural presentation, introducing novel concepts of “real-time” and “remote” experiences within the built environment, in form of spatial instrument.
Through the spatialization of the Live Set framework in electronic music, it delves into the mechanisms of spatialized mixing within an acoustic space, conceptualizing it as an interactive acoustic museum. In this installation, gestures unlock the potential for real-time sound generation, transforming each audience into a performer, to re-interprete the sounds collected on Kulangsu. The narrative gives rise to two distinct sets on different scales: “GAP+” embodies architecture, offering a structural expanse, while “GAP-” serves as an accessible instrument/installation, inviting engagement from all.
ACOUSTIC GARDEN
2021
Augmenting Decisions and Experiences at Walkable Scale through AR Spatial Audio
In the realm of AR/XR design, spatial audio and audio-driven narratives are generally considered as secondary roles. The production of non-visual oriented content and research into accessibility for the visually impaired user group remain inadequately addressed. Meanwhile, within the music field, in discussions over gesture-based sound synthesis and interactive music, complex visual interfaces are prioritized. The potential for intuitive, audience-driven interactive musical experiences remains largely untapped. In the domains of architecture and music, scholarly investigations into spatial concepts and musical sequences have been longstanding. However, these studies have predominantly been anchored in symbolic representations on print media.
In this project, inspired by the progressive structure of electronic music, we introduce a spatialized sound synthesis method based on distance-related audio effect modulation combined with binaural spatialization. This approach is intended to navigate users through space without depending on visual indicators, utilizing auditory cues from a multitude of virtual audio objects instead. These objects are responsive to user movements, providing immersive musical experiences on a walking scale. Interaction with specific audio objects enables users to dictate different musical progressions, which follows a self-similar spatial structure.
Distinct from conventional audio spatialization and sonification techniques which cater to simulation and direct translation of data, our method emphasizes the affective feedback of the translated auditory messages. It reveals the potential of interactive, multi-faceted musical narratives driven by spatial interactions, allowing audiences to participate as performers. During our experiments across desktop applications, handheld AR and AR headsets, we encountered challenges from sound design, hardware constraints, and the human brain’s capacity for information processing.
MAELSTROM
2023
Survival in the Ever-evolving Whirpool of Message
Maelstrom explores navigating today’s data overload in the form of an audio-visual installation and an ever-evolving spatial instrument. It examines the impact of immersive technology on perception and memory, offering a novel approach to managing information through pattern recognition.
‘Maelstrom’ presents a journey through the symbiotic relationship between humans and the environment, with a groundbreaking real-time audio-visual interactive interface. This piece reflects on the prevailing overflow of data and technology that shapes our information ecosystem and the sensory overload imposed on humanity. It delves into the impact of “immersive” information environments and generative technologies on our cognition: within today’s data ecosystem, how do immersive experiences and generative technologies influence our capacity to process information? How do we filter pertinent information, discern the operational patterns of our surroundings, and find a way amidst the tumult of information flows?
How can we perceive patterns in otherwise seemingly chaotic and unpredictable streams of activity as a means of not only surviving but also embracing the inevitable interdependence of an unpredictable future? Marshall McLuhan recognized parallels of today to Edgar Allan Poe’s novel “A Descent into the Maelstrom.” In this short story, a sailor becomes so absorbed in thought that he is caught in a maelstrom. Realizing he cannot row his boat out of the situation, he must observe the objects within the maelstrom and grab onto the recurring ones as a means of escape. The lesson: only through pattern recognition can we cope with information overload. This vortex embodies the challenges of our information age today, and the exhibit replicates the process of “detachment” through information overload, yet it possesses cyclical patterns in time. Through such sensory patterns, collective memories and habits become an inheritable feature of environment, offering a way out of the maelstrom.
Commissioned by SUSAS (Shanghai Urban Space Art Season), this audio-visual installation, developed in Unreal Engine and utilizing Dolby Atmos spatial audio technology, uses 7.1 surround sound and generative projection, empowering the audience to create their own unique responsive spatial auditory experience by shifting the original map and soundscape of Shanghai. It explores the concept of “information ecology” and perceptive memories. The installation plunges visitors into a sensory vortex, providing an innovative guided experience. It helps us embrace the overwhelming forces of data today to bridge the gap between evolution and technology, and between technology and humanity.
SPATIAL DAW
Begin with GAP+, the series of research
ARTICLE
PUBLICATION
WORLD BUILDING FOR AN ACOUSTIC FUTURE
IML 475 - JUNK World Building
IML 521 - Looks Good, Sounds Great
Music/Soundtracks
Consultation Work: Refik Anadol Studio
Sense of Space : Connectome AI Architecture
Computational Designer (Form Study & 3D Printing)
Casa Batllo - Living Architecture
Computational Designer (Geometry Generation & Animation of Chapter 2)
Zaha Hadid - Architecting the Metaverse
Computational Designer (Unreal Engine visualization and geometry reconstruction of DALLE 2 Image)
MoMA - Unsupervised
Previs and visualization