Speech-derived music: computational and electromechanical methods in DIY sound art

This practice research thesis develops an experimental artistic practice in which musical material is computationally derived from speech and realised through lo-fi, DIY electromechanical devices. The widespread use of speech as a compositional source points to its exceptional richness, including acoustic features as well as emotional, semantic, and syntactic qualities. While computational approaches to speech-derived music are well-established, their application within electromechanical sound-art remains underexplored. Through seven creative outputs, this research investigates how speech features can be transformed into computer-controlled acoustic sound, shaped by algorithmic processes and the material limitations of found objects.

Varying in terms of materials, interfaces and source speech-features, these works are structured in three stages reflecting the application of particular computational tools and methods: the algorithmic transcription and mapping of speech audio, the incorporation of Neural Audio Synthesis (NAS) as RAVE (Realtime Audio Variational autoEncoder) generated audio, and the use of text and language models to generate linguistic structures. Each investigates a unique speech feature, mapping method, and compositional application, reflecting on the technical and creative processes specific to its physical realisation. Improvisation is central to this process, from experimentation with objects and actuators, to electric motors and hardware, interfaces, performance, and composition.

The project finds that speech-derived features have applications to both compositional material and structure: as improvisational palettes, defining formal structures such as the gradual emergence of coherent text, and establishing generative musical systems that draw on underlying syntactic and semantic patterns. The act of translating high dimensional computational features into low-resolution mechanical movement results in an approximate transcription of the source material. This imperfect rendering is developed as a generative method, in which algorithmic abstraction and the physical unpredictability of the devices create new musical outputs with traces of the original speech source.

The contributions include field reviews providing an overview of current practice; a set of compositional strategies for mapping speech and language to DIY electromechanical devices, methods for the performative shaping of algorithmic output, and techniques for using syntax and semantic data as a non-standard control signal for musical structure; publicly available code for each method and device; and RAVE audio and LSTM text models trained throughout the research. These methods provide practical ways of working with speech in experimental artistic practice, offering future directions for speech-derived music, sound-art, and DIY electromechanical performance.