Investigating the cross-modal relationship between music and motion in an improvised music production context

When responding to music, humans move their bodies with various motional patterns varying in speed, spatial dimensions and continuity. This cross-modal association has been widely examined by motion-induction studies; however, it is less studied in a music-production context. This research examined the transformation from the visualized motion patterns into musical characteristics of performed sounds in a creative production environment. Pianists were required to play expressively on either a single tone or a sequence of musical tones several times after watching different video stimuli. The results revealed that perceived speed in visual stimuli had an impact on performance tempo; walking distance (increasing/decreasing) from the camera influenced performance volume; and movement continuity affected performance articulation. This is consistent with previous findings on music-motion analogies. This research also revealed several potential correspondence patterns: Visualized motional height had an impact on musical articulation (higher/staccato, lower/legato), and motional speed may influence musical loudness (faster/louder, slower/softer). This research implied that expressive performance intention of pianists is associated with specific movement patterns.

Keywords: cross-modal mapping, motion-music association, embodied cognition, musical production, expressive performance.