PHYSIS: PHYsically informed and Semantically controllable Interactive Sound synthesis

PHYSIS is an industrial project focused on the modeling, transformation and real-time synthesis of diegetic sounds for interactive virtual worlds (video games, simulations, serious games…) and augmented reality. By diegetic sounds, we mean all sounds generated by identifiable objects in the virtual scene (e.g. glass, weapons, liquids, fire, water, fabric…) and their possible interactions: physical impacts, footsteps, slidings, rollings, sweepings… It does not comprise musical sounds. 

PHYSIS seek to improve our knowledge about of how to create interactive real-life sounds and how to interact in real-time with them using semantic and physical controls or emergent tangible interfaces. Moreover, PHYSIS has a systemic approach and considers the problem in its entirety: from the necessary core researches on sound analysis and modelling up to optimized compiled code and interactive synthesis models and transformations. PHYSIS will also provide high level analysis and synthesis tools targeting potential industrial users like video game creatorsthat give interactive semantic control over the generated content. 

Principal results of PHYSIS will lead to: 
• significant advances in modeling acoustic properties of the targeted sound corpus, 
• new synthesis models and synthesis engine strategies targeted at creating and transforming sounds interactively with high level semantic and physical controls, 
• technological demonstrators of the benefits of such innovations. 

Past researches have focused a lot on the synthesis of musical sounds and voices, with very good results both from scientific standpoints (mpeg7 norm, MIR community and MIREX contests…) and industrial standpoints (Applied Acoustics, founded by a former Ircam researcher, Acapela, now working with Apple, Vocally, Native Instruments…). The synthesis of other types of sounds, however, is largely underexplored. For instance, the Music Information Retrieval (MIR) research community has no equivalent for non-musical sounds. 

The recent emergence of complex video games and virtual universes populated with hundreds of objects and situations, such as Second Life, has increased the need for new and efficient solutions to generate sound automatically rather than recording all sounds in advance and playing them back in an almost un-interactive way. Sound in today’s video games is still produced through pre-recorded audio when image is purely synthesized in real-time! Moreover, simulations and serious games are based on physical models which make them more realistic. Such techniques produce information used to render graphics in a natural way, but pre-recorded audio can’t use that valuable information. The rise in available processing power has made possible the precise simulation of audio and acoustic properties of diegetic sounds based on physical parameters or information and PHYSIS will bring the missing link. 

Moreover, equipments like smartphones, tablets, video game consoles and even computers now integrate new kinds of interfaces, starting a real revolution in the way we access digital information. Smartphones are now equipped with tactile interfaces, Nintendo’s Wii has a gestural input interface, Microsoft’s Xbox 360 has the Kinect, a high performance camera capturing body positions, Sony’s PlayStation 3 has the Move, a device detecting movements and a camera detecting static positions, the new WiiU has a touch screen resembling the iPad, and Windows 7 now supports touch screen by standard. Real-time interactive audio synthesis is perfectly suited to those new interfaces and new usages arising from them. 

In conclusion, PHYSIS will create a solid scientific and technological ground to respond to all aforementioned new challenges. 

PARTENAIRES

AudioGaming AudioGaming

CNRS DR 12_LMA  Centre National de la Recherche Scientifique Délégation Provence et Corse_Laboratoire de Mécanique et d’Acoustique

GameAudioFactory GameAudioFactory

Ircam Institut de Recherche et Coordination Acoustique/Musique