Physics-Based Sound Synthesis for Games and Interactive Systems
Description
This course introduces the basics of Digital Signal Processing and computational acoustics, motivated by the vibrational physics of real-world objects and systems. We will build from a simple mass-spring and pendulum to demonstrate oscillation, how to simulate those systems in the computer, and also prove that simple oscillation behaves as a sine wave. From that we move to plucked strings and struck bars, showing both solutions as combined traveling waves and combined sine wave harmonics. We continue to build and simulate more complex systems containing many vibrating objects and resonators (mandolin, drum, plate), and also learn how to simulate echos and room reverberation. Through this process, we will learn about digital signals, filters, oscillators, harmonics, spectral analysis, linear and non-linear systems, particle models, and all the necessary building blocks to synthesize essentially any sound. The free open-source software provided make it possible for anyone to use physical models in their art-making, game or movie sound, or any other application.
Schedule
Session 1: The Time Domain: Sound, Digital Audio, PCM Files, Noise Vs. Pitch, A Hint Of Spectra
a) Sound in Air, Traveling Waves b) Digital Audio, Sampling, Quantization, Aliasing c) Soundfiles, Wavetables, Manipulating PCM d) Pitch (vs. Noise), Spectral Analysis 0.1 e) Time-domain Pitch/Noise Detection: ZeroXings, AMDF, Autocorrelation
a) Sound in Air, Traveling Waves b) Digital Audio, Sampling, Quantization, Aliasing c) Soundfiles, Wavetables, Manipulating PCM d) Pitch (vs. Noise), Spectral Analysis 0.1 e) Time-domain Pitch/Noise Detection: ZeroXings, AMDF, Autocorrelation
Session 2: Physics, Oscillators, Sines & Spectra, Spectral/Additive Synthesis
a) Mass-Spring-Damper system, also simple Pendulum b) Fourier analysis/synthesis, Spectrum Analysis 1.0 c) More on additive Sine-wave synthesis
a) Mass-Spring-Damper system, also simple Pendulum b) Fourier analysis/synthesis, Spectrum Analysis 1.0 c) More on additive Sine-wave synthesis
Session 3: Digital Filters, Modal Synthesis
a) Digital Filters, Finite Impulse Response (FIR) b) Linearity, Time-invariance, Convolution c) Infinite Impulse Response (IIR) Digital Filters d) BiQuad Resonator Filter, Modal Synthesis
a) Digital Filters, Finite Impulse Response (FIR) b) Linearity, Time-invariance, Convolution c) Infinite Impulse Response (IIR) Digital Filters d) BiQuad Resonator Filter, Modal Synthesis
Session 4: Physical Modeling Synthesis: 1D Systems
a) 1-D systems, Strings, Modal (Fourier) Solution b) Strings II: Waveguide (D'Alembert) Solution c) 1-D systems, Bars, Tubes, solutions d) Advanced Waveguide Synthesis for 1-D systems
a) 1-D systems, Strings, Modal (Fourier) Solution b) Strings II: Waveguide (D'Alembert) Solution c) 1-D systems, Bars, Tubes, solutions d) Advanced Waveguide Synthesis for 1-D systems
Session 5: Physical Modeling II: 2 And 3-D Systems
a) 2-D systems, plates, drums, higher-order modes Fourier (Sine and/or Modal) Solutions, Waveguide Solutions b) 3-D systems, rooms, resonators, Meshes, Waveguide synthesis c) Resonator/Modal view and solution of 3-D systems Pop bottles and other lumped resonators
a) 2-D systems, plates, drums, higher-order modes Fourier (Sine and/or Modal) Solutions, Waveguide Solutions b) 3-D systems, rooms, resonators, Meshes, Waveguide synthesis c) Resonator/Modal view and solution of 3-D systems Pop bottles and other lumped resonators
Session 6: Subtractive Synthesis, Vocal Sounds And Models
a) Subtractive Synthesis, Voice Synthesis, Formants b) Linear Prediction, LPC c) FOFs d) FM Synthesis: Horns, Bells, Voices
a) Subtractive Synthesis, Voice Synthesis, Formants b) Linear Prediction, LPC c) FOFs d) FM Synthesis: Horns, Bells, Voices
Session 7: Grains, Particles And Statistical Models
a) Wavelets (just for completeness) b) Granular Synthesis c) Particle Models, Statistical Modal Synthesis d) Wind, Water, Surf, and Other Whooshing Sounds
Session 8: Extending And Refining Physical Synthesis Models
a) Waveshaping Synthesis, Distortion Modeling b) Time-Varying Systems c) Stiffness, All-Pass Filters, Banded Waveguides d) Commuted Synthesis e) JULIUS on KS, strings, demos
a) Waveshaping Synthesis, Distortion Modeling b) Time-Varying Systems c) Stiffness, All-Pass Filters, Banded Waveguides d) Commuted Synthesis e) JULIUS on KS, strings, demos
Session 9: Tying It All Together: Applications, Sonification, Interactions, And Control
a) Scanned Synthesis b) Don't forget the laptop!!! SMELT: c) Controlling Synthesis with game controllers (Wii, mobile TouchOSC, more) d) Walking Synthesis, a complete system e) Procedural Audio: Driving synthesis from process, game state, etc. f) Data set Sonification
a) Scanned Synthesis b) Don't forget the laptop!!! SMELT: c) Controlling Synthesis with game controllers (Wii, mobile TouchOSC, more) d) Walking Synthesis, a complete system e) Procedural Audio: Driving synthesis from process, game state, etc. f) Data set Sonification
Instructors
Perry R. Cook, Emeritus Professor of Computer Science (also Music) at Princeton University, founding advisor/consultant to social music company SMule, and consulting professor at CalArts, Stanford CCRMA, and University of Arizona
Julius O. Smith, rPofessor of music and (by courtesy) electrical engineering, CCRMA, Stanford University.