Instruction Foley: Multi-Instruction Foley for Video to Audio Generation
Anonymous
Vanilla Foley v.s Instruction-Foley: (a) Vanilla Foley, despite being a common foley manner, produces the entire audio only from video inputs once, demonstrating limitations in controllability and flexibility. (b) INSTRUCTION-FOLEY, in contrast, offers a more appealing alternative that employs multiple instructions to produce high-quality synchronized audio and can produce mixed audio in multi-stages, thereby exhibiting greater practical application.
Demos
It is recommended to use earphones to hear the demos videos, raise the volume and zoom in the videos.
Ground Truth
SpecVQGAN
Diff-Foley
Instruction-Foley (Ours)
The train moving down the tracks, the train horn being blown, and the traffic lights changing colors, all of which contribute to the overall soundscape of the scene.
The ball being hit by a baseball bat, the players running and shouting during the game, and the movements of the players on the field.
A goat vocalizing softly and then screaming loudly.
Crickets chirping very loudly.
A helicopter is taking flight with the blades slapping.
Camera muffling and wind blowing into a microphone as a vehicle engine runs while idle.
A man is playing the violin in front of a crowd, and another man is playing the cello, both of them producing sound through their instruments as they perform for the audience.