Visual-effects grade AI that alters an actor's mouth movements to realistically match newly dubbed target audio.
LipDub, developed by MARZ, is a highly sophisticated visual-effects grade AI system that alters an actor's actual mouth movements in video footage to realistically match newly dubbed target language audio. The technology addresses the most visible quality problem in dubbed video content — the mismatch between a speaker's lip movements and the words in the dubbed audio track — by synthesizing new facial footage where the speaker appears to be naturally speaking the dubbed language. MARZ brings a visual effects production background to this technical challenge, applying VFX-grade image synthesis to produce results indistinguishable from naturally filmed performance in the target language.