Building an AI Dubbing Pipeline with faster-whisper and XTTS-v2
A step-by-step guide to building an open-source AI dubbing pipeline with faster-whisper, XTTS-v2 and FFmpeg that clones voices while keeping audio in sync.
This piece walks through hand-building an end-to-end AI dubbing pipeline that turns an English video into a Spanish version narrated in a cloned version of the original speaker's voice. It combines faster-whisper for timestamped transcription, a length-constrained LLM/MT step for translation, XTTS-v2 for voice cloning and synthesis, and FFmpeg for final assembly, explicitly aiming to demystify what turnkey tools like open-dubbing and KrillinAI do under the hood.
The core engineering challenge is that translated speech rarely fits its original time slot, since Spanish typically runs longer than English. The pipeline measures each synthesized segment against its allotted duration, absorbs small overflows into silence, applies gentle FFmpeg atempo stretching for moderate overflow, and re-translates with a tighter length budget for the worst outliers. This measure-then-fix loop is presented as the real differentiator in dubbing quality.
The guide also flags practical pitfalls: XTTS-v2's model weights carry a license restricting commercial use, voice cloning requires the speaker's written consent, numbers and code identifiers can be mangled by TTS, background music disappears unless stems are separated, and multi-speaker content needs diarization. Finally, it recommends shipping dubs as an additional audio track on the same file rather than as separate localized video files.