AVTensor: A Rust Media Decoder Built for Training Pipelines
Runway's open-source Rust library AVTensor decodes audio and video in one pass, eliminating silent sync bugs in training data and boosting MFU by 1.8 points.
Runway engineers uncovered a silent synchronization bug in their audio-visual training pipeline: some MP4 assets had video and audio streams starting from different zero points due to negative timestamps and variable frame rates, and their existing setup combining torchcodec and torchaudio misaligned the two streams. The result was a frozen first frame while audio played on, teaching the model that lips and speech are only loosely coupled—a data problem that looked deceptively like an architecture issue.
To fix this at the root, the team built and open-sourced AVTensor, a Rust library that talks directly to FFmpeg's C API. It demuxes a container in a single pass, guaranteeing audio and video share the same timeline, pre-allocates output tensors to avoid mid-decode reallocation, and releases Python's GIL so thread-based dataloaders keep running during decode. The API deliberately seeks by timestamp rather than frame index and streams directly from object storage, cutting out redundant downloads and memory copies.
The rewrite delivered a measurable 1.8 percentage-point gain in training MFU and demonstrates Rust's memory-safety benefits in a domain still dominated by C and C++. The broader lesson for engineers: subtle timing bugs in the data pipeline can masquerade as architectural flaws, and root causes are often best sought in the most fundamental layer—data reading and decoding.