Multimodal Models Fail at Sampling, Not Understanding
Multimodal models aren't bottlenecked by capability but by sampling defaults—frame rate, chunking, cropping—that silently limit what they perceive.
The article argues that production multimodal systems are usually bottlenecked not by model capability but by the sampling layer that turns video, audio, and images into tokens. Frame rate, chunk length, resolution, and crop are typically left at framework defaults, silently determining what a model can ever perceive regardless of how strong it is. Demos avoid this problem because they use short clips, single-tile images, and clean short audio—conditions where naive defaults happen to work.
In video, uniform frame sampling misses fast, localized events that fall between sampled frames, so the model isn't failing to reason—it never received the relevant frames. In audio, fixed-length chunking cuts across sentence and phoneme boundaries, quietly degrading transcription, diarization, and sentiment analysis in ways that look fine in aggregate benchmarks. In images, anything beyond the encoder's native resolution gets cropped or downsampled, and models tend to answer confidently about regions they never actually saw clearly, producing plausible-sounding hallucinations instead of admitting uncertainty.
The piece calls for treating sampling as a first-class architectural decision rather than plumbing: motion-aware frame selection, voice-activity-triggered chunking, and monitoring of crop/downsample ratios against failure reports. It also urges evaluation sets to include long videos, overlapping speech, and dense documents instead of only clean, easy inputs. The practical takeaway for engineers is that the model is usually good enough—the signal reaching it usually isn't.