At-the-Roofline Sparse Tensor Contractions on Vector Processors for Inference
Ventaglio enhances sparse tensor contractions on vector processors, boosting inference performance significantly.
Fine-grained weight pruning and activation sparsification have become effective methods for reducing the compute and memory costs of inference in Transformer models. However, existing RVV architectures do not natively support these patterns, limiting performance. The Ventaglio system introduces a runtime-configurable sparse execution unit that, along with RVV ISA extensions, enhances the execution of sparse tensor contractions. Integrated into an open-source vector processing cluster, Ventaglio achieves significant speedups of 6.9 to 7.4 times over optimized baselines, demonstrating its potential for improving inference efficiency.