Benchmarking Inference Energy Costs of LLM: A LLaMA Study
Researchers benchmark LLaMA model sizes on V100 and A100 GPUs to analyze the energy and compute costs of LLM inference at scale.
While training costs for large language models get most of the attention, real-world deployments like ChatGPT rely heavily on inference, which remains understudied from an energy perspective. This paper benchmarks different sizes of Meta AI's LLaMA model on NVIDIA V100 and A100 GPUs using the Alpaca and GSM8K datasets, measuring both computational performance and energy consumption during inference. The experiments extend to multi-node, multi-GPU setups with model sharding across as many as 32 GPUs.
As LLMs see growing adoption in fields like law, finance, and medicine, understanding their resource footprint becomes essential for cost control, hardware planning, and efficient scaling. By quantifying the energy-performance tradeoffs across GPU generations and model sizes, this work gives engineers concrete data to make more informed infrastructure and deployment decisions.