llama.cpp CPU Offload Optimizations
Explore CPU offload optimizations for llama.cpp: disable CUDA graphs and target FFN sub-layers.
Recent tests on llama.cpp indicate that disabling CUDA graphs enhances CPU offloading efficiency. The recommended approach involves using --override-tensor to direct FFN sub-layers to the CPU, minimizing PCIe traffic while maintaining GPU processing for attention and KV tasks. This targeted offloading allows users to achieve better performance with reduced data transfer.