» Tag

llama.cpp

13 posts

Jul 31, 20269.02
Why reformatting prompts beats quantization for CPU LLM inference
Measured on a free ARM CPU box: rewriting prompts as label:value fact sheets cut LLM prefill time 6.5x while quantization tricks hurt accuracy.
cpu inference llama.cpp llm
Jul 12, 20268.90
9.9x Lower TTFT on Real Android Phone via llama.cpp KV Reuse
EdgeSync-LLM reuses llama.cpp's public KV APIs to skip reprocessing shared prefixes, cutting Android TTFT 9.9x while catching a fake 8.8x speedup.
android edge-ai inference-optimization kv-cache
Jul 12, 20268.43
Tesla P100's silent FP16 precision bug in llama.cpp fixed
A 3-line CUDA patch in llama.cpp fixes a years-old silent FP16 precision bug on Tesla P100 GPUs, with zero performance cost.
cuda gpu-inference llama.cpp local-llm
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com
Aug 11, 20268.30
Apple Silicon macOS VMs Get 11-16x Faster LLM Inference via Llama.cpp
Cua's Metal capability shim delivers 11-16x faster llama.cpp LLM inference in macOS VMs on Apple Silicon, with open benchmarks and source.
apple-silicon llama.cpp llm-inference macos
Jul 26, 20268.30
llama.cpp Adds Lossless F32 Compression via QFX32/QFX16
llama.cpp PR adds QFX32/QFX16 GGUF formats that losslessly compress F32 models by 2.05x with bit-identical weight reconstruction.
compression gguf inference llama.cpp
Jul 24, 20268.31
Quantprobe Runs a 110B-Parameter LLM on a 16GB RAM 2016 Desktop
Quantprobe shows how careful memory-tier placement lets a 2016 desktop with 16GB RAM run a 110B-parameter LLM, validated with pre-registered predictions.
benchmarking llama.cpp llm memory-management
Jul 18, 20268.31
MoE Expert Streaming Runs a 120B-Parameter LLM on an Android Phone
An open-source Android app streams MoE experts from flash storage to run a 120B-parameter LLM on phone CPUs, with 30B models at usable speeds.
android llama.cpp llm moe
Jul 16, 20268.30
Local Model Showdown Round 9: Qwen 3.6, Nemotron, Qwythos Coding Test
Five LLMs face off on a real coding task via llama.cpp on an RTX 5090 in Round 9 of the Local Model Showdown series.
benchmarking coding-agents llama.cpp llm
Jul 12, 20268.33
Intel Arc B70 32GB Runs Qwen3.6-35B-A3B at 130 t/s via Vulkan
Benchmark: Intel Arc Pro B70 32GB runs Qwen3.6-35B-A3B via llama.cpp Vulkan at ~130 t/s (4-bit, 262k ctx) and ~69 t/s hybrid 8-bit offload.
benchmark intel-arc llama.cpp moe
Jul 25, 20268.00
Speculative Decoding: The Free Speedup Most Local LLM Setups Skip
Speculative decoding speeds up local LLM inference 1.5-2.5x with identical output; 2026 saw it built into models via multi-token prediction.
gpu inference llama.cpp llm