Flint: Efficiently Leveraging High Bandwidth Flash for LLM Inference

LLM inference is increasingly bottlenecked by accelerator memory capacity rather than compute, especially on single-accelerator or small-node systems. High Bandwidth Flash (HBF), an emerging 3D-stacked NAND technology offering multi-terabyte near-accelerator capacity, is a promising fix, but existing proposals struggle with coarse static prefetching, refresh operations leaking into the critical inference path, and missed workload-specific optimizations.

FLINT addresses these gaps with three mechanisms: a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads to sustain bandwidth using existing NAND buffers, a phantom-plane refresh scheme that offloads refresh operations outside the critical read path via low-cost resource duplication, and a read-only Flash Translation Layer that swaps general SSD-style write support for a compact table mapping logical weight bursts to physical HBF locations.

By integrating HBF as a scalable capacity tier alongside HBM, FLINT aims to let engineers deploy larger LLMs on constrained hardware without sacrificing inference throughput, offering a concrete architectural path for capacity-scalable inference systems.