New Storage Technology Could Enable GPUs to Reach Terabyte Capacities

High Bandwidth Memory (HBM) used in GPUs may soon be complemented by a new storage technology called High-Bandwidth Flash (HBF), developed by Sandisk and SK Hynix. This innovation aims to provide SSD-like capacities at HBM-like speeds, potentially increasing memory capacity from hundreds of gigabytes to terabytes. While HBF offers significant advantages in capacity, it also presents challenges such as write endurance and access latency. The technology is expected to enhance AI systems' cost-effectiveness by supplementing HBM for read-heavy tasks.