model-capacity

New study finds language models memorize about 3.6 bits per parameter

Researchers unveil a method to measure LLM memorization capacity, finding GPT-style models store roughly 3.6 bits of information per parameter, with implications for grokking and scaling.