28.9M Parameter LLM Runs Fully On-Device on an $8 Microcontroller

A developer built a 28.9 million parameter language model that runs entirely on an ESP32-S3 microcontroller costing roughly $8, with no server connection, generating text at about 9 tokens per second. This is roughly a hundred times larger than the previous smallest on-chip model, which had only 260,000 parameters.

The breakthrough comes from applying Google's Per-Layer Embeddings technique, originally used in Gemma models, to microcontroller memory layout. Since most of a language model's parameters live in an embedding table that is only read from (not computed on), the 25-million-row table can stay in slow flash memory while only the few rows each token needs (~450 bytes) are pulled at inference time. The small compute core that actually reasons stays in fast SRAM, sidestepping the chip's 512KB SRAM ceiling entirely.

Trained on the TinyStories dataset, the model can only generate short, coherent stories — it cannot answer questions, follow instructions, or write code, since that reasoning capacity is bounded by the small compute core, not the memory trick. The significance here is architectural: proving a large model can be squeezed onto extremely constrained hardware, not what a 28.9M model itself can do. Firmware, training code, and full benchmark results have been released.