56M-parameter LLM runs distributed across three ESP32-S3 boards via ESP-NOW

A developer built a distributed inference system that runs a 56M-parameter language model across three ESP32-S3 N16R8 microcontrollers connected over the ESP-NOW wireless protocol. The model, trained on WikiText-103 using Per-Layer Embeddings (PLE) inspired by Google's Gemma architecture, is quantized to 4-bit and its 50.3M-parameter PLE table is split across two boards to fit flash constraints, while a third board handles tokenization, the output head, and a web UI.

The most significant improvement came from adding a KV cache in PSRAM (1.5MB for 256 positions) on the board running the transformer core. Previously each token was processed with seq_len=1, so attention never saw prior context; with caching and correct RoPE position indexing, the model finally attends across the full generated sequence as designed, substantially improving output coherence.

Inter-board communication runs over a custom ESP-NOW protocol with 250-byte packet limits and 1-5ms round-trip latency, streaming generated text to a browser in real time via SSE. For embedded engineers, the project is a concrete demonstration of the tradeoffs involved in squeezing transformer inference onto microcontroller clusters through aggressive quantization, parameter sharding, and low-latency inter-chip messaging.