Why Rust services hold onto memory: glibc, jemalloc and munmap

A team running an event-driven Rust service (consuming from Kafka/Redis Streams/NATS) noticed that RSS stayed pinned near its peak long after a burst of ~100k events finished processing. Heap profiling with dhat showed that nearly all allocations were freed by the end of the run, ruling out a classic memory leak. The real cause traced back to glibc's ptmalloc arena model: memory blocks from concurrently running Tokio tasks get interleaved within arenas, and the heap can only shrink from its top chunk downward. Even when short-lived tasks free their memory, unconsolidated free chunks or a single still-live allocation can act as a 'deadbolt,' trapping everything below it and preventing the OS from reclaiming pages.

The service's bursty pattern — concurrency capped by a Semaphore and tasks scheduled across arenas via Tokio's work-stealing runtime — made this worse. Thread arenas grow using mmap-backed sub-heaps, and if the topmost sub-heap retains unconsolidated free chunks, older sub-heaps beneath it stay mapped even when empty. Because RSS never exceeded its previous peak, each new burst simply reused the stranded free chunks from the last one, keeping memory flat and high indefinitely. Manually calling malloc_trim(0) dropped memory to baseline instantly, confirming the diagnosis, but there was no safe, deterministic point to invoke it in a production request path.

Switching the global allocator to jemalloc (via tikv_jemallocator in Rust) resolved the issue, letting memory settle back down after each burst. The takeaway for engineers: elevated RSS in Kubernetes doesn't always mean a code-level leak — allocator internals, especially arena-based heap trimming in glibc, can hold freed memory hostage under bursty, sparse workloads, and allocator choice can be a decisive fix.