A Fake Python Memory Leak: ru_maxrss vs VmRSS Confusion
A RAG pipeline seemed to leak memory as RSS kept climbing, but the real bug was measuring ru_maxrss (a peak value) instead of live VmRSS. Fix, baseline check, and batching included.
While building a RAG ingestion pipeline that chunks web articles, generates embeddings, and pushes them into Qdrant, a developer noticed RSS memory readings kept climbing past a 1GB limit no matter how many gc.collect() or del calls were added. The real culprit wasn't a leak but a measurement bug: Python's resource.getrusage() ru_maxrss field reports the peak resident set size since process start, not current usage — so once memory touches a high point, the number never drops and every later limit check fails.
The fix was reading live VmRSS from /proc/self/status on Linux instead. After the change, logs showed memory rising and falling naturally, revealing that actual usage stabilized around 1.8GB after model load — no leak at all. It also turned out the 1GB limit was unrealistic from the start, since the multilingual-e5-base embedding model alone consumed about 1.6GB right after loading.
The author additionally improved throughput by batching embedding calls (50 chunks at a time) instead of encoding one by one, used torch.no_grad() to avoid unnecessary gradient buffers, and flagged the importance of e5-family models' required 'passage:'/'query:' text prefixes. The pipeline was redesigned to resume cleanly by tracking a qdrant_id per chunk and reprocessing only unprocessed rows after interruption. The key takeaway: before suspecting a memory leak, verify whether the metric you're reading reflects current state or a historical peak.