My Rust Service's Memory Never Went Down. It Wasn't a Leak.

The symptom

Idle service. 33 MB of live data. RSS parked at 77 MB. For days. Deploy something, memory spikes +20 MB, and the spike never comes down.

First instinct: leak hunt. A 2000-request soak test over the per-request paths, the heartbeat loop, the background jobs. All flat. Hours of my life.

Two decisions got it to 33 MB.

Decision one: switch to jemalloc

The system allocator (glibc malloc) doesn't really give memory back to the OS. It frees internally and keeps the pages; the kernel only reclaims them by swapping under pressure. Your RSS becomes the high-water mark, forever.

jemalloc gives you control over how quickly freed pages are released to the OS. For a service sharing a box with other containers, "RSS tracks what I actually use" is the behavior I wanted. That switch alone fixed the worst of it.

Decision two: make the policy match the workload

jemalloc's defaults hold freed pages for 10 seconds before returning them, spread across 4 arenas per CPU. Holding pages is a cache. If you're a database with a steady hot set, that cache is gold.

My orchestrator is not a database. Its workload is a burst of work (deploys, TLS handshakes, config builds) and then nothing for a while. Pages retained during idle aren't a cache. They're rent.

_RJEM_MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0,narenas:2

Muzzy pages (already handed to the kernel via madvise) unmap immediately. For the rest, I tested dirty decay at 0: about 2 MB better, and a stream of syscalls. One second was the balance.

The numbers

On the 4-core master (which runs 9 projects, so its RAM is a shared pool), defaults meant 16 arenas and 77 MB idle for 33 MB live. With the change: 33 MB, and deploy spikes drain in seconds.

On the 1-core 512 MB box the win was modest, because at 1 CPU, 4 arenas was already near-correct. The sprawling-arena problem is mostly a multi-core problem.

If your service runs hot all day, keep the defaults. If it bursts and idles, tell the allocator to give the RAM back. Measure first: RSS and live allocations side by side, and the CPU cost as you shorten decay. I checked: contention from fewer arenas costs you under heavy allocation, and this service sits at a few percent CPU. No point hoarding for throughput we don't have.

Two gotchas

• tikv-jemalloc ignores plain MALLOC_CONF. It must be _RJEM_MALLOC_CONF, because the crate prefixes its symbols.
• narenas does most of the work on multi-core boxes; the decay timer drains what's left.

The takeaway

If your long-running service's RSS looks like its peak history instead of its current usage, it's probably not a leak. It's the allocator keeping score.