Allocators: Memory Retention and Performance

1 · What an allocator actually does

Your program constantly needs scratch memory: a list grows, a request gets parsed, a buffer fills. When it needs memory it calls malloc; when it's done it calls free. But malloc and free don't talk to the operating system on every call, the overhead would be too much.

Instead, a library called the allocator sits between your program and the OS. It grabs memory from the kernel in big chunks, then hands out small pieces from those chunks as your code calls malloc, taking them back into its own pool on free.

Here's the part that surprises people who are abstracted away from allocators: when application code frees memory, the allocator can still keep it. In that case, it holds the freed space to satisfy your next malloc instantly, rather than handing it back to the kernel. So a long-running service can free nearly everything and still look "fat" to the OS — the memory is sitting idle inside the allocator's pool, reserved for reuse.

Every program already has an allocator whether you chose one or not. On a typical Linux Python image it's glibc's built-in ptmalloc2. You can swap it for another — jemalloc, tcmalloc, mimalloc — by setting one environment variable (LD_PRELOAD) at startup. No code changes. That's what makes this measurable: same binary, same traffic, only the allocator changes.

2 · Why memory climbs: fragmentation

If the allocator just kept a flat pool and reused it, footprint would plateau and we'd have no story. The reason it climbs upward instead is fragmentation.

Picture the allocator's pool as a long shelf, filled left to right. Some buffers are short-lived (a parsed request, a serialized response); some stick around for the whole process lifetime (caches, router tables, connection state). When the short-lived ones are freed, they leave gaps — but those gaps sit between long-lived buffers that are still in place.

The allocator usually can only hand a whole region back to the OS when everything in that returnable unit is free.

This is the RSS staircase: each burst of traffic carves new holes that get pinned under survivors, and total resident memory ratchets up step by step — even though much of it is technically free. The workload in this study is built to provoke exactly this: a /churn endpoint that keeps a large, bounded set of mixed-size (4–64 KB) buffers alive and frees a random subset on each call, so freed holes are reliably trapped beneath survivors. That's the fragmentation engine the allocators have to cope with.

3 · The experiment

One synthetic FastAPI web service, four uvicorn workers, no per-request recycling (recycling would mask the very fragmentation we're measuring). It's hit with a constant 200 req/s of mixed traffic for ten minutes, then left to idle. The only thing that changes between runs is the allocator.

A note on "settled." We also fire a forced memory.reclaim at the end, but with swap off it returns essentially nothing — the kernel can't evict dirty anonymous pages with nowhere to put them. So the real normalizer is the idle phase: memory is returned before the reclaim call, either during load (by proactive allocators) or during idle (by decay timers). "Settled" means "after the allocator's own return mechanisms have run," not "after we forced a reclaim." That distinction matters, and getting it wrong is an easy methodology trap.

4 · The headline: six allocators ranked

Here is the result with each allocator on its stock defaults
Faint bar = peak during load
solid bar = settled after idle
Sorted leanest-settled first.

Interpretation:

1. The leanest settled allocators are the modern mimalloc forks (bun and v3, ~1100 MB), with the default glibc surprisingly competitive (1209).
2. The bars tell a richer story than a single ranking: look at how differently each one moves from peak to settled. tcmalloc's two bars are almost identical (1216 → 1213) — it barely grabs more than it keeps.
3. jemalloc and mimalloc-v2 have a wide gap — they spike high under load, then claw a lot back during idle.

5 · Retain vs. return  |  why they differ

Every allocator faces a version of the choice from §1: when memory is freed, keep it nearby for reuse, or make some of it reclaimable by the OS? Returning memory sounds like the tidy default. It isn't free:

• It usually requires a syscall, such as madvise to discard physical backing from an existing mapping, or munmap to remove a mapping entirely.
• If that memory is touched again after being discarded, the process pays for a page fault, and the kernel may need to provide a freshly zeroed page.
• Changing mappings can require TLB invalidation on CPUs using the same address space — a real cross-core cost on a busy box.

Retained memory avoids most of that: already mapped, already faulted in, ready for reuse. So retention is a speed bet. Allocators such as jemalloc and tcmalloc were designed for allocation-heavy server workloads where malloc and free can sit on hot paths.

The cleanest way to think about retained memory is: it's a cache. With RAM to spare, it's useful headroom: memory the allocator can reuse without going back through the kernel. Under a hard container limit or on a packed machine, that same cache can become a liability: pages kept for future speed still count against your RSS until the allocator returns them.

The peak-to-settled gap is each allocator's fingerprint

That single policy explains the shapes in the chart above. Reading the gap between each allocator's two bars:

The mimalloc lineage is a real, large improvement

The three mimalloc builds are the same allocator family at three points in its evolution, and the progression is unambiguous:

6 · Why request latency barely moved

Across all six allocators, p99 request latency sat in a flat band of roughly 8–11 ms. That is the important negative result: allocator choice moved resident memory much more than it moved request latency in this service.

The likely reason is that allocator speed is not the dominant cost in this workload. Three layers explain why:

• Process-level concurrency. The service uses multiple uvicorn workers, but each worker runs Python bytecode under the GIL, so allocator-heavy multithreaded contention inside a single process is limited.
• pymalloc. CPython has its own small-object allocator that intercepts everything up to 512 bytes, so many small allocations never reach the allocator swapped in with LD_PRELOAD.
• Endpoint work. The handlers spend enough time parsing and processing JSON/XML that allocator speed is not a visible part of p99 request latency here.

That makes this study useful for comparing allocator-driven resident memory under this workload, but not a general allocator throughput benchmark. To expose raw malloc / free speed differences, allocations need to sit directly on the hot path.

7 · C++ control: touched churn throughput

To separate the allocator from the Python runtime, we reran the same six allocators in a bare C++ program: no GIL, no pymalloc, one process, four threads, and the same churn pattern sized to match the service's live set. This is not a pure malloc/free loop. Each allocation is touched page by page, one random survivor is freed, and the live set stays bounded. The metric is churn-with-page-touch throughput.

Memory collapses into a tight band. All six land between 718 and 760 MB — a 42 MB spread, versus the 600+ MB spread the service showed. The dumbbell below puts the two side by side: the hollow dot is the bare-C++ footprint, the filled dot is the service footprint, and the amber gap is mostly the added footprint from the runtime and mixed request workload.

In the C++ control, throughput spreads out. Three repeats kept the ordering stable, with max-min spreads under 2% for every allocator:

8 · The full data table

The six allocators on their stock defaults — every library running its own design philosophy, which is the only fair stock-vs-stock comparison. Sorted by settled, leanest first.

"C++ M/s" is the bare-C++ churn throughput from §7 (million allocations per second). The service columns are single-run, so read sub-2% gaps — mimalloc-bun vs v3 settled, or 8 vs 9 ms p99 — as ties, not rankings.

9 · What this means for you

Picking an allocator: choose by your constraint

There's no single winner, because peak and settled answer different questions:

• If your limit is sized for the busy moment (a container memory cap, an autoscaler threshold), peak is what kills you. Favor a proactive returner — tcmalloc had the lowest peak (1216) and barely bulges.
• If you care about steady footprint between bursts (density, cost per idle instance), settled is the number. The mimalloc v3/bun forks lead (~1100 MB).
• If you can't change the allocator at all, the default glibc is more competitive than its reputation (1209 settled) — its weakness is that it structurally can't return mid-heap holes, not that it's wasteful by default.

Final words:

Remember that retained memory is a cache that flips to a liability when constrained. Most deployments are constrained because RAM is expensive. Especially long running processes/workers with no built in recycling can suffer more from fragmentation and a choosing the right allocator and configuration might help along the way.

This is why container deployments tune memory down — capping glibc's arena count with MALLOC_ARENA_MAX, lowering its MALLOC_TRIM_THRESHOLD_, and shortening jemalloc/mimalloc decay timers: they trade a slice of allocation speed — which did not show up in p99 for this service — to keep the footprint under the limit.