Memory Profiler For Numpy

I have a numpy script that -- according to top -- is using about 5GB of RAM: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16994 aix 25 0 5813m 5.2g 5.1g S

Solution 1:

Numpy (and its library bindings, more on that in a minute) use C malloc to allocate space, which is why memory used by big numpy allocations doesn't show up in the profiling of things like heapy and never gets cleaned up by the garbage collector.

The usual suspects for big leaks are actually scipy or numpy library bindings, rather than python code itself. I got burned badly last year by the default scipy.linalg interface to umfpack, which leaked memory at the rate of about 10Mb a call. You might want to try something like valgrind to profile the code. It can often give some hints as to where to look at where there might be leaks.