How Do I Use `setrlimit` To Limit Memory Usage? Rlimit_as Kills Too Soon; Rlimit_data, Rlimit_rss, Rlimit_stack Kill Not At All

I'm trying to use setrlimit to limit my memory usage on a Linux system, in order to stop my process from crashing the machine (my code was crashing nodes on a high performance clus

Solution 1:

Alas I have no answer for your question. But I hope the following might help:

• Your script works as expected on my system. Please share exact spec for yours, might be there is a known problem with Linux distro, kernel or even numpy...
• You should be OK with RLIMIT_AS. As explained here this should limit the entire virtual memory used by the process. And virtual memory includes all: swap memory, shared libraries, code and data. More details here.

• You may add the following function (adopted from this answer) to your script to check actual virtual memory usage at any point:

  defpeak_virtual_memory_mb():
      withopen('/proc/self/status') as f:
          status = f.readlines()
          vmpeak = next(s for s in status if s.startswith("VmPeak:"))
          return vmpeak
  

• A general advice, disable swap memory. In my experience with high performance servers it does more harm than solves problems.