Linux memory limits (rlimits)

It is useful, frequently, to set limits on the amount of memory any user program can consume. There are a few mechanisms for doing this.

But first, a look at the two types of memory limits available (ignoring stack). I refer to the man page for setrlimit(p). 

1. RLIMIT_DATA -- “The  maximum  size  of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2).” The man page for brk(2) warns that malloc(3) should be used instead of brk(2) or sbrk(2).
2. RLIMIT_AS -- “The maximum size of the process’s virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2)”

Limits can be set system-wide in /etc/security/limits.conf. The limit set here is a hard limit, i.e. users cannot manually increase their limits beyond the bounds set there, though they can set lower limits.

Limits can be set in the shell. With bash, it is the ulimit command. With tcsh, it is the limit command. In  bash, the "-d" option corresponds to RLIMIT_DATA, and the "-v" option corresponds to RLIMIT_AS. In tcsh, the "datasize" option corresponds to RLIMIT_DATA, and "vmemoryuse" corresponds to RLIMIT_AS. The "memoryuse" (maximum amount of physical memory a process may have allocated to it at a given time) option does not correspond to any rlimit: this is also noted in the man page for tcsh.

Limits can also be set using the setrlimit(p) system function.

It turns out that RLIMIT_DATA is completely ignored. It took a couple of days' worth of Googling and reading forum and mailing list postings to find this out. And then some testing on my own to confirm it.

The only memory-related limit that matters is RLIMIT_AS. I ran tests setting the limits in all three ways, and in both shells. No matter how limits are set, RLIMIT_DATA does not prevent malloc(3) from allocating memory. RLIMIT_AS, however, will terminate a program that tries to allocate too much.

Here is some abstracted output from my test program, hopefully self-explanatory:

Current system limits:
    cur data = 204800000 = 195 MiB
    max data = unlimited
    cur as = 716800000 = 683 MiB
    max as = 15734530048 = 15005 MiB

Allocating 680 MiB
        x[0] = 0
        x[11141120] = 11141120
        x[22282240] = 22282240
        x[33423360] = 33423360
        x[44564480] = 44564480
        x[55705600] = 55705600
        x[66846720] = 66846720
        x[77987840] = 77987840
Sleeping............. Done.

Current system limits:
    cur data = 204800000 = 195 MiB
    max data = unlimited
    cur as = 716800000 = 683 MiB
    max as = 15734530048 = 15005 MiB
    cur stack = 10485760 = 10 MiB
    max stack = unlimited

Allocating 683 MiB
allocation failed: Cannot allocate memory

A copy of glom.c is here.