Description of problem:

Discussed in bug 862177 and bug 1062288, newer kernels (RHEL7, likely RHEL6, latest upstream) contain the following change:

http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/mm/readahead.c?id=47db8bd21e9914006b3bd2b9e90e0aaf2c04cbd2

While this change may be beneficial for some NUMA systems, it completely changes the readahead(2) behavior for what seem to be all systems (x86 arch at least) by limiting the readahead size.


# cat readahead.c 
#define _GNU_SOURCE
#include <unistd.h>
#include <fcntl.h>

int main(int argc, char **argv)
{
    int fd;
    off_t fsize;

    (void)argc;
    fd = open(argv[1], O_RDONLY);

    fsize = lseek(fd, 0, SEEK_END);
    lseek(fd, 0, SEEK_SET);

    readahead(fd, 0, fsize);

    return 0;
}

# gcc readahead.c -Wall -Wextra -o readahead
# dd if=/dev/zero of=bigfile bs=500M count=1


kernel-3.10.0-98.el7:

# echo 3 > /proc/sys/vm/drop_caches; free -m; ./readahead bigfile; free -m
             total       used       free     shared    buffers     cached
Mem:          1841        135       1705          8          0         19
-/+ buffers/cache:        116       1725
Swap:            0          0          0
             total       used       free     shared    buffers     cached
Mem:          1841        635       1205          8          0        519
-/+ buffers/cache:        115       1725
Swap:            0          0          0

kernel-3.10.0-99.el7:

# echo 3 > /proc/sys/vm/drop_caches; free -m; ./readahead bigfile; free -m
             total       used       free     shared    buffers     cached
Mem:          1841        136       1704          8          0         19
-/+ buffers/cache:        116       1724
Swap:            0          0          0
             total       used       free     shared    buffers     cached
Mem:          1841        138       1702          8          0         21
-/+ buffers/cache:        116       1724
Swap:            0          0          0


This is a regular qemu-kvm based virtual machine,

# numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2
node 0 size: 2047 MB
node 0 free: 1201 MB
node distances:
node   0 
  0:  10 

but the effect can be observed on several "real" amd64 machines as well.


The problem is that readahead(2) doesn't do what the manpage claims;
    "readahead() blocks until the specified data has been read."
is not true - on newer kernels, it has an artificial limit, which seems to be 2MB (512 * 4K pages?).

Unlike read(2), readahead(2) has no way of telling the userspace application how much data has been read, so - unless the application takes kernel internals into account - it has no way of reliably caching the entire file (or at least a >2MB portion).

This change can negatively affect performance of applications that rely on readahead(2) working correctly, eg. reading 'count' bytes if there's enough system memory.


Is the upstream aware of this limitation? I have found some concerns, like https://lkml.org/lkml/2014/1/10/122, but these seem to be left unaddressed.


Version-Release number of selected component (if applicable):
kernel-3.10.0-123.el7

How reproducible:
always

Actual results:
readahead(2) reads less than 'count' bytes

Expected results:
readahead(2) reads exactly 'count' bytes

Additional info:
This bug may also affect RHEL6 (as the original bug was cloned to RHEL6).