Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1464406 - Apache consumes too much memory for CGI output.
Apache consumes too much memory for CGI output.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: httpd (Show other bugs)
7.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Luboš Uhliarik
Maryna Nalbandian
:
Depends On:
Blocks: 1473612
  Show dependency treegraph
 
Reported: 2017-06-23 07:21 EDT by Tetsuo Handa
Modified: 2018-04-10 10:47 EDT (History)
6 users (show)

See Also:
Fixed In Version: httpd-2.4.6-76.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 10:45:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Apache Bugzilla 61222 None None None 2017-09-11 13:35 EDT
Red Hat Product Errata RHBA-2018:0826 None None None 2018-04-10 10:47 EDT

  None (edit)
Description Tetsuo Handa 2017-06-23 07:21:02 EDT
Description of problem:

  I am trying to develop an Apache CGI program written in C which handles
  upload/download of large files (magnitude of many gigabytes), and noticed
  that Apache consumes too much memory trying to spool the output from the CGI
  program if the CGI program started sending the output immediately.


Version-Release number of selected component (if applicable):

  httpd-2.4.6-45.el7_3.4.x86_64
  kernel-3.10.0-514.21.2.el7.x86_64


How reproducible:

  This problem seems to be timing dependent, but not difficult to trigger.
  I'm hitting this problem on a VM with 4 CPUs / 4GB RAM / no swap.


Steps to Reproduce:

(1) Compile below program as /var/www/cgi-bin/test.cgi , where the value of
    "total" should be something large that will fit or will not fit memory
    available.

----------
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
        static char buffer[1048576];
        unsigned long long total = 10 * 1024ULL * sizeof(buffer);
        unsigned int i;
        printf("Status: 200 OK\r\n");
        printf("Content-Type: application/octed-stream\r\n");
        printf("Content-Length: %llu\r\n\r\n", total);
        fflush(stdout);
	sleep(0);
        for (i = 0; i < total / sizeof(buffer); i++)
                write(1, buffer, sizeof(buffer));
        return 0;
}
----------

(2) Start Apache service.

# systemctl start httpd

(3) Start /usr/bin/top in order to monitor memory usage by
    /usr/sbin/httpd processes.

# /usr/bin/top -o %MEM

(4) Open another shell, and start below command line.

$ /usr/bin/curl http://127.0.0.1/cgi-bin/test.cgi > /dev/null

(5) Check the output of /usr/bin/top at (3).


Actual results:

  Case 1:

    The Apache worker process is killed by the OOM killer after the
    CGI program is killed (as a child of the biggest memory consumer)
    by the OOM killer. In this case, only current request from current
    client will be affected due to premature output. But the system
    might hang up completely due to lack of the OOM reaper support
    in the kernel.

  Case 2:

    The Apache worker process triggers HTTP 500 Internal Server Error
    due to unable to create child process for executing the CGI program
    if the Apache worker process is not killed by the OOM killer, for heap
    memory which holds the output from previous request is not freed before
    processing next request. In this case, future requests from future
    clients will be affected due to service unavailable. But sum of memory
    consumed by all Apache worker processes might after all lead to Case 1.

  Case 3:

    The Apache worker process survives. The amount of consumed memory changes
    if we do some sleep (e.g. change sleep(0) to sleep(1) in the source code
    shown above).


Expected results:

  The Apache worker process should not consume too much memory trying to
  spool the output from the CGI program. Even if spooling is necessary for
  some reason, it should not consume too much memory and it should free
  the memory after current request completed.
Comment 2 Tetsuo Handa 2017-06-26 01:45:41 EDT
Similar result with Fedora 25

  httpd-2.4.25-1.fc25.x86_64
  kernel-4.11.6-201.fc25.x86_64

and upstream version. Thus, I opened upstream report.

  https://bz.apache.org/bugzilla/show_bug.cgi?id=61222
Comment 4 Joe Orton 2017-07-13 10:59:21 EDT
Is this an otherwise default CGI configuration?

Thanks for checking against Fedora httpd.  Please file a support ticket, and we can investigate further.
Comment 5 Tetsuo Handa 2017-07-14 00:41:59 EDT
> Is this an otherwise default CGI configuration?
Yes, all default configuration.
Here is complete proceedure for RHEL 7.4 beta.



(1) Install "Basic Web Server" using rhel-server-7.4-beta-1-x86_64-dvd.iso .
    I created only / partition (XFS, 20GB) on a VM (4 CPUs, 4GB RAM).

(2) Install packages needed for compiling C programs.

# rpm -ivh gcc-4.8.5-14.el7.x86_64.rpm cpp-4.8.5-14.el7.x86_64.rpm glibc-devel-2.17-194.el7.x86_64.rpm glibc-headers-2.17-194.el7.x86_64.rpm kernel-headers-3.10.0-663.el7.x86_64.rpm mpfr-3.1.1-4.el7.x86_64.rpm libmpc-1.0.1-3.el7.x86_64.rpm

(3) Compile the CGI program.

# gcc -Wall -O3 -o /var/www/cgi-bin/test.cgi -x c - << "EOF"
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
        static char buffer[1048576];
        unsigned long long total = 10 * 1024ULL * sizeof(buffer);
        unsigned int i;
        printf("Status: 200 OK\r\n");
        printf("Content-Type: application/octed-stream\r\n");
        printf("Content-Length: %llu\r\n\r\n", total);
        fflush(stdout);
	sleep(0);
        for (i = 0; i < total / sizeof(buffer); i++)
                write(1, buffer, sizeof(buffer));
        return 0;
}
EOF

(4) Start the Apache server.

# systemctl start httpd

(5) Access the CGI program.

# curl http://127.0.0.1/cgi-bin/test.cgi > /dev/null



Before accessing the CGI:
----------------------------------------
top - 22:13:40 up 1 min,  2 users,  load average: 0.46, 0.20, 0.08
Tasks: 211 total,   1 running, 210 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  3873864 total,  3486932 free,   180872 used,   206060 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3444600 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  684 root      20   0  333912  28676   6712 S   0.0  0.7   0:00.97 firewalld
 1015 root      20   0  562416  16588   5904 S   0.0  0.4   0:00.16 tuned
  824 root      20   0  113376  15916   3428 S   0.0  0.4   0:00.07 dhclient
  642 polkitd   20   0  536984  13940   4936 S   0.0  0.4   0:00.08 polkitd
  637 root      20   0  212120  10092   3096 S   0.0  0.3   0:00.16 rsyslogd
  700 root      20   0  693412   9464   7072 S   0.0  0.2   0:00.20 NetworkManager
 2427 root      20   0  251252   8620   6124 S   0.0  0.2   0:00.17 httpd
    1 root      20   0  128120   6780   4012 S   0.0  0.2   0:02.57 systemd
 2361 root      20   0  341436   6484   4524 S   0.0  0.2   0:00.01 abrt-dbus
  640 root      20   0  231300   6176   4756 S   0.7  0.2   0:00.18 vmtoolsd
  630 root      20   0   99608   6084   4484 S   0.0  0.2   0:00.05 VGAuthService
  643 root      20   0  219132   5444   3796 S   0.0  0.1   0:00.03 abrtd
 2384 root      20   0  145728   5364   4100 S   0.0  0.1   0:00.16 sshd
 1786 root      20   0  145728   5360   4100 S   0.0  0.1   0:00.20 sshd
  644 root      20   0  216848   4692   3292 S   0.0  0.1   0:00.04 abrt-watch-log
  493 root      20   0   46352   4528   2764 S   0.0  0.1   0:00.57 systemd-udevd
 1013 root      20   0  106000   4056   3084 S   0.0  0.1   0:00.02 sshd
  490 root      20   0  121312   4044   2552 S   0.0  0.1   0:00.00 lvmetad
 2204 postfix   20   0   91804   4036   3024 S   0.0  0.1   0:00.00 qmgr
 2203 postfix   20   0   91736   4016   3004 S   0.0  0.1   0:00.00 pickup
 2429 apache    20   0  253468   3828   1304 S   0.0  0.1   0:00.00 httpd
 2430 apache    20   0  253468   3828   1304 S   0.0  0.1   0:00.00 httpd
 2431 apache    20   0  253468   3828   1304 S   0.0  0.1   0:00.00 httpd
 2432 apache    20   0  253468   3828   1304 S   0.0  0.1   0:00.00 httpd
 2433 apache    20   0  253468   3828   1304 S   0.0  0.1   0:00.00 httpd
 2428 apache    20   0  253264   3540   1040 S   0.0  0.1   0:00.00 httpd
  463 root      20   0   36832   3040   2720 S   0.0  0.1   0:00.22 systemd-journal
 2342 root      20   0  116048   2680   1668 S   0.0  0.1   0:00.06 bash
 2388 root      20   0  116048   2660   1644 S   0.0  0.1   0:00.02 bash
 2439 root      20   0  157856   2348   1536 R   0.3  0.1   0:00.06 top
  635 root      20   0  127892   2292   1720 S   0.0  0.1   0:00.07 smartd
 2184 root      20   0   91632   2132   1112 S   0.0  0.1   0:00.00 master
  646 dbus      20   0   34916   2060   1448 S   0.0  0.1   0:00.17 dbus-daemon
  638 root      20   0   24208   1660   1372 S   0.0  0.0   0:00.03 systemd-logind
  670 root      20   0  126236   1632   1012 S   0.0  0.0   0:00.44 crond
  636 root      20   0   21628   1300    968 S   0.0  0.0   0:00.03 irqbalance
  673 root      20   0   25860    912    712 S   0.0  0.0   0:00.00 atd
  607 root      16  -4   55456    888    484 S   0.0  0.0   0:00.01 auditd
  697 root      20   0  110048    820    696 S   0.0  0.0   0:00.02 agetty
  632 libstor+  20   0    8536    796    648 S   0.0  0.0   0:00.05 lsmd
 1026 root      20   0  115648    640    488 S   0.0  0.0   0:00.00 rhsmcertd
  633 root      20   0    4380    592    500 S   0.0  0.0   0:00.72 rngd
 1028 root      20   0  107896    284    184 S   0.0  0.0   0:00.00 rhnsd
----------------------------------------

After accessing the CGI (if OOM killed):
----------------------------------------
[  168.014205] irqbalance invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[  168.014210] irqbalance cpuset=/ mems_allowed=0
[  168.014212] CPU: 0 PID: 636 Comm: irqbalance Not tainted 3.10.0-663.el7.x86_64 #1
[  168.014214] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[  168.014215]  ffff9526743d9f60 0000000073aad44d ffff9525f6933840 ffffffffaf09fc54
[  168.014217]  ffff9525f69338d0 ffffffffaf09b0bf ffff9525f69338d8 ffffffffaecb66ab
[  168.014219]  ffff952677dbaae8 ffff9525f6933920 ffffffff00000202 fffeefff00000000
[  168.014221] Call Trace:
[  168.014231]  [<ffffffffaf09fc54>] dump_stack+0x19/0x1b
[  168.014233]  [<ffffffffaf09b0bf>] dump_header+0x90/0x229
[  168.014237]  [<ffffffffaecb66ab>] ? cred_has_capability+0x6b/0x120
[  168.014242]  [<ffffffffaeb860c4>] oom_kill_process+0x254/0x3d0
[  168.014243]  [<ffffffffaecb687c>] ? selinux_capable+0x1c/0x40
[  168.014245]  [<ffffffffaeb86906>] out_of_memory+0x4b6/0x4f0
[  168.014247]  [<ffffffffaf09bbc3>] __alloc_pages_slowpath+0x5d6/0x724
[  168.014250]  [<ffffffffaeb8cab5>] __alloc_pages_nodemask+0x405/0x420
[  168.014252]  [<ffffffffaebd0a08>] alloc_pages_current+0x98/0x110
[  168.014254]  [<ffffffffaeb8733e>] __get_free_pages+0xe/0x40
[  168.014256]  [<ffffffffaecb6ea4>] inode_doinit_with_dentry+0x2f4/0x570
[  168.014257]  [<ffffffffaecb793c>] selinux_d_instantiate+0x1c/0x20
[  168.014260]  [<ffffffffaecb02fb>] security_d_instantiate+0x1b/0x30
[  168.014263]  [<ffffffffaec17d72>] d_instantiate+0x52/0x70
[  168.014265]  [<ffffffffaec756aa>] proc_lookup_de+0xaa/0xc0
[  168.014267]  [<ffffffffaec756db>] proc_lookup+0x1b/0x20
[  168.014269]  [<ffffffffaec6fbcc>] proc_root_lookup+0x1c/0x40
[  168.014272]  [<ffffffffaec0a51d>] lookup_real+0x1d/0x50
[  168.014273]  [<ffffffffaec0daf3>] do_last+0xb93/0x1280
[  168.014441]  [<ffffffffc033f51c>] ? xfs_iunlock+0x11c/0x130 [xfs]
[  168.014444]  [<ffffffffaebdf1a6>] ? kmem_cache_alloc_trace+0x1d6/0x200
[  168.014446]  [<ffffffffaec10792>] path_openat+0xc2/0x490
[  168.014449]  [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90
[  168.014450]  [<ffffffffaf0a8308>] ? page_fault+0x28/0x30
[  168.014452]  [<ffffffffaec11f5b>] do_filp_open+0x4b/0xb0
[  168.014454]  [<ffffffffaec1f00a>] ? __alloc_fd+0x8a/0x130
[  168.014458]  [<ffffffffaebfee93>] do_sys_open+0xf3/0x1f0
[  168.014459]  [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90
[  168.014461]  [<ffffffffaebfefae>] SyS_open+0x1e/0x20
[  168.014463]  [<ffffffffaf0b0dc9>] system_call_fastpath+0x16/0x1b
[  168.014464] Mem-Info:
[  168.014468] active_anon:907589 inactive_anon:2437 isolated_anon:0
 active_file:142 inactive_file:474 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:4205 slab_unreclaimable:10052
 mapped:926 shmem:2523 pagetables:4171 bounce:0
 free:20546 free_pcp:10 free_cma:0
[  168.014470] Node 0 DMA free:15332kB min:276kB low:344kB high:412kB active_anon:160kB inactive_anon:0kB active_file:4kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1625 all_unreclaimable? yes
[  168.014474] lowmem_reserve[]: 0 2814 3765 3765
[  168.014476] Node 0 DMA32 free:53972kB min:50308kB low:62884kB high:75460kB active_anon:2751240kB inactive_anon:6236kB active_file:564kB inactive_file:1872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2884272kB mlocked:0kB dirty:0kB writeback:0kB mapped:680kB shmem:6480kB slab_reclaimable:8468kB slab_unreclaimable:18008kB kernel_stack:2208kB pagetables:12804kB unstable:0kB bounce:0kB free_pcp:40kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3748 all_unreclaimable? yes
[  168.014479] lowmem_reserve[]: 0 0 950 950
[  168.014481] Node 0 Normal free:12880kB min:16992kB low:21240kB high:25488kB active_anon:878956kB inactive_anon:3512kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1048576kB managed:973688kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:3612kB slab_reclaimable:8272kB slab_unreclaimable:22112kB kernel_stack:2512kB pagetables:3876kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[  168.014483] lowmem_reserve[]: 0 0 0 0
[  168.014485] Node 0 DMA: 4*4kB (M) 6*8kB (UEM) 4*16kB (UM) 2*32kB (UE) 1*64kB (U) 2*128kB (UE) 2*256kB (EM) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15360kB
[  168.014493] Node 0 DMA32: 716*4kB (UEM) 348*8kB (UEM) 110*16kB (UEM) 50*32kB (UEM) 31*64kB (EM) 11*128kB (UEM) 7*256kB (UEM) 2*512kB (UE) 2*1024kB (UE) 2*2048kB (EM) 8*4096kB (UM) = 54128kB
[  168.014499] Node 0 Normal: 470*4kB (UM) 209*8kB (UEM) 79*16kB (UEM) 26*32kB (UEM) 7*64kB (UEM) 1*128kB (E) 2*256kB (E) 0*512kB 4*1024kB (U) 1*2048kB (M) 0*4096kB = 12880kB
[  168.014507] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[  168.014517] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  168.014518] 3286 total pagecache pages
[  168.014519] 0 pages in swap cache
[  168.014520] Swap cache stats: add 0, delete 0, find 0/0
[  168.014521] Free swap  = 0kB
[  168.014522] Total swap = 0kB
[  168.014522] 1048445 pages RAM
[  168.014523] 0 pages HighMem/MovableOnly
[  168.014523] 79979 pages reserved
[  168.014524] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  168.014534] [  463]     0   463     9208      469      21        0             0 systemd-journal
[  168.014535] [  490]     0   490    30328      373      29        0             0 lvmetad
[  168.014537] [  493]     0   493    11588      442      23        0         -1000 systemd-udevd
[  168.014539] [  607]     0   607    13864      101      27        0         -1000 auditd
[  168.014540] [  630]     0   630    24902      400      43        0             0 VGAuthService
[  168.014541] [  632]   998   632     2134       37      10        0             0 lsmd
[  168.014543] [  633]     0   633     1095       23       8        0             0 rngd
[  168.014544] [  635]     0   635    31973      143      20        0             0 smartd
[  168.014545] [  636]     0   636     5407       83      15        0             0 irqbalance
[  168.014546] [  637]     0   637    53030     2065      42        0             0 rsyslogd
[  168.014547] [  638]     0   638     6052       72      16        0             0 systemd-logind
[  168.014549] [  640]     0   640    57825      382      60        0             0 vmtoolsd
[  168.014550] [  642]   999   642   134246     2251      57        0             0 polkitd
[  168.014551] [  643]     0   643    54783      412      59        0             0 abrtd
[  168.014552] [  644]     0   644    54212      351      58        0             0 abrt-watch-log
[  168.014554] [  646]    81   646     8729      153      19        0          -900 dbus-daemon
[  168.014579] [  670]     0   670    31559      155      19        0             0 crond
[  168.014580] [  673]     0   673     6465       50      18        0             0 atd
[  168.014582] [  684]     0   684    83478     5491      85        0             0 firewalld
[  168.014583] [  697]     0   697    27512       32      10        0             0 agetty
[  168.014584] [  700]     0   700   173353      600      88        0             0 NetworkManager
[  168.014586] [  824]     0   824    28344     3123      57        0             0 dhclient
[  168.014587] [ 1013]     0  1013    26500      243      55        0         -1000 sshd
[  168.014589] [ 1015]     0  1015   140604     2671      93        0             0 tuned
[  168.014590] [ 1026]     0  1026    28912       38      14        0             0 rhsmcertd
[  168.014591] [ 1028]     0  1028    26974       25      10        0             0 rhnsd
[  168.014592] [ 1786]     0  1786    36432      324      74        0             0 sshd
[  168.014593] [ 2184]     0  2184    22908      261      45        0             0 master
[  168.014595] [ 2203]    89  2203    22934      254      45        0             0 pickup
[  168.014596] [ 2204]    89  2204    22951      253      45        0             0 qmgr
[  168.014597] [ 2342]     0  2342    29012      254      16        0             0 bash
[  168.014598] [ 2361]     0  2361    85359      490      64        0          -900 abrt-dbus
[  168.014599] [ 2384]     0  2384    36432      321      74        0             0 sshd
[  168.014600] [ 2388]     0  2388    29012      254      15        0             0 bash
[  168.014602] [ 2427]     0  2427    62813      969     126        0             0 httpd
[  168.014603] [ 2428]     0  2428    63316      633     125        0             0 httpd
[  168.014604] [ 2429]    48  2429   957965   883970    1851        0             0 httpd
[  168.014605] [ 2430]    48  2430    63367      636     125        0             0 httpd
[  168.014606] [ 2431]    48  2431    63367      632     125        0             0 httpd
[  168.014607] [ 2432]    48  2432    63367      632     125        0             0 httpd
[  168.014608] [ 2433]    48  2433    63367      632     125        0             0 httpd
[  168.014609] [ 2439]     0  2439    39464      205      32        0             0 top
[  168.014611] [ 2440]     0  2440    44659      224      43        0             0 curl
[  168.014612] [ 2441]    48  2441     1299       21       7        0             0 test.cgi
[  168.014613] [ 2442]    48  2442    63367      631     125        0             0 httpd
[  168.014614] Out of memory: Kill process 2429 (httpd) score 914 or sacrifice child
[  168.014682] Killed process 2441 (test.cgi) total-vm:5196kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
[  168.024533] irqbalance invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[  168.024537] irqbalance cpuset=/ mems_allowed=0
[  168.024540] CPU: 2 PID: 636 Comm: irqbalance Not tainted 3.10.0-663.el7.x86_64 #1
[  168.024541] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[  168.024542]  ffff9526743d9f60 0000000073aad44d ffff9525f6933840 ffffffffaf09fc54
[  168.024544]  ffff9525f69338d0 ffffffffaf09b0bf ffff9525f69338d8 ffffffffaecb66ab
[  168.024546]  ffff952677dbaae8 0000000000000206 ffffffff00000206 fffeefff00000000
[  168.024547] Call Trace:
[  168.024555]  [<ffffffffaf09fc54>] dump_stack+0x19/0x1b
[  168.024557]  [<ffffffffaf09b0bf>] dump_header+0x90/0x229
[  168.024561]  [<ffffffffaecb66ab>] ? cred_has_capability+0x6b/0x120
[  168.024565]  [<ffffffffaeb860c4>] oom_kill_process+0x254/0x3d0
[  168.024567]  [<ffffffffaecb687c>] ? selinux_capable+0x1c/0x40
[  168.024569]  [<ffffffffaeb86906>] out_of_memory+0x4b6/0x4f0
[  168.024571]  [<ffffffffaf09bbc3>] __alloc_pages_slowpath+0x5d6/0x724
[  168.024573]  [<ffffffffaeb8cab5>] __alloc_pages_nodemask+0x405/0x420
[  168.024575]  [<ffffffffaebd0a08>] alloc_pages_current+0x98/0x110
[  168.024577]  [<ffffffffaeb8733e>] __get_free_pages+0xe/0x40
[  168.024579]  [<ffffffffaecb6ea4>] inode_doinit_with_dentry+0x2f4/0x570
[  168.024581]  [<ffffffffaecb793c>] selinux_d_instantiate+0x1c/0x20
[  168.024584]  [<ffffffffaecb02fb>] security_d_instantiate+0x1b/0x30
[  168.024587]  [<ffffffffaec17d72>] d_instantiate+0x52/0x70
[  168.024589]  [<ffffffffaec756aa>] proc_lookup_de+0xaa/0xc0
[  168.024590]  [<ffffffffaec756db>] proc_lookup+0x1b/0x20
[  168.024593]  [<ffffffffaec6fbcc>] proc_root_lookup+0x1c/0x40
[  168.024595]  [<ffffffffaec0a51d>] lookup_real+0x1d/0x50
[  168.024596]  [<ffffffffaec0daf3>] do_last+0xb93/0x1280
[  168.024630]  [<ffffffffc033f51c>] ? xfs_iunlock+0x11c/0x130 [xfs]
[  168.024632]  [<ffffffffaebdf1a6>] ? kmem_cache_alloc_trace+0x1d6/0x200
[  168.024634]  [<ffffffffaec10792>] path_openat+0xc2/0x490
[  168.024636]  [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90
[  168.024638]  [<ffffffffaf0a8308>] ? page_fault+0x28/0x30
[  168.024639]  [<ffffffffaec11f5b>] do_filp_open+0x4b/0xb0
[  168.024641]  [<ffffffffaec1f00a>] ? __alloc_fd+0x8a/0x130
[  168.024643]  [<ffffffffaebfee93>] do_sys_open+0xf3/0x1f0
[  168.024645]  [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90
[  168.024647]  [<ffffffffaebfefae>] SyS_open+0x1e/0x20
[  168.024649]  [<ffffffffaf0b0dc9>] system_call_fastpath+0x16/0x1b
[  168.024650] Mem-Info:
[  168.024653] active_anon:907589 inactive_anon:2437 isolated_anon:0
 active_file:142 inactive_file:437 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:4205 slab_unreclaimable:10052
 mapped:926 shmem:2523 pagetables:4171 bounce:0
 free:20515 free_pcp:94 free_cma:0
[  168.024655] Node 0 DMA free:15332kB min:276kB low:344kB high:412kB active_anon:160kB inactive_anon:0kB active_file:4kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1651 all_unreclaimable? yes
[  168.024659] lowmem_reserve[]: 0 2814 3765 3765
[  168.024661] Node 0 DMA32 free:53972kB min:50308kB low:62884kB high:75460kB active_anon:2751240kB inactive_anon:6236kB active_file:564kB inactive_file:1724kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2884272kB mlocked:0kB dirty:0kB writeback:0kB mapped:680kB shmem:6480kB slab_reclaimable:8468kB slab_unreclaimable:18008kB kernel_stack:2208kB pagetables:12804kB unstable:0kB bounce:0kB free_pcp:252kB local_pcp:116kB free_cma:0kB writeback_tmp:0kB pages_scanned:3812 all_unreclaimable? yes
[  168.024664] lowmem_reserve[]: 0 0 950 950
[  168.024666] Node 0 Normal free:12756kB min:16992kB low:21240kB high:25488kB active_anon:878956kB inactive_anon:3512kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1048576kB managed:973688kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:3612kB slab_reclaimable:8272kB slab_unreclaimable:22112kB kernel_stack:2512kB pagetables:3876kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[  168.024668] lowmem_reserve[]: 0 0 0 0
[  168.024670] Node 0 DMA: 4*4kB (M) 6*8kB (UEM) 4*16kB (UM) 2*32kB (UE) 1*64kB (U) 2*128kB (UE) 2*256kB (EM) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15360kB
[  168.024677] Node 0 DMA32: 716*4kB (UEM) 348*8kB (UEM) 110*16kB (UEM) 50*32kB (UEM) 31*64kB (EM) 11*128kB (UEM) 7*256kB (UEM) 2*512kB (UE) 2*1024kB (UE) 2*2048kB (EM) 8*4096kB (UM) = 54128kB
[  168.024684] Node 0 Normal: 439*4kB (UM) 209*8kB (UEM) 79*16kB (UEM) 26*32kB (UEM) 7*64kB (UEM) 1*128kB (E) 2*256kB (E) 0*512kB 4*1024kB (U) 1*2048kB (M) 0*4096kB = 12756kB
[  168.024691] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[  168.024693] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  168.024694] 3286 total pagecache pages
[  168.024695] 0 pages in swap cache
[  168.024696] Swap cache stats: add 0, delete 0, find 0/0
[  168.024696] Free swap  = 0kB
[  168.024697] Total swap = 0kB
[  168.024698] 1048445 pages RAM
[  168.024698] 0 pages HighMem/MovableOnly
[  168.024699] 79979 pages reserved
[  168.024699] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  168.024707] [  463]     0   463     9208      469      21        0             0 systemd-journal
[  168.024708] [  490]     0   490    30328      373      29        0             0 lvmetad
[  168.024710] [  493]     0   493    11588      442      23        0         -1000 systemd-udevd
[  168.024711] [  607]     0   607    13864      101      27        0         -1000 auditd
[  168.024712] [  630]     0   630    24902      400      43        0             0 VGAuthService
[  168.024714] [  632]   998   632     2134       37      10        0             0 lsmd
[  168.024715] [  633]     0   633     1095       23       8        0             0 rngd
[  168.024716] [  635]     0   635    31973      143      20        0             0 smartd
[  168.024717] [  636]     0   636     5407       83      15        0             0 irqbalance
[  168.024718] [  637]     0   637    53030     2065      42        0             0 rsyslogd
[  168.024720] [  638]     0   638     6052       72      16        0             0 systemd-logind
[  168.024721] [  640]     0   640    57825      382      60        0             0 vmtoolsd
[  168.024722] [  642]   999   642   134246     2251      57        0             0 polkitd
[  168.024723] [  643]     0   643    54783      412      59        0             0 abrtd
[  168.024724] [  644]     0   644    54212      351      58        0             0 abrt-watch-log
[  168.024726] [  646]    81   646     8729      153      19        0          -900 dbus-daemon
[  168.024727] [  670]     0   670    31559      155      19        0             0 crond
[  168.024728] [  673]     0   673     6465       50      18        0             0 atd
[  168.024729] [  684]     0   684    83478     5491      85        0             0 firewalld
[  168.024730] [  697]     0   697    27512       32      10        0             0 agetty
[  168.024731] [  700]     0   700   173353      600      88        0             0 NetworkManager
[  168.024733] [  824]     0   824    28344     3123      57        0             0 dhclient
[  168.024734] [ 1013]     0  1013    26500      243      55        0         -1000 sshd
[  168.024735] [ 1015]     0  1015   140604     2671      93        0             0 tuned
[  168.024736] [ 1026]     0  1026    28912       38      14        0             0 rhsmcertd
[  168.024737] [ 1028]     0  1028    26974       25      10        0             0 rhnsd
[  168.024739] [ 1786]     0  1786    36432      324      74        0             0 sshd
[  168.024740] [ 2184]     0  2184    22908      261      45        0             0 master
[  168.024741] [ 2203]    89  2203    22934      254      45        0             0 pickup
[  168.024742] [ 2204]    89  2204    22951      253      45        0             0 qmgr
[  168.024743] [ 2342]     0  2342    29012      254      16        0             0 bash
[  168.024744] [ 2361]     0  2361    85359      490      64        0          -900 abrt-dbus
[  168.024745] [ 2384]     0  2384    36432      321      74        0             0 sshd
[  168.024746] [ 2388]     0  2388    29012      254      15        0             0 bash
[  168.024748] [ 2427]     0  2427    62813      969     126        0             0 httpd
[  168.024749] [ 2428]     0  2428    63316      633     125        0             0 httpd
[  168.024750] [ 2429]    48  2429   957965   883970    1851        0             0 httpd
[  168.024751] [ 2430]    48  2430    63367      636     125        0             0 httpd
[  168.024752] [ 2431]    48  2431    63367      632     125        0             0 httpd
[  168.024753] [ 2432]    48  2432    63367      632     125        0             0 httpd
[  168.024754] [ 2433]    48  2433    63367      632     125        0             0 httpd
[  168.024755] [ 2439]     0  2439    39464      205      32        0             0 top
[  168.024756] [ 2440]     0  2440    44659      224      43        0             0 curl
[  168.024758] [ 2442]    48  2442    63367      631     125        0             0 httpd
[  168.024759] Out of memory: Kill process 2429 (httpd) score 914 or sacrifice child
[  168.024836] Killed process 2429 (httpd) total-vm:3831860kB, anon-rss:3535868kB, file-rss:0kB, shmem-rss:12kB
----------------------------------------

After accessing the CGI (if not OOM killed):
----------------------------------------
top - 22:20:55 up 8 min,  2 users,  load average: 6.71, 3.24, 1.42
Tasks: 176 total,   2 running, 174 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  6.7 sy,  0.0 ni, 12.6 id, 79.5 wa,  0.0 hi,  1.2 si,  0.0 st
KiB Mem :  3873864 total,   102876 free,  3702464 used,    68524 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    27316 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2431 apache    20   0 3835660 3.364g     12 S   0.0 91.1   0:03.78 httpd
  684 root      20   0  333912  21976      0 S   0.0  0.6   0:00.98 firewalld
  824 root      20   0  113376  12492      4 S   0.0  0.3   0:00.07 dhclient
 1015 root      20   0  562416  10684      0 S   3.3  0.3   0:05.71 tuned
  642 polkitd   20   0  536984   9056      0 S   0.0  0.2   0:00.09 polkitd
  637 root      20   0  212120   8496   1436 S   0.5  0.2   0:02.43 rsyslogd
 2427 root      20   0  251252   3876   1380 S   0.7  0.1   0:02.33 httpd
    1 root      20   0  128120   2864     96 D   5.2  0.1   0:06.42 systemd
 2432 apache    20   0  253468   2544     12 S   0.1  0.1   0:00.06 httpd
 2433 apache    20   0  253468   2528      4 S   0.0  0.1   0:00.00 httpd
 2442 apache    20   0  253468   2524      4 S   0.0  0.1   0:00.00 httpd
 2447 apache    20   0  253468   2524      4 S   0.0  0.1   0:00.00 httpd
 2457 apache    20   0  253468   2524      4 S   0.0  0.1   0:00.00 httpd
 2428 apache    20   0  253264   2508      4 D   0.5  0.1   0:01.16 httpd
  700 root      20   0  693412   2400      0 S   0.4  0.1   0:01.49 NetworkManager
  463 root      20   0   37224   2188   1828 S   0.0  0.1   0:00.39 systemd-journal
  493 root      20   0   46352   1768      4 S   0.0  0.0   0:00.57 systemd-udevd
  643 root      20   0  219132   1648      0 S   0.0  0.0   0:00.03 abrtd
  630 root      20   0   99608   1600      0 S   0.0  0.0   0:00.05 VGAuthService
  490 root      20   0  121312   1492      0 S   0.0  0.0   0:00.00 lvmetad
  640 root      20   0  231300   1488     68 D   2.9  0.0   0:05.18 vmtoolsd
  644 root      20   0  216848   1404      4 S   0.0  0.0   0:00.05 abrt-watch-log
 1786 root      20   0  145728   1308      0 S   1.0  0.0   0:00.73 sshd
 2384 root      20   0  145728   1292      0 S   0.0  0.0   0:05.50 sshd
 2388 root      20   0  116048   1100      0 S   0.0  0.0   0:00.21 bash
 2342 root      20   0  116048   1048      0 S   0.0  0.0   0:00.08 bash
 2184 root      20   0   91632   1044      0 S   0.1  0.0   0:00.06 master
 2203 postfix   20   0   91736   1016      0 S   0.1  0.0   0:00.03 pickup
 2204 postfix   20   0   91804   1012      0 S   0.0  0.0   0:00.01 qmgr
 1013 root      20   0  106000    972      0 S   0.0  0.0   0:00.02 sshd
 2454 root      20   0  157728    812     36 R   5.4  0.0   0:04.60 top
  670 root      20   0  126236    628      0 S   0.0  0.0   0:00.54 crond
  646 dbus      20   0   34916    612      0 D   1.9  0.0   0:01.10 dbus-daemon
  635 root      20   0  127892    572      0 S   0.0  0.0   0:00.07 smartd
  607 root      16  -4   55456    440      0 S   0.0  0.0   0:00.03 auditd
  636 root      20   0   21628    332      0 D   0.6  0.0   0:01.32 irqbalance
  638 root      20   0   24208    288      0 D   0.5  0.0   0:00.38 systemd-logind
  673 root      20   0   25860    200      0 S   0.0  0.0   0:00.00 atd
 1026 root      20   0  115648    152      0 S   0.0  0.0   0:00.00 rhsmcertd
  632 libstor+  20   0    8536    148      0 S   0.1  0.0   0:00.10 lsmd
  697 root      20   0  110048    128      4 S   0.0  0.0   0:00.02 agetty
 1028 root      20   0  107896    100      0 S   0.0  0.0   0:00.00 rhnsd
  633 root      20   0    4380     92      0 R   0.6  0.0   0:02.87 rngd
----------------------------------------

Note that /proc/2431/smaps contains a huge heap mapping.
----------------------------------------
56134b00c000-561421a69000 rw-p 00000000 00:00 0                          [heap]
Size:            3516788 kB
Rss:             3516784 kB
Pss:             3516784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   3516784 kB
Referenced:      3516784 kB
Anonymous:       3516784 kB
AnonHugePages:      2048 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd
----------------------------------------
Comment 6 Joe Orton 2017-07-18 08:51:45 EDT
Thanks, I can reproduce this.  Please file a support ticket, we will investigate further.
Comment 7 Tetsuo Handa 2017-07-18 09:06:08 EDT
(In reply to Joe Orton from comment #6)
> Thanks, I can reproduce this.  Please file a support ticket, we will
> investigate further.

I already filed a support ticket and status is "Waiting on Red Hat".
I think you will be notified shortly.

Regards.
Comment 8 Joe Orton 2017-09-11 13:35:28 EDT
Thanks for filing the bug upstream!  I'm discussing further upstream to work out how best to fix this.
Comment 10 Joe Orton 2017-09-13 07:10:21 EDT
This is now fixed in upstream trunk.  Since this ended up as a very significant change to a core server function it needs some time in testing before we can expect to integrate it into either upstream httpd 2.4 or RHEL httpd.

https://svn.apache.org/viewvc?view=revision&revision=1808230
Comment 19 errata-xmlrpc 2018-04-10 10:45:22 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0826

Note You need to log in before you can comment on or make changes to this bug.