Bug 1464406
| Summary: | Apache consumes too much memory for CGI output. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Tetsuo Handa <penguin-kernel> |
| Component: | httpd | Assignee: | Luboš Uhliarik <luhliari> |
| Status: | CLOSED ERRATA | QA Contact: | Maryna Nalbandian <mnalband> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | bnater, jhouska, jorton, luhliari, mhatanak, penguin-kernel |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | httpd-2.4.6-76.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-10 14:45:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1473612 | ||
Similar result with Fedora 25 httpd-2.4.25-1.fc25.x86_64 kernel-4.11.6-201.fc25.x86_64 and upstream version. Thus, I opened upstream report. https://bz.apache.org/bugzilla/show_bug.cgi?id=61222 Is this an otherwise default CGI configuration? Thanks for checking against Fedora httpd. Please file a support ticket, and we can investigate further. > Is this an otherwise default CGI configuration? Yes, all default configuration. Here is complete proceedure for RHEL 7.4 beta. (1) Install "Basic Web Server" using rhel-server-7.4-beta-1-x86_64-dvd.iso . I created only / partition (XFS, 20GB) on a VM (4 CPUs, 4GB RAM). (2) Install packages needed for compiling C programs. # rpm -ivh gcc-4.8.5-14.el7.x86_64.rpm cpp-4.8.5-14.el7.x86_64.rpm glibc-devel-2.17-194.el7.x86_64.rpm glibc-headers-2.17-194.el7.x86_64.rpm kernel-headers-3.10.0-663.el7.x86_64.rpm mpfr-3.1.1-4.el7.x86_64.rpm libmpc-1.0.1-3.el7.x86_64.rpm (3) Compile the CGI program. # gcc -Wall -O3 -o /var/www/cgi-bin/test.cgi -x c - << "EOF" #include <stdio.h> #include <unistd.h> int main(int argc, char *argv[]) { static char buffer[1048576]; unsigned long long total = 10 * 1024ULL * sizeof(buffer); unsigned int i; printf("Status: 200 OK\r\n"); printf("Content-Type: application/octed-stream\r\n"); printf("Content-Length: %llu\r\n\r\n", total); fflush(stdout); sleep(0); for (i = 0; i < total / sizeof(buffer); i++) write(1, buffer, sizeof(buffer)); return 0; } EOF (4) Start the Apache server. # systemctl start httpd (5) Access the CGI program. # curl http://127.0.0.1/cgi-bin/test.cgi > /dev/null Before accessing the CGI: ---------------------------------------- top - 22:13:40 up 1 min, 2 users, load average: 0.46, 0.20, 0.08 Tasks: 211 total, 1 running, 210 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 3873864 total, 3486932 free, 180872 used, 206060 buff/cache KiB Swap: 0 total, 0 free, 0 used. 3444600 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 684 root 20 0 333912 28676 6712 S 0.0 0.7 0:00.97 firewalld 1015 root 20 0 562416 16588 5904 S 0.0 0.4 0:00.16 tuned 824 root 20 0 113376 15916 3428 S 0.0 0.4 0:00.07 dhclient 642 polkitd 20 0 536984 13940 4936 S 0.0 0.4 0:00.08 polkitd 637 root 20 0 212120 10092 3096 S 0.0 0.3 0:00.16 rsyslogd 700 root 20 0 693412 9464 7072 S 0.0 0.2 0:00.20 NetworkManager 2427 root 20 0 251252 8620 6124 S 0.0 0.2 0:00.17 httpd 1 root 20 0 128120 6780 4012 S 0.0 0.2 0:02.57 systemd 2361 root 20 0 341436 6484 4524 S 0.0 0.2 0:00.01 abrt-dbus 640 root 20 0 231300 6176 4756 S 0.7 0.2 0:00.18 vmtoolsd 630 root 20 0 99608 6084 4484 S 0.0 0.2 0:00.05 VGAuthService 643 root 20 0 219132 5444 3796 S 0.0 0.1 0:00.03 abrtd 2384 root 20 0 145728 5364 4100 S 0.0 0.1 0:00.16 sshd 1786 root 20 0 145728 5360 4100 S 0.0 0.1 0:00.20 sshd 644 root 20 0 216848 4692 3292 S 0.0 0.1 0:00.04 abrt-watch-log 493 root 20 0 46352 4528 2764 S 0.0 0.1 0:00.57 systemd-udevd 1013 root 20 0 106000 4056 3084 S 0.0 0.1 0:00.02 sshd 490 root 20 0 121312 4044 2552 S 0.0 0.1 0:00.00 lvmetad 2204 postfix 20 0 91804 4036 3024 S 0.0 0.1 0:00.00 qmgr 2203 postfix 20 0 91736 4016 3004 S 0.0 0.1 0:00.00 pickup 2429 apache 20 0 253468 3828 1304 S 0.0 0.1 0:00.00 httpd 2430 apache 20 0 253468 3828 1304 S 0.0 0.1 0:00.00 httpd 2431 apache 20 0 253468 3828 1304 S 0.0 0.1 0:00.00 httpd 2432 apache 20 0 253468 3828 1304 S 0.0 0.1 0:00.00 httpd 2433 apache 20 0 253468 3828 1304 S 0.0 0.1 0:00.00 httpd 2428 apache 20 0 253264 3540 1040 S 0.0 0.1 0:00.00 httpd 463 root 20 0 36832 3040 2720 S 0.0 0.1 0:00.22 systemd-journal 2342 root 20 0 116048 2680 1668 S 0.0 0.1 0:00.06 bash 2388 root 20 0 116048 2660 1644 S 0.0 0.1 0:00.02 bash 2439 root 20 0 157856 2348 1536 R 0.3 0.1 0:00.06 top 635 root 20 0 127892 2292 1720 S 0.0 0.1 0:00.07 smartd 2184 root 20 0 91632 2132 1112 S 0.0 0.1 0:00.00 master 646 dbus 20 0 34916 2060 1448 S 0.0 0.1 0:00.17 dbus-daemon 638 root 20 0 24208 1660 1372 S 0.0 0.0 0:00.03 systemd-logind 670 root 20 0 126236 1632 1012 S 0.0 0.0 0:00.44 crond 636 root 20 0 21628 1300 968 S 0.0 0.0 0:00.03 irqbalance 673 root 20 0 25860 912 712 S 0.0 0.0 0:00.00 atd 607 root 16 -4 55456 888 484 S 0.0 0.0 0:00.01 auditd 697 root 20 0 110048 820 696 S 0.0 0.0 0:00.02 agetty 632 libstor+ 20 0 8536 796 648 S 0.0 0.0 0:00.05 lsmd 1026 root 20 0 115648 640 488 S 0.0 0.0 0:00.00 rhsmcertd 633 root 20 0 4380 592 500 S 0.0 0.0 0:00.72 rngd 1028 root 20 0 107896 284 184 S 0.0 0.0 0:00.00 rhnsd ---------------------------------------- After accessing the CGI (if OOM killed): ---------------------------------------- [ 168.014205] irqbalance invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 168.014210] irqbalance cpuset=/ mems_allowed=0 [ 168.014212] CPU: 0 PID: 636 Comm: irqbalance Not tainted 3.10.0-663.el7.x86_64 #1 [ 168.014214] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013 [ 168.014215] ffff9526743d9f60 0000000073aad44d ffff9525f6933840 ffffffffaf09fc54 [ 168.014217] ffff9525f69338d0 ffffffffaf09b0bf ffff9525f69338d8 ffffffffaecb66ab [ 168.014219] ffff952677dbaae8 ffff9525f6933920 ffffffff00000202 fffeefff00000000 [ 168.014221] Call Trace: [ 168.014231] [<ffffffffaf09fc54>] dump_stack+0x19/0x1b [ 168.014233] [<ffffffffaf09b0bf>] dump_header+0x90/0x229 [ 168.014237] [<ffffffffaecb66ab>] ? cred_has_capability+0x6b/0x120 [ 168.014242] [<ffffffffaeb860c4>] oom_kill_process+0x254/0x3d0 [ 168.014243] [<ffffffffaecb687c>] ? selinux_capable+0x1c/0x40 [ 168.014245] [<ffffffffaeb86906>] out_of_memory+0x4b6/0x4f0 [ 168.014247] [<ffffffffaf09bbc3>] __alloc_pages_slowpath+0x5d6/0x724 [ 168.014250] [<ffffffffaeb8cab5>] __alloc_pages_nodemask+0x405/0x420 [ 168.014252] [<ffffffffaebd0a08>] alloc_pages_current+0x98/0x110 [ 168.014254] [<ffffffffaeb8733e>] __get_free_pages+0xe/0x40 [ 168.014256] [<ffffffffaecb6ea4>] inode_doinit_with_dentry+0x2f4/0x570 [ 168.014257] [<ffffffffaecb793c>] selinux_d_instantiate+0x1c/0x20 [ 168.014260] [<ffffffffaecb02fb>] security_d_instantiate+0x1b/0x30 [ 168.014263] [<ffffffffaec17d72>] d_instantiate+0x52/0x70 [ 168.014265] [<ffffffffaec756aa>] proc_lookup_de+0xaa/0xc0 [ 168.014267] [<ffffffffaec756db>] proc_lookup+0x1b/0x20 [ 168.014269] [<ffffffffaec6fbcc>] proc_root_lookup+0x1c/0x40 [ 168.014272] [<ffffffffaec0a51d>] lookup_real+0x1d/0x50 [ 168.014273] [<ffffffffaec0daf3>] do_last+0xb93/0x1280 [ 168.014441] [<ffffffffc033f51c>] ? xfs_iunlock+0x11c/0x130 [xfs] [ 168.014444] [<ffffffffaebdf1a6>] ? kmem_cache_alloc_trace+0x1d6/0x200 [ 168.014446] [<ffffffffaec10792>] path_openat+0xc2/0x490 [ 168.014449] [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90 [ 168.014450] [<ffffffffaf0a8308>] ? page_fault+0x28/0x30 [ 168.014452] [<ffffffffaec11f5b>] do_filp_open+0x4b/0xb0 [ 168.014454] [<ffffffffaec1f00a>] ? __alloc_fd+0x8a/0x130 [ 168.014458] [<ffffffffaebfee93>] do_sys_open+0xf3/0x1f0 [ 168.014459] [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90 [ 168.014461] [<ffffffffaebfefae>] SyS_open+0x1e/0x20 [ 168.014463] [<ffffffffaf0b0dc9>] system_call_fastpath+0x16/0x1b [ 168.014464] Mem-Info: [ 168.014468] active_anon:907589 inactive_anon:2437 isolated_anon:0 active_file:142 inactive_file:474 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:4205 slab_unreclaimable:10052 mapped:926 shmem:2523 pagetables:4171 bounce:0 free:20546 free_pcp:10 free_cma:0 [ 168.014470] Node 0 DMA free:15332kB min:276kB low:344kB high:412kB active_anon:160kB inactive_anon:0kB active_file:4kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1625 all_unreclaimable? yes [ 168.014474] lowmem_reserve[]: 0 2814 3765 3765 [ 168.014476] Node 0 DMA32 free:53972kB min:50308kB low:62884kB high:75460kB active_anon:2751240kB inactive_anon:6236kB active_file:564kB inactive_file:1872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2884272kB mlocked:0kB dirty:0kB writeback:0kB mapped:680kB shmem:6480kB slab_reclaimable:8468kB slab_unreclaimable:18008kB kernel_stack:2208kB pagetables:12804kB unstable:0kB bounce:0kB free_pcp:40kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3748 all_unreclaimable? yes [ 168.014479] lowmem_reserve[]: 0 0 950 950 [ 168.014481] Node 0 Normal free:12880kB min:16992kB low:21240kB high:25488kB active_anon:878956kB inactive_anon:3512kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1048576kB managed:973688kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:3612kB slab_reclaimable:8272kB slab_unreclaimable:22112kB kernel_stack:2512kB pagetables:3876kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 168.014483] lowmem_reserve[]: 0 0 0 0 [ 168.014485] Node 0 DMA: 4*4kB (M) 6*8kB (UEM) 4*16kB (UM) 2*32kB (UE) 1*64kB (U) 2*128kB (UE) 2*256kB (EM) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15360kB [ 168.014493] Node 0 DMA32: 716*4kB (UEM) 348*8kB (UEM) 110*16kB (UEM) 50*32kB (UEM) 31*64kB (EM) 11*128kB (UEM) 7*256kB (UEM) 2*512kB (UE) 2*1024kB (UE) 2*2048kB (EM) 8*4096kB (UM) = 54128kB [ 168.014499] Node 0 Normal: 470*4kB (UM) 209*8kB (UEM) 79*16kB (UEM) 26*32kB (UEM) 7*64kB (UEM) 1*128kB (E) 2*256kB (E) 0*512kB 4*1024kB (U) 1*2048kB (M) 0*4096kB = 12880kB [ 168.014507] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 168.014517] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 168.014518] 3286 total pagecache pages [ 168.014519] 0 pages in swap cache [ 168.014520] Swap cache stats: add 0, delete 0, find 0/0 [ 168.014521] Free swap = 0kB [ 168.014522] Total swap = 0kB [ 168.014522] 1048445 pages RAM [ 168.014523] 0 pages HighMem/MovableOnly [ 168.014523] 79979 pages reserved [ 168.014524] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 168.014534] [ 463] 0 463 9208 469 21 0 0 systemd-journal [ 168.014535] [ 490] 0 490 30328 373 29 0 0 lvmetad [ 168.014537] [ 493] 0 493 11588 442 23 0 -1000 systemd-udevd [ 168.014539] [ 607] 0 607 13864 101 27 0 -1000 auditd [ 168.014540] [ 630] 0 630 24902 400 43 0 0 VGAuthService [ 168.014541] [ 632] 998 632 2134 37 10 0 0 lsmd [ 168.014543] [ 633] 0 633 1095 23 8 0 0 rngd [ 168.014544] [ 635] 0 635 31973 143 20 0 0 smartd [ 168.014545] [ 636] 0 636 5407 83 15 0 0 irqbalance [ 168.014546] [ 637] 0 637 53030 2065 42 0 0 rsyslogd [ 168.014547] [ 638] 0 638 6052 72 16 0 0 systemd-logind [ 168.014549] [ 640] 0 640 57825 382 60 0 0 vmtoolsd [ 168.014550] [ 642] 999 642 134246 2251 57 0 0 polkitd [ 168.014551] [ 643] 0 643 54783 412 59 0 0 abrtd [ 168.014552] [ 644] 0 644 54212 351 58 0 0 abrt-watch-log [ 168.014554] [ 646] 81 646 8729 153 19 0 -900 dbus-daemon [ 168.014579] [ 670] 0 670 31559 155 19 0 0 crond [ 168.014580] [ 673] 0 673 6465 50 18 0 0 atd [ 168.014582] [ 684] 0 684 83478 5491 85 0 0 firewalld [ 168.014583] [ 697] 0 697 27512 32 10 0 0 agetty [ 168.014584] [ 700] 0 700 173353 600 88 0 0 NetworkManager [ 168.014586] [ 824] 0 824 28344 3123 57 0 0 dhclient [ 168.014587] [ 1013] 0 1013 26500 243 55 0 -1000 sshd [ 168.014589] [ 1015] 0 1015 140604 2671 93 0 0 tuned [ 168.014590] [ 1026] 0 1026 28912 38 14 0 0 rhsmcertd [ 168.014591] [ 1028] 0 1028 26974 25 10 0 0 rhnsd [ 168.014592] [ 1786] 0 1786 36432 324 74 0 0 sshd [ 168.014593] [ 2184] 0 2184 22908 261 45 0 0 master [ 168.014595] [ 2203] 89 2203 22934 254 45 0 0 pickup [ 168.014596] [ 2204] 89 2204 22951 253 45 0 0 qmgr [ 168.014597] [ 2342] 0 2342 29012 254 16 0 0 bash [ 168.014598] [ 2361] 0 2361 85359 490 64 0 -900 abrt-dbus [ 168.014599] [ 2384] 0 2384 36432 321 74 0 0 sshd [ 168.014600] [ 2388] 0 2388 29012 254 15 0 0 bash [ 168.014602] [ 2427] 0 2427 62813 969 126 0 0 httpd [ 168.014603] [ 2428] 0 2428 63316 633 125 0 0 httpd [ 168.014604] [ 2429] 48 2429 957965 883970 1851 0 0 httpd [ 168.014605] [ 2430] 48 2430 63367 636 125 0 0 httpd [ 168.014606] [ 2431] 48 2431 63367 632 125 0 0 httpd [ 168.014607] [ 2432] 48 2432 63367 632 125 0 0 httpd [ 168.014608] [ 2433] 48 2433 63367 632 125 0 0 httpd [ 168.014609] [ 2439] 0 2439 39464 205 32 0 0 top [ 168.014611] [ 2440] 0 2440 44659 224 43 0 0 curl [ 168.014612] [ 2441] 48 2441 1299 21 7 0 0 test.cgi [ 168.014613] [ 2442] 48 2442 63367 631 125 0 0 httpd [ 168.014614] Out of memory: Kill process 2429 (httpd) score 914 or sacrifice child [ 168.014682] Killed process 2441 (test.cgi) total-vm:5196kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB [ 168.024533] irqbalance invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 168.024537] irqbalance cpuset=/ mems_allowed=0 [ 168.024540] CPU: 2 PID: 636 Comm: irqbalance Not tainted 3.10.0-663.el7.x86_64 #1 [ 168.024541] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013 [ 168.024542] ffff9526743d9f60 0000000073aad44d ffff9525f6933840 ffffffffaf09fc54 [ 168.024544] ffff9525f69338d0 ffffffffaf09b0bf ffff9525f69338d8 ffffffffaecb66ab [ 168.024546] ffff952677dbaae8 0000000000000206 ffffffff00000206 fffeefff00000000 [ 168.024547] Call Trace: [ 168.024555] [<ffffffffaf09fc54>] dump_stack+0x19/0x1b [ 168.024557] [<ffffffffaf09b0bf>] dump_header+0x90/0x229 [ 168.024561] [<ffffffffaecb66ab>] ? cred_has_capability+0x6b/0x120 [ 168.024565] [<ffffffffaeb860c4>] oom_kill_process+0x254/0x3d0 [ 168.024567] [<ffffffffaecb687c>] ? selinux_capable+0x1c/0x40 [ 168.024569] [<ffffffffaeb86906>] out_of_memory+0x4b6/0x4f0 [ 168.024571] [<ffffffffaf09bbc3>] __alloc_pages_slowpath+0x5d6/0x724 [ 168.024573] [<ffffffffaeb8cab5>] __alloc_pages_nodemask+0x405/0x420 [ 168.024575] [<ffffffffaebd0a08>] alloc_pages_current+0x98/0x110 [ 168.024577] [<ffffffffaeb8733e>] __get_free_pages+0xe/0x40 [ 168.024579] [<ffffffffaecb6ea4>] inode_doinit_with_dentry+0x2f4/0x570 [ 168.024581] [<ffffffffaecb793c>] selinux_d_instantiate+0x1c/0x20 [ 168.024584] [<ffffffffaecb02fb>] security_d_instantiate+0x1b/0x30 [ 168.024587] [<ffffffffaec17d72>] d_instantiate+0x52/0x70 [ 168.024589] [<ffffffffaec756aa>] proc_lookup_de+0xaa/0xc0 [ 168.024590] [<ffffffffaec756db>] proc_lookup+0x1b/0x20 [ 168.024593] [<ffffffffaec6fbcc>] proc_root_lookup+0x1c/0x40 [ 168.024595] [<ffffffffaec0a51d>] lookup_real+0x1d/0x50 [ 168.024596] [<ffffffffaec0daf3>] do_last+0xb93/0x1280 [ 168.024630] [<ffffffffc033f51c>] ? xfs_iunlock+0x11c/0x130 [xfs] [ 168.024632] [<ffffffffaebdf1a6>] ? kmem_cache_alloc_trace+0x1d6/0x200 [ 168.024634] [<ffffffffaec10792>] path_openat+0xc2/0x490 [ 168.024636] [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90 [ 168.024638] [<ffffffffaf0a8308>] ? page_fault+0x28/0x30 [ 168.024639] [<ffffffffaec11f5b>] do_filp_open+0x4b/0xb0 [ 168.024641] [<ffffffffaec1f00a>] ? __alloc_fd+0x8a/0x130 [ 168.024643] [<ffffffffaebfee93>] do_sys_open+0xf3/0x1f0 [ 168.024645] [<ffffffffaf0ac0e5>] ? do_page_fault+0x35/0x90 [ 168.024647] [<ffffffffaebfefae>] SyS_open+0x1e/0x20 [ 168.024649] [<ffffffffaf0b0dc9>] system_call_fastpath+0x16/0x1b [ 168.024650] Mem-Info: [ 168.024653] active_anon:907589 inactive_anon:2437 isolated_anon:0 active_file:142 inactive_file:437 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:4205 slab_unreclaimable:10052 mapped:926 shmem:2523 pagetables:4171 bounce:0 free:20515 free_pcp:94 free_cma:0 [ 168.024655] Node 0 DMA free:15332kB min:276kB low:344kB high:412kB active_anon:160kB inactive_anon:0kB active_file:4kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:88kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1651 all_unreclaimable? yes [ 168.024659] lowmem_reserve[]: 0 2814 3765 3765 [ 168.024661] Node 0 DMA32 free:53972kB min:50308kB low:62884kB high:75460kB active_anon:2751240kB inactive_anon:6236kB active_file:564kB inactive_file:1724kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2884272kB mlocked:0kB dirty:0kB writeback:0kB mapped:680kB shmem:6480kB slab_reclaimable:8468kB slab_unreclaimable:18008kB kernel_stack:2208kB pagetables:12804kB unstable:0kB bounce:0kB free_pcp:252kB local_pcp:116kB free_cma:0kB writeback_tmp:0kB pages_scanned:3812 all_unreclaimable? yes [ 168.024664] lowmem_reserve[]: 0 0 950 950 [ 168.024666] Node 0 Normal free:12756kB min:16992kB low:21240kB high:25488kB active_anon:878956kB inactive_anon:3512kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1048576kB managed:973688kB mlocked:0kB dirty:0kB writeback:0kB mapped:3044kB shmem:3612kB slab_reclaimable:8272kB slab_unreclaimable:22112kB kernel_stack:2512kB pagetables:3876kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 168.024668] lowmem_reserve[]: 0 0 0 0 [ 168.024670] Node 0 DMA: 4*4kB (M) 6*8kB (UEM) 4*16kB (UM) 2*32kB (UE) 1*64kB (U) 2*128kB (UE) 2*256kB (EM) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15360kB [ 168.024677] Node 0 DMA32: 716*4kB (UEM) 348*8kB (UEM) 110*16kB (UEM) 50*32kB (UEM) 31*64kB (EM) 11*128kB (UEM) 7*256kB (UEM) 2*512kB (UE) 2*1024kB (UE) 2*2048kB (EM) 8*4096kB (UM) = 54128kB [ 168.024684] Node 0 Normal: 439*4kB (UM) 209*8kB (UEM) 79*16kB (UEM) 26*32kB (UEM) 7*64kB (UEM) 1*128kB (E) 2*256kB (E) 0*512kB 4*1024kB (U) 1*2048kB (M) 0*4096kB = 12756kB [ 168.024691] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 168.024693] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 168.024694] 3286 total pagecache pages [ 168.024695] 0 pages in swap cache [ 168.024696] Swap cache stats: add 0, delete 0, find 0/0 [ 168.024696] Free swap = 0kB [ 168.024697] Total swap = 0kB [ 168.024698] 1048445 pages RAM [ 168.024698] 0 pages HighMem/MovableOnly [ 168.024699] 79979 pages reserved [ 168.024699] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 168.024707] [ 463] 0 463 9208 469 21 0 0 systemd-journal [ 168.024708] [ 490] 0 490 30328 373 29 0 0 lvmetad [ 168.024710] [ 493] 0 493 11588 442 23 0 -1000 systemd-udevd [ 168.024711] [ 607] 0 607 13864 101 27 0 -1000 auditd [ 168.024712] [ 630] 0 630 24902 400 43 0 0 VGAuthService [ 168.024714] [ 632] 998 632 2134 37 10 0 0 lsmd [ 168.024715] [ 633] 0 633 1095 23 8 0 0 rngd [ 168.024716] [ 635] 0 635 31973 143 20 0 0 smartd [ 168.024717] [ 636] 0 636 5407 83 15 0 0 irqbalance [ 168.024718] [ 637] 0 637 53030 2065 42 0 0 rsyslogd [ 168.024720] [ 638] 0 638 6052 72 16 0 0 systemd-logind [ 168.024721] [ 640] 0 640 57825 382 60 0 0 vmtoolsd [ 168.024722] [ 642] 999 642 134246 2251 57 0 0 polkitd [ 168.024723] [ 643] 0 643 54783 412 59 0 0 abrtd [ 168.024724] [ 644] 0 644 54212 351 58 0 0 abrt-watch-log [ 168.024726] [ 646] 81 646 8729 153 19 0 -900 dbus-daemon [ 168.024727] [ 670] 0 670 31559 155 19 0 0 crond [ 168.024728] [ 673] 0 673 6465 50 18 0 0 atd [ 168.024729] [ 684] 0 684 83478 5491 85 0 0 firewalld [ 168.024730] [ 697] 0 697 27512 32 10 0 0 agetty [ 168.024731] [ 700] 0 700 173353 600 88 0 0 NetworkManager [ 168.024733] [ 824] 0 824 28344 3123 57 0 0 dhclient [ 168.024734] [ 1013] 0 1013 26500 243 55 0 -1000 sshd [ 168.024735] [ 1015] 0 1015 140604 2671 93 0 0 tuned [ 168.024736] [ 1026] 0 1026 28912 38 14 0 0 rhsmcertd [ 168.024737] [ 1028] 0 1028 26974 25 10 0 0 rhnsd [ 168.024739] [ 1786] 0 1786 36432 324 74 0 0 sshd [ 168.024740] [ 2184] 0 2184 22908 261 45 0 0 master [ 168.024741] [ 2203] 89 2203 22934 254 45 0 0 pickup [ 168.024742] [ 2204] 89 2204 22951 253 45 0 0 qmgr [ 168.024743] [ 2342] 0 2342 29012 254 16 0 0 bash [ 168.024744] [ 2361] 0 2361 85359 490 64 0 -900 abrt-dbus [ 168.024745] [ 2384] 0 2384 36432 321 74 0 0 sshd [ 168.024746] [ 2388] 0 2388 29012 254 15 0 0 bash [ 168.024748] [ 2427] 0 2427 62813 969 126 0 0 httpd [ 168.024749] [ 2428] 0 2428 63316 633 125 0 0 httpd [ 168.024750] [ 2429] 48 2429 957965 883970 1851 0 0 httpd [ 168.024751] [ 2430] 48 2430 63367 636 125 0 0 httpd [ 168.024752] [ 2431] 48 2431 63367 632 125 0 0 httpd [ 168.024753] [ 2432] 48 2432 63367 632 125 0 0 httpd [ 168.024754] [ 2433] 48 2433 63367 632 125 0 0 httpd [ 168.024755] [ 2439] 0 2439 39464 205 32 0 0 top [ 168.024756] [ 2440] 0 2440 44659 224 43 0 0 curl [ 168.024758] [ 2442] 48 2442 63367 631 125 0 0 httpd [ 168.024759] Out of memory: Kill process 2429 (httpd) score 914 or sacrifice child [ 168.024836] Killed process 2429 (httpd) total-vm:3831860kB, anon-rss:3535868kB, file-rss:0kB, shmem-rss:12kB ---------------------------------------- After accessing the CGI (if not OOM killed): ---------------------------------------- top - 22:20:55 up 8 min, 2 users, load average: 6.71, 3.24, 1.42 Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 6.7 sy, 0.0 ni, 12.6 id, 79.5 wa, 0.0 hi, 1.2 si, 0.0 st KiB Mem : 3873864 total, 102876 free, 3702464 used, 68524 buff/cache KiB Swap: 0 total, 0 free, 0 used. 27316 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2431 apache 20 0 3835660 3.364g 12 S 0.0 91.1 0:03.78 httpd 684 root 20 0 333912 21976 0 S 0.0 0.6 0:00.98 firewalld 824 root 20 0 113376 12492 4 S 0.0 0.3 0:00.07 dhclient 1015 root 20 0 562416 10684 0 S 3.3 0.3 0:05.71 tuned 642 polkitd 20 0 536984 9056 0 S 0.0 0.2 0:00.09 polkitd 637 root 20 0 212120 8496 1436 S 0.5 0.2 0:02.43 rsyslogd 2427 root 20 0 251252 3876 1380 S 0.7 0.1 0:02.33 httpd 1 root 20 0 128120 2864 96 D 5.2 0.1 0:06.42 systemd 2432 apache 20 0 253468 2544 12 S 0.1 0.1 0:00.06 httpd 2433 apache 20 0 253468 2528 4 S 0.0 0.1 0:00.00 httpd 2442 apache 20 0 253468 2524 4 S 0.0 0.1 0:00.00 httpd 2447 apache 20 0 253468 2524 4 S 0.0 0.1 0:00.00 httpd 2457 apache 20 0 253468 2524 4 S 0.0 0.1 0:00.00 httpd 2428 apache 20 0 253264 2508 4 D 0.5 0.1 0:01.16 httpd 700 root 20 0 693412 2400 0 S 0.4 0.1 0:01.49 NetworkManager 463 root 20 0 37224 2188 1828 S 0.0 0.1 0:00.39 systemd-journal 493 root 20 0 46352 1768 4 S 0.0 0.0 0:00.57 systemd-udevd 643 root 20 0 219132 1648 0 S 0.0 0.0 0:00.03 abrtd 630 root 20 0 99608 1600 0 S 0.0 0.0 0:00.05 VGAuthService 490 root 20 0 121312 1492 0 S 0.0 0.0 0:00.00 lvmetad 640 root 20 0 231300 1488 68 D 2.9 0.0 0:05.18 vmtoolsd 644 root 20 0 216848 1404 4 S 0.0 0.0 0:00.05 abrt-watch-log 1786 root 20 0 145728 1308 0 S 1.0 0.0 0:00.73 sshd 2384 root 20 0 145728 1292 0 S 0.0 0.0 0:05.50 sshd 2388 root 20 0 116048 1100 0 S 0.0 0.0 0:00.21 bash 2342 root 20 0 116048 1048 0 S 0.0 0.0 0:00.08 bash 2184 root 20 0 91632 1044 0 S 0.1 0.0 0:00.06 master 2203 postfix 20 0 91736 1016 0 S 0.1 0.0 0:00.03 pickup 2204 postfix 20 0 91804 1012 0 S 0.0 0.0 0:00.01 qmgr 1013 root 20 0 106000 972 0 S 0.0 0.0 0:00.02 sshd 2454 root 20 0 157728 812 36 R 5.4 0.0 0:04.60 top 670 root 20 0 126236 628 0 S 0.0 0.0 0:00.54 crond 646 dbus 20 0 34916 612 0 D 1.9 0.0 0:01.10 dbus-daemon 635 root 20 0 127892 572 0 S 0.0 0.0 0:00.07 smartd 607 root 16 -4 55456 440 0 S 0.0 0.0 0:00.03 auditd 636 root 20 0 21628 332 0 D 0.6 0.0 0:01.32 irqbalance 638 root 20 0 24208 288 0 D 0.5 0.0 0:00.38 systemd-logind 673 root 20 0 25860 200 0 S 0.0 0.0 0:00.00 atd 1026 root 20 0 115648 152 0 S 0.0 0.0 0:00.00 rhsmcertd 632 libstor+ 20 0 8536 148 0 S 0.1 0.0 0:00.10 lsmd 697 root 20 0 110048 128 4 S 0.0 0.0 0:00.02 agetty 1028 root 20 0 107896 100 0 S 0.0 0.0 0:00.00 rhnsd 633 root 20 0 4380 92 0 R 0.6 0.0 0:02.87 rngd ---------------------------------------- Note that /proc/2431/smaps contains a huge heap mapping. ---------------------------------------- 56134b00c000-561421a69000 rw-p 00000000 00:00 0 [heap] Size: 3516788 kB Rss: 3516784 kB Pss: 3516784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 3516784 kB Referenced: 3516784 kB Anonymous: 3516784 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd ---------------------------------------- Thanks, I can reproduce this. Please file a support ticket, we will investigate further. (In reply to Joe Orton from comment #6) > Thanks, I can reproduce this. Please file a support ticket, we will > investigate further. I already filed a support ticket and status is "Waiting on Red Hat". I think you will be notified shortly. Regards. Thanks for filing the bug upstream! I'm discussing further upstream to work out how best to fix this. This is now fixed in upstream trunk. Since this ended up as a very significant change to a core server function it needs some time in testing before we can expect to integrate it into either upstream httpd 2.4 or RHEL httpd. https://svn.apache.org/viewvc?view=revision&revision=1808230 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0826 |
Description of problem: I am trying to develop an Apache CGI program written in C which handles upload/download of large files (magnitude of many gigabytes), and noticed that Apache consumes too much memory trying to spool the output from the CGI program if the CGI program started sending the output immediately. Version-Release number of selected component (if applicable): httpd-2.4.6-45.el7_3.4.x86_64 kernel-3.10.0-514.21.2.el7.x86_64 How reproducible: This problem seems to be timing dependent, but not difficult to trigger. I'm hitting this problem on a VM with 4 CPUs / 4GB RAM / no swap. Steps to Reproduce: (1) Compile below program as /var/www/cgi-bin/test.cgi , where the value of "total" should be something large that will fit or will not fit memory available. ---------- #include <stdio.h> #include <unistd.h> int main(int argc, char *argv[]) { static char buffer[1048576]; unsigned long long total = 10 * 1024ULL * sizeof(buffer); unsigned int i; printf("Status: 200 OK\r\n"); printf("Content-Type: application/octed-stream\r\n"); printf("Content-Length: %llu\r\n\r\n", total); fflush(stdout); sleep(0); for (i = 0; i < total / sizeof(buffer); i++) write(1, buffer, sizeof(buffer)); return 0; } ---------- (2) Start Apache service. # systemctl start httpd (3) Start /usr/bin/top in order to monitor memory usage by /usr/sbin/httpd processes. # /usr/bin/top -o %MEM (4) Open another shell, and start below command line. $ /usr/bin/curl http://127.0.0.1/cgi-bin/test.cgi > /dev/null (5) Check the output of /usr/bin/top at (3). Actual results: Case 1: The Apache worker process is killed by the OOM killer after the CGI program is killed (as a child of the biggest memory consumer) by the OOM killer. In this case, only current request from current client will be affected due to premature output. But the system might hang up completely due to lack of the OOM reaper support in the kernel. Case 2: The Apache worker process triggers HTTP 500 Internal Server Error due to unable to create child process for executing the CGI program if the Apache worker process is not killed by the OOM killer, for heap memory which holds the output from previous request is not freed before processing next request. In this case, future requests from future clients will be affected due to service unavailable. But sum of memory consumed by all Apache worker processes might after all lead to Case 1. Case 3: The Apache worker process survives. The amount of consumed memory changes if we do some sleep (e.g. change sleep(0) to sleep(1) in the source code shown above). Expected results: The Apache worker process should not consume too much memory trying to spool the output from the CGI program. Even if spooling is necessary for some reason, it should not consume too much memory and it should free the memory after current request completed.