Bug 1757196

Summary: httpd 2.4.41 does not handle CGI PIPES correctly. *Production Server in danger*
Product: [Fedora] Fedora Reporter: customercare
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 29CC: anon.amish, jkaluza, jorton, luhliari, pahan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-02 10:11:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description customercare 2019-09-30 19:36:58 UTC
Description of problem:

this morning 2.4.41 got auto installed. Since then, PHP scripts with bigger outputs, like PDF files are not terminating, because httpd does not read the CGI mode pipe.

PHP processes are stuck in a write().

Because they are stuck and not terminating, they stack with new page visits coming in, to a point where all memory is consumed => OOM/DOS

a httpd restart (with 2.4.41 ) does not change anything, so it's not a shm problem, or a stuck pipe in one of the main processes. 

for smaller pages, 2.4.41 reacts normally, so not all php scripts get stuck.

A strace and ltrace confirm the finding. ltrace does not show any activity and strace shows write ( ...... ) = 543456 and no more activity.

soon as 2.4.39-3 was reinstalled, php started to work normally again.

httpd is configured with:

- MPM_EVENT
- cgiwrapper


        ├─httpd─┬─httpd(apache)─┬─php-cgi(XXXXXXXXXXXXXXX)
        │       │               ├─10*[php-cgi(werbegem)]
        │       │               ├─php-cgi(zorbas)
        │       │               └─php-cgi(XXXXXXXXXX)
        │       └─4*[httpd(apache)───63*[{httpd}]]

via
                AddHandler php-script .php .pl .rb .py .exe .cgi .sh 
                Action php-script /cgi-sys/cgiwrap64

cgiwraper is unchanged since months now and just drops privileges and executes php-cgi ... scriptname .. So pipelining is done between php and httpd 

Example for proplematic php code:

                file_put_contents( "/tmp/.temp.html" , $htmlcode);
                passthru ( "/wkhtmltox/bin/wkhtmltopdf --print-media-type --load-error-handling skip /tmp/.temp.html /tmp/.temp.pdf", $rc);
                ob_end_clean();
                print file_get_contents("/tmp/.temp.pdf"); <<<<<< never returns
                // file_put_contents( "/tmp/.temp.rc" , $rc);
                unlink("/tmp/.temp.html");
                unlink("/tmp/.temp.pdf");





Version-Release number of selected component (if applicable):

2.4.41-1 F29 x86_64


Actual results:

500kb php outputs get stuck in write()

Expected results:

correct handling of the pipe as in 2.4.39-3


SOLUTION for now:

dnf downgrade httpd
dnf update https://kojipkgs.fedoraproject.org//packages/httpd/2.4.39/3.fc29/x86_64/httpd-2.4.39-3.fc29.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/httpd/2.4.39/3.fc29/x86_64/mod_ssl-2.4.39-3.fc29.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/httpd/2.4.39/3.fc29/x86_64/httpd-tools-2.4.39-3.fc29.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/httpd/2.4.39/3.fc29/noarch/httpd-filesystem-2.4.39-3.fc29.noarch.rpm
systemctl restart httpd

Comment 1 Joe Orton 2019-10-02 10:11:49 UTC
Let's track this in 1757683 - probably common root cause.

*** This bug has been marked as a duplicate of bug 1757683 ***