Bug 1412016

Summary: check_http segfaults on a specific (CGI-bin generated) web page
Product: [Fedora] Fedora EPEL Reporter: Jochen Bern <Jochen.Bern>
Component: nagios-pluginsAssignee: Stephen John Smoogen <smooge>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: el6CC: alessandro.crespi, b.heden, mhayden, ondrejj, smooge, smooge, swilkerson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: nagios-plugins-2.2.1-1.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-29 18:17:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Stripped-down CGI-bin that still triggers the SEGV in check_http none

Description Jochen Bern 2017-01-11 01:18:35 UTC
Created attachment 1239282 [details]
Stripped-down CGI-bin that still triggers the SEGV in check_http

Description of problem:

check_http segfaults on a specific (CGI-bin generated) web page.

Version-Release number of selected component (if applicable):

nagios-plugins-http-2.1.4-2.el6.x86_64
(worked with previously installed nagios-plugins-http-2.0.3-3.el6.x86_64)
(page is served by thttpd-2.25b-23.el6.x86_64, also from EPEL, base OS is CentOS 6.8)

How reproducible:

Running check_http from shell to retrieve said page reliably causes SIGSEGV.

Steps to Reproduce:
1. Install nagios-plugins-http and thttpd
2. Configure thttpd to serve a CGI-bin (excerpt of config on our server:
# grep '^[^#]' /etc/thttpd.conf
dir=/var/www/thttpd
pidfile=/var/run/thttpd.pid
port=9480
cgipat=/index.cgi|/minimal.cgi
)
3. Install CGI-bin (minimal version still triggering the problem attached; expanding the HTTP headers so as to resemble those thttpd generates for static pages did not change the results for me)
4. Run check_http as shown in the example results below

Actual results:

# /usr/lib64/nagios/plugins/check_http -I 127.0.0.1 -H localhost -p 9480 -u /minimal.cgi -m 400:500 -w 1 -c 3 -v
GET /minimal.cgi HTTP/1.1
User-Agent: check_http/v2.1.4 (nagios-plugins 2.1.4)
Connection: close
Host: localhost:9480
Accept: */*


http://127.0.0.1:9480/minimal.cgi is 150 characters
STATUS: HTTP/1.0 200 OK
Segmentation fault
# echo $?
139

Expected results:

(Textual page content redacted.)

# /usr/lib64/nagios/plugins/check_http -I 127.0.0.1 -H localhost -p 9480 -u /index.html -m 400:500 -w 1 -c 3 -v
GET /index.html HTTP/1.1
User-Agent: check_http/v2.1.4 (nagios-plugins 2.1.4)
Connection: close
Host: localhost:9480
Accept: */*


http://127.0.0.1:9480/index.html is 554 characters
STATUS: HTTP/1.1 200 OK
**** HEADER ****
Server: thttpd
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 11 Jan 2017 01:09:50 GMT
Last-Modified: Tue, 26 Nov 2013 12:22:24 GMT
Accept-Ranges: bytes
Connection: close
Content-Length: 329

**** CONTENT ****
<HTML><HEAD><TITLE>Redacted redacted redacted</TITLE>
        <META HTTP-EQUIV="refresh" CONTENT="3; URL=/index.cgi"></HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#000077" VLINK="#000033" ALINK="#770033">
<H1>Redacted redacted redacted</H1>
Redacted redacted redacted ...
</BODY></HTML>

HTTP WARNING: HTTP/1.1 200 OK - page size 554 too large - 554 bytes in 0.009 second response time |time=0.008997s;1.000000;3.000000;0.000000 size=554B;400;0;0
# echo $?
1

Additional info:

Comment 1 Stephen John Smoogen 2017-01-18 00:29:39 UTC
I am the new nagios-plugins packager. I am checking with upstream if they are aware of this and if not will file the bug there for you. Thank you for the test case.

Comment 2 Stephen John Smoogen 2017-01-18 00:41:07 UTC
Issue is reported upstream as https://github.com/nagios-plugins/nagios-plugins/issues/226

Comment 3 Alessandro Crespi 2017-01-31 20:31:45 UTC
Isn't this related to this issue? https://github.com/nagios-plugins/nagios-plugins/issues/172

Since updating nagios-plugins, I have this exact crash, and while observing the network traffic, I can see that the server sends back a malformed HTTP response that uses \n instead of \r\n. (partial) hex-dump captures with tcpdump follows:

Request (OK):
00000000  47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a GET / HT TP/1.1..
00000010  55 73 65 72 2d 41 67 65  6e 74 3a 20 63 68 65 63 User-Age nt: chec
00000020  6b 5f 68 74 74 70 2f 76  32 2e 31 2e 34 20 28 6e k_http/v 2.1.4 (n
00000030  61 67 69 6f 73 2d 70 6c  75 67 69 6e 73 20 32 2e agios-pl ugins 2.
00000040  31 2e 34 29 0d 0a 43 6f  6e 6e 65 63 74 69 6f 6e 1.4)..Co nnection
00000050  3a 20 63 6c 6f 73 65 0d  0a 48 6f 73 74 3a 20 62 : close. .Host: b
00000060  69 6f 72 6f 62 73 72 76  34 2e 65 70 66 6c 2e 63 iorobsrv 4.epfl.c
00000070  68 3a 32 35 37 35 0d 0a  41 63 63 65 70 74 3a 20 h:2575.. Accept: 
00000080  2a 2f 2a 0d 0a 0d 0a                             */*....

Response (bad line endings):
00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0a HTTP/1.1  200 OK.
00000010  43 6f 6e 74 65 6e 74 2d  54 79 70 65 3a 20 74 65 Content- Type: te
00000020  78 74 2f 68 74 6d 6c 0a  43 6f 6e 74 65 6e 74 2d xt/html. Content-
00000030  4c 65 6e 67 74 68 3a 20  20 31 30 32 39 0a 0a 3c Length:   1029..<
00000040  68 74 6d 6c 3e 3c 68 65  61 64 65 72 3e 0a 3c 74 html><he ader>.<t
...

Comment 4 Jochen Bern 2017-01-31 21:07:27 UTC
(In reply to Alessandro Crespi from comment #3)
> Isn't this related to this issue?
> https://github.com/nagios-plugins/nagios-plugins/issues/172

It seems to be; taking the (original) CGI-bin and replacing all \n in potential output by \r\n makes check_http produce the expected results again, thanks.

(I wasn't aware that the CGI-bin is supposed to produce "network format" output, rather than the server OS's standard text format ... or is that an oddity with thttpd?)

Comment 5 Alessandro Crespi 2017-01-31 21:18:54 UTC
(In reply to Jochen Bern from comment #4)
> (I wasn't aware that the CGI-bin is supposed to produce "network format"
> output, rather than the server OS's standard text format ... or is that an
> oddity with thttpd?)
It is possibly some oddity of thttpd, the CGI spec (RFC 3875) mandates a "NL" terminator, saying that NL is a "newline" and notes "that newline (NL) need not be a single control character, but can be a sequence of control characters.". So it's potentially system and/or server dependent. As far as I can remember I never outputed any \r\n when writing CGIs, the server (usually Apache) did the job (but of course I had to use \r\n when I was directly serving content to a network socket).

Comment 6 Jochen Bern 2017-01-31 21:32:35 UTC
(In reply to Alessandro Crespi from comment #5)
> It is possibly some oddity of thttpd, [...] As far as
> I can remember I never outputed any \r\n when writing CGIs, the server
> (usually Apache) did the job [...].

I meanwhile checked on our newer platforms, where much the same CGI-bin is served by lighttpd on CentOS 7. tcpdump showed \n instead of \r\n there as well, so I applies the same changes. Might be that just Apache is particularly careful with line separators ...

Comment 7 Alessandro Crespi 2017-01-31 21:43:42 UTC
(In reply to Jochen Bern from comment #6)
> Might be that just Apache is particularly careful with line separators ...
Looks like it is: just tested on our Apache 2.4 server and it definitely translates "\n" to "\r\n" in the headers. It is actually parsing each of them: while testing, I first forgot the empty line between the headers and the "Hello world" text, and I got a 500 error, with logs saying that "Hello world" is not a valid header...

Comment 8 Fedora Update System 2017-02-15 14:32:08 UTC
nagios-plugins-2.1.4-5.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4e77054bfa

Comment 9 Fedora Update System 2017-02-16 14:16:11 UTC
nagios-plugins-2.1.4-5.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4e77054bfa

Comment 10 Fedora Update System 2017-02-16 21:15:52 UTC
nagios-plugins-2.1.4-6.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-b820953367

Comment 11 Fedora Update System 2017-02-17 15:30:58 UTC
nagios-plugins-2.1.4-7.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-994c77a4cd

Comment 12 Fedora Update System 2017-02-18 00:16:51 UTC
nagios-plugins-2.1.4-7.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-994c77a4cd

Comment 13 Fedora Update System 2017-02-25 21:53:14 UTC
nagios-plugins-2.2.0-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-5d813cd00d

Comment 14 Fedora Update System 2017-02-25 22:27:31 UTC
nagios-plugins-2.2.0-4.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-b0accaba31

Comment 15 Fedora Update System 2017-02-26 22:48:42 UTC
nagios-plugins-2.2.0-4.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-b0accaba31

Comment 16 Fedora Update System 2017-03-11 00:48:31 UTC
nagios-plugins-2.2.0-6.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4a502a08b4

Comment 17 Fedora Update System 2017-03-11 12:48:03 UTC
nagios-plugins-2.2.0-6.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4a502a08b4

Comment 18 Fedora Update System 2017-03-13 13:30:25 UTC
nagios-plugins-2.2.0-7.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1623674064

Comment 19 Fedora Update System 2017-03-14 00:16:56 UTC
nagios-plugins-2.2.0-7.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1623674064

Comment 20 Fedora Update System 2017-04-21 00:17:07 UTC
nagios-plugins-2.2.1-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-306cbf64b0

Comment 21 Fedora Update System 2017-06-29 18:17:35 UTC
nagios-plugins-2.2.1-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.