Bug 1482407

Summary:	nagios-4.3.2-8.el7 crash caused by (potential) result size issue in wproc
Product:	[Fedora] Fedora EPEL	Reporter:	Peter Bieringer <pb>
Component:	nagios	Assignee:	Stephen John Smoogen <smooge>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	epel7	CC:	affix, athmanem, b.heden, jose.p.oliveira.oss, lemenkov, linux, ondrejj, pb, shawn.starr, smooge, smooge, s, swilkerson
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	nagios-4.4.3-1.fc28 nagios-4.4.3-1.fc29 nagios-4.4.3-1.el6 nagios-4.4.3-1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-01-30 01:32:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Peter Bieringer 2017-08-17 08:18:50 UTC

Description of problem:
having a special check script (perl, can't be unfortunately disclosed) will crash nagios after < 40 min and some executions.

Version-Release number of selected component (if applicable):
nagios-4.3.2-8.el7.x86_64


How reproducible:
Always


Steps to Reproduce:
1. configure a special service with a special check script (which acts as a http client)
2. for the configured host, the result from webserver is big (> 500 kByte) and processed in perl (JSON decode/encode + check(


Actual results:
crash after some executions (not on the first one!)

last log lines before the crash:

[1502850410] Warning: The check of service '***' on host '***' looks like it was orphaned (results never came back; last_check=1502433038; next_check=1502849690).  I'm scheduling an immediate check of the service...
[1502850410] wproc: iocache_read() from Core Worker 1337 returned -1: Connection reset by peer
[1502850410] wproc: Socket to worker Core Worker 1337 broken, removing
[1502850410] Caught SIGSEGV, shutting down...
[1502850410] Caught SIGTERM, shutting down...


Expected results:
not crashing


Additional info:

Backtrace from coredump:

(gdb) bt
#0  0x00007fc49f6d223d in fo_reassign_wproc_job ()
#1  0x00007fc49f730f12 in fanout_destroy ()
#2  0x00007fc49f6d2e26 in handle_worker_result ()
#3  0x00007fc49f72fde3 in iobroker_poll ()
#4  0x00007fc49f6eb651 in event_execution_loop ()
#5  0x00007fc49f6cbad6 in main ()

(unfortunately there is no debuginfo provided somehow)

Have found in archives similar issue: http://tracker.nagios.org/view.php?id=504 which disappeared there, but not here.

Particular check disabled, nagios is not crashing anymore.

Comment 1 Stephen John Smoogen 2017-09-15 19:39:39 UTC

I think this bug needs to be opened upstream as the developers there can ask better detailed questions on what your undisclosable script has in it. I have opened it as https://github.com/NagiosEnterprises/nagioscore/issues/432

Comment 2 Bryan Heden 2017-09-19 13:40:17 UTC

Peter,

I have some questions on github about recompiling and re-checking. This may be too much to ask - so I have another idea.

I also have a question in regards to the specific amount of output that the plugin is returning.

My suggestion is to change the plugin so that the output (stdout and stderr combined) never reaches more than 524,287 characters.

If it already is returning less than that (although that is slightly more than 500 k) then please ignore my request :)

Comment 3 Peter Bieringer 2017-10-02 13:53:51 UTC

regarding the wproc issue: yes, plugin had by accident created more than 500k output, but this should not result in crashes.

Plugin enhanced with output limiter, now it behaves better.

Comment 4 Stephen John Smoogen 2017-11-20 23:05:41 UTC

Should I keep this ticket open or close as "fixed by user?"

Comment 5 Peter Bieringer 2017-11-21 06:25:18 UTC

Good question, in general it should be tested and fixed by upstream as it is not "fixed by user" but simply a workaround was applied.

Comment 6 Stephen John Smoogen 2017-11-21 17:05:41 UTC

Ah ok. I wasn't sure if when you fixed it that you did anything in the nagios code. Since you didn't it looks like it will need to addressed upstream. I will keep this open.

Comment 7 Bryan Heden 2017-12-18 19:11:23 UTC

Can you supply me with a perl script that can reproduce the issue? I've tried the following:

#!/usr/bin/perl

print '-' x (800 * 1024);
print "\n";

As I assumed this would duplicate the issue well enough - it does not - and I am unable to reproduce. I am running 4.3.2 compiled.

Comment 8 Fedora Update System 2018-11-30 19:58:27 UTC

nagios-4.4.2-3.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-0346a55d0f

Comment 9 Fedora Update System 2018-11-30 20:52:05 UTC

nagios-4.4.2-3.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-42555731d2

Comment 10 Fedora Update System 2018-11-30 21:03:39 UTC

nagios-4.4.2-3.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-70fe6a4d75

Comment 11 Fedora Update System 2018-11-30 21:38:01 UTC

nagios-4.4.2-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-61fe7c6e70

Comment 12 Fedora Update System 2018-12-01 01:38:34 UTC

nagios-4.4.2-3.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-70fe6a4d75

Comment 13 Fedora Update System 2018-12-01 01:55:00 UTC

nagios-4.4.2-3.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-0346a55d0f

Comment 14 Fedora Update System 2018-12-01 02:03:49 UTC

nagios-4.4.2-3.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-61fe7c6e70

Comment 15 Fedora Update System 2018-12-01 02:43:42 UTC

nagios-4.4.2-3.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-42555731d2

Comment 16 Fedora Update System 2019-01-17 00:14:44 UTC

nagios-4.4.3-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-d661b588d2

Comment 17 Fedora Update System 2019-01-17 00:25:24 UTC

nagios-4.4.3-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-17b388679b

Comment 18 Fedora Update System 2019-01-17 00:43:05 UTC

nagios-4.4.3-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-376ecc221c

Comment 19 Fedora Update System 2019-01-17 00:55:20 UTC

nagios-4.4.3-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0b44528ff1

Comment 20 Fedora Update System 2019-01-18 01:00:27 UTC

nagios-4.4.3-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-d661b588d2

Comment 21 Fedora Update System 2019-01-18 01:31:50 UTC

nagios-4.4.3-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-17b388679b

Comment 22 Fedora Update System 2019-01-18 03:04:58 UTC

nagios-4.4.3-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0b44528ff1

Comment 23 Fedora Update System 2019-01-18 03:36:17 UTC

nagios-4.4.3-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-376ecc221c

Comment 24 Fedora Update System 2019-01-30 01:32:00 UTC

nagios-4.4.3-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2019-01-30 02:06:42 UTC

nagios-4.4.3-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 26 Fedora Update System 2019-02-02 00:36:24 UTC

nagios-4.4.3-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2019-02-02 00:39:25 UTC

nagios-4.4.3-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.