Bug 1275870 - NRPE initscript does not read PID file when calling status/killproc
NRPE initscript does not read PID file when calling status/killproc
Status: CLOSED ERRATA
Product: Fedora EPEL
Classification: Fedora
Component: nrpe (Show other bugs)
el6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Stephen John Smoogen
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-27 21:26 EDT by Aaron Russo
Modified: 2017-07-07 18:55 EDT (History)
9 users (show)

See Also:
Fixed In Version: nrpe-3.1.1-1.fc24 nrpe-3.1.1-1.fc25 nrpe-3.1.1-1.el6 nrpe-3.1.1-1.el7 nrpe-3.1.1-1.fc26
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-03 20:19:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch nrpe initscript to check pid file for status/killproc calls (705 bytes, patch)
2015-10-27 21:26 EDT, Aaron Russo
no flags Details | Diff

  None (edit)
Description Aaron Russo 2015-10-27 21:26:26 EDT
Created attachment 1087122 [details]
patch nrpe initscript to check pid file for status/killproc calls

Description of problem:

When running nrpe as a standalone service, the default nrpe configuration file places the pid in pid_file=/var/run/nrpe/nrpe.pid

However, the initscript does not check the pid file when calling status/killproc and simply looks for the process 'nrpe' running in the process table.

This causes a race-condition when using the initscript to test the status service, causing it to falsely believe nrpe is started using the initscript when it may not be -- ie. when started by xinetd. 

Version-Release number of selected component (if applicable): 2.15-7.el6


How reproducible:

Consistently

Steps to Reproduce:
1. Configure xinetd to front nrpe and not from initscript
2. Create a long-running nrpe check. I used: 'command[really_long_check]=/bin/sleep 9'
3. Continuously run "check_nrpe -H <hostname> -c really_long_check"
4. Test service status using initscript

Actual results:

'service nrpe status' returns something like:
nrpe (pid 4807 4806) is running...

Expected results:

'service nrpe status' should return:
nrpe is stopped

Additional info:

It looks like this can be solved by adding '-p /var/run/nrpe/nrpe.pid' to calls to the 'status' and 'killproc' functions inside the initscript.

patch attached.
Comment 1 Sven Esbjerg 2015-11-12 04:18:12 EST
This issue also causes the nrpe init script to kill all other nrpe processes on the host. 

At TDC Hosting we have our own nrpe package which is completely separated from other nrpe's which we use for our purposes. This allows us to let customers install nrpe from epel etc.

But now the epel nrpe will kill our nrpe upon stop and restart because of the pid file issue described above.

Pls fix.
Comment 2 Fedora Update System 2017-02-07 15:17:00 EST
nrpe-3.0.1-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-b64fc8eec9
Comment 3 Fedora Update System 2017-02-09 18:16:56 EST
nrpe-3.0.1-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-b64fc8eec9
Comment 4 Fedora Update System 2017-03-03 17:21:19 EST
nrpe-3.0.1-4.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-16f636413b
Comment 5 Fedora Update System 2017-03-03 17:39:11 EST
nrpe-3.0.1-4.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7d833d4735
Comment 6 Fedora Update System 2017-03-04 22:18:24 EST
nrpe-3.0.1-4.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-16f636413b
Comment 7 Fedora Update System 2017-03-04 22:19:58 EST
nrpe-3.0.1-4.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-7d833d4735
Comment 8 Fedora Update System 2017-04-20 19:05:01 EDT
nrpe-3.1.0-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-4b4962389a
Comment 9 Fedora Update System 2017-04-20 19:06:42 EDT
nrpe-3.1.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-6efc362080
Comment 10 Fedora Update System 2017-04-22 17:47:56 EDT
nrpe-3.1.0-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-6efc362080
Comment 11 Fedora Update System 2017-04-27 17:17:39 EDT
nrpe-3.1.0-2.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-d67b84547b
Comment 12 Fedora Update System 2017-04-27 17:20:07 EDT
nrpe-3.1.0-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2c0b443f54
Comment 13 Sven Esbjerg 2017-04-28 07:13:15 EDT
I have tested nrpe-3.1.0-2.el6.x86_64 on RedHat 6.9 with our own nrpe running also. 

The new EPEL provided nrpe no longer stops our nrpe when using /etc/init.d/nrpe stop.
So this is good.


I do notice that I am unable to start the EPEL nrpe with sudo /etc/init.d/nrpe start. It reports OK but silently fails to start. If I run sudo /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d it silently fails to start. If I run sudo strace -f /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d it starts but does not tell me anything is weird or broken.
Seems a bit bad and weird to me.
Comment 14 Stephen John Smoogen 2017-04-28 14:04:42 EDT
@Sven thanks for testing. I am not able to replicate the crash on my 6.9 CentOS box or RHEL6 box. Could you see if there is an selinux issue on your box? If not could you attach the strace so I can try to figure out what might be causing it.

Thanks
Comment 15 Sven Esbjerg 2017-05-04 09:38:54 EDT
SELinux is completely disabled. 

I am unable to give an strace where it fails since it works when I run with strace.
Comment 16 Sven Esbjerg 2017-05-04 10:02:16 EDT
Tested a bit more.

If I run sudo -b strace -f -o /tmp/muh /etc/init.d/nrpe start
nrpe starts

[tdchsve@puprh6v01]~% ps ax|grep nrpe
13184 ?        Ss     0:00 /prod/nagios/sbin/nrpe -c /prod/nagios/etc/nrpe.cfg -d -n
21277 pts/0    S      0:00 sudo -b strace -f -o /tmp/muh /etc/init.d/nrpe start
21278 pts/2    Ss+    0:00 sudo -b strace -f -o /tmp/muh /etc/init.d/nrpe start
21279 pts/2    S      0:00 strace -f -o /tmp/muh /etc/init.d/nrpe start
21292 ?        Ss     0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
21433 pts/0    S+     0:00 grep nrpe

But notice how the sudo strace is still running.

The last bits from the /tmp/muh file are:
21292 fcntl(4, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
21292 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
21292 bind(4, {sa_family=AF_INET, sin_port=htons(5666), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
21292 listen(4, 5)                      = 0
21292 sendto(3, "<30>May  4 15:57:18 nrpe[21292]:"..., 71, MSG_NOSIGNAL, NULL, 0) = 71
21292 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
21292 sendto(3, "<27>May  4 15:57:18 nrpe[21292]:"..., 81, MSG_NOSIGNAL, NULL, 0) = 81
21292 sendto(3, "<30>May  4 15:57:18 nrpe[21292]:"..., 71, MSG_NOSIGNAL, NULL, 0) = 71
21292 sendto(3, "<30>May  4 15:57:18 nrpe[21292]:"..., 70, MSG_NOSIGNAL, NULL, 0) = 70
21292 select(5, [4], NULL, NULL, NULL

Quick guess. The daemonizing of nrpe is not done correctly.
Comment 17 Stephen John Smoogen 2017-05-04 18:40:27 EDT
Thank you for the updates. I ran into the opposite problem with a new system with 3.1.0-2 installed. It starts but won't restart or stop the process.

[You said this earlier] 
The new EPEL provided nrpe no longer stops our nrpe when using /etc/init.d/nrpe stop.
So this is good.

I am confused.. the whole purpose of /etc/init.d/nrpe is to stop the process so that is not good. [That problem is caused because I set the /var/run/nrpe/nrpe.pid to be the start pid but the script goes to /var/run/nrpe.pid]

I also realized I have been dealing with EL7 and above for too long and forgot that rhel-6 uses sh scripts to start the program and should have told you to run via sh -x.

Looking at the output, I think the problem may be that you are running 2 versions of nrpe at the same time:

13184 ?        Ss     0:00 /prod/nagios/sbin/nrpe -c /prod/nagios/etc/nrpe.cfg -d -n

would conflict with the /usr/sbin/nrpe in many cases.
Comment 18 Fedora Update System 2017-05-04 18:48:51 EDT
nrpe-3.1.0-3.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-928f1e8448
Comment 19 Sven Esbjerg 2017-05-05 05:23:40 EDT
It is correct that we have two versions of nrpe on the same host.

Our own version 2.15 which we compile so it does not interfere with other nrpe's.
Thus it has its own pid file called tdchnrpe.pid. It lives in /prod/nagios. It runs on its own port and uses its own user/uid which we have chosen so it does not conflict with anything else.

The issue I have report was that the EPEL provided nrpe killed ALL running nrpe's which is wrong. The EPEL provided nrpe ran ps ax|grep nrpe to find nrpe's or used killall nrpe (cannot remember the specifics) which caused it to kill our nrpe upon stop.

The new package fixes that and only kills the EPEL nrpe. So far so good.
We are happy.


The other thing I reported is that it seems the EPEL nrpe does not work well with sudo which is most likely due to not closing all fd's when forking as a daemon.
I agree that this is a different issue and that the original issues has been fixed.
Comment 20 Fedora Update System 2017-05-05 15:18:09 EDT
nrpe-3.1.0-3.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-928f1e8448
Comment 21 Stephen John Smoogen 2017-06-14 14:46:37 EDT
Thank you for your patience. I believe I have a fix for this to try in all the releases later today.
Comment 22 Fedora Update System 2017-06-14 19:42:35 EDT
nrpe-3.1.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-69a58c7a69
Comment 23 Fedora Update System 2017-06-14 20:02:56 EDT
nrpe-3.1.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-eb24165ee1
Comment 24 Fedora Update System 2017-06-14 20:36:56 EDT
nrpe-3.1.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-be117b53a8
Comment 25 Fedora Update System 2017-06-14 20:47:10 EDT
nrpe-3.1.1-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-92879f40b9
Comment 26 Fedora Update System 2017-06-14 21:02:25 EDT
nrpe-3.1.1-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-f37341bbab
Comment 27 Fedora Update System 2017-06-15 05:48:13 EDT
nrpe-3.1.1-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-92879f40b9
Comment 28 Fedora Update System 2017-06-15 05:48:20 EDT
nrpe-3.1.1-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-f37341bbab
Comment 29 Fedora Update System 2017-06-15 06:57:34 EDT
nrpe-3.1.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-be117b53a8
Comment 30 Fedora Update System 2017-06-15 06:59:29 EDT
nrpe-3.1.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-eb24165ee1
Comment 31 Fedora Update System 2017-06-15 10:01:32 EDT
nrpe-3.1.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-69a58c7a69
Comment 32 Fedora Update System 2017-07-03 20:19:36 EDT
nrpe-3.1.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 33 Fedora Update System 2017-07-03 21:50:56 EDT
nrpe-3.1.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Comment 34 Fedora Update System 2017-07-04 00:17:28 EDT
nrpe-3.1.1-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
Comment 35 Fedora Update System 2017-07-04 00:19:41 EDT
nrpe-3.1.1-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Comment 36 Fedora Update System 2017-07-07 18:55:27 EDT
nrpe-3.1.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.