Bug 1504814

... it *looks* like the cure is worse ...

I have not been able to replicate this on my el6 systems. Could you let me know if you have selinux enabled, and if you do if you have the nagios-selinux rpm installed? After that I would edit the file and comment out the /dev/null part and see if it saying something.

We can go from there.

No, no selinux here. There is nagiosgraph and a whole lot of custom monitoring scripts (some of which seems to have b0rked on this update) but nothing obvious to explain this. Without the redirect:
```
[root@hereford init.d]# ./nagios stop
Stopping nagios:. done.
0+0 records in
0+0 records out
0 bytes (0 B) copied, 7.2405e-05 s, 0.0 kB/s
```
and it's hanging:
```
[root@hereford ~]# ps auxw | grep nagios
root      5875  0.0  0.0 106512  1668 pts/2    S+   18:07   0:00 /bin/sh ./nagios stop
root      5887  0.0  0.0 106512   860 pts/2    S+   18:07   0:00 /bin/sh ./nagios stop
root      5929  0.0  0.0 103324   824 pts/0    S+   18:10   0:00 grep nagios
```

I do have two more notes after looking:

1. does this look right:
```
(dd if=$NagiosCommandFile~ count=0 2>/dev/null & echo -n "" >$NagiosCommandFile~)
```
-- I mean the single ampersand before echo.

2. Stopping nagios should probably print "done" after it's removed command file and cleaned up etc.

I tried this:
dd if=/var/spool/nagios/cmd/nagios.cmd~ count=0 of=/dev/null
just to make sure it's not hanging trying to write to the tty, and it's still sitting there. So it looks to me like it's waiting on the pipe -- no idea why

PPS: sending `dd` a USR1:
```
dd: opening `/var/spool/nagios/cmd/nagios.cmd~': Interrupted system call
+ rm -f /var/spool/nagios/cmd/nagios.cmd /var/spool/nagios/cmd/nagios.cmd~
+ rm -f /var/log/nagios/status.dat /var/run/nagios/nagios.pid /var/lock/subsys/nagios
```

Ok the & is correct because it is putting that process in as a background process. The echo "" > $NagiosCommandFile~ should then cause the background task and any other items attached to the pipe to fail and close. 

Could you see if the following script also locks?
#!/bin/bash
rm -f /tmp/test.pipe
mkfifo /tmp/test.pipe
ls -1 /tmp > /tmp/test.pipe &
mv /tmp/test.pipe /tmp/test.pipe~
## The () make this a subshell. The & is intentional as we want to bg this.
( dd if=/tmp/test.pipe~ count=0 2>/dev/null & 
  echo -n "" > /tmp/test.pipe~ )
jobs

rm -f /tmp/test.pipe~
### end of script

OK from the dd that says something is locking the sub dd process from being able to read the pipe file but the only thing I can think of would be selinux or a filesystem error. What kind of filesystem is /var/spool on this box and could you 
do a

ls -lZ /var/spool/nagios/cmd/nagios.cmd*

when it is running so I can see what the file is setup as?

Thanks.

```
# ls -lZ /var/spool/nagios/cmd/nagios.cmd*
prw-rw---- nagios nagios ?                                /var/spool/nagios/cmd/nagios.cmd
```

It's an ext4 on spinning rust with like 4 out 200GB used.

The script hangs about 15% of the time. Can't say I see a rhyme or reason to it.
```
[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
[1]+  Running                 ls -1 /tmp > /tmp/test.pipe &

[root@hereford ~]# ./testpipe.sh
^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call

[root@hereford ~]# ./testpipe.sh
^C./testpipe.sh: line 8: /tmp/test.pipe~: Interrupted system call
```

OK that is weird. I haven't been able to replicate yet, but I fear that nagios is going to hang whether or not the line is in the services because something is holding on the named pipe and isn't allowing an interrupt. 

I am going to get some input from more experienced engineers on what should be done here.

This is an old dual-core pentium D with 4GB RAM, but about the only other thing that's running there is salt-minion. There's maybe a GB of RAM used and an occasional 0.2% iowait or sys load... can't really say the hardware is strained. Other than that it's a stock centos 6.latest x86_64, it's been running nagios happily for years until this update, nagiosgraph is about the only non-stock thing on it that I can think of.

The consensus I have gotten so far is that it working for me is a fluke and I will raise it with upstream to remove this from their code. 

https://github.com/NagiosEnterprises/nagioscore/issues/443

I am commenting this out and will put in an updated build.

LOL

It is a weird one though, I managed to do a couple of /etc/init.d/nagios stops without the hang, too, while playing with it.

I'm observing/affected by the same behavior.

Interesting enough: It worked fine, once I restarted the server. I now added the fix from upstream, as I regularly restart nagios due to config changes, I assume I might see it again quite soon, if it still happens.

OK I got a probable fix for this. The problem is that the echo also needs to be backgrounded so that the () goes completely into the background. I have tried it on my test boxes and it doesn't hang. But I was not able to trigger it very often. A -5 with this will appear in epel-testing soon.

if accessing through ssh, adding 
   -t  
sometimes helps as does, when application, adding a 
   < /dev/null

to keep it from hanging up looking for stdin

ssh would probably be the most common: restart after reconfig, but you could fire it from e.g. salt state or something too...

nagios-4.3.4-5.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-723d26389e

nagios-4.3.4-5.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-723d26389e

nagios-4.3.4-6.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-ecb67df0a6

nagios-4.3.4-7.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-ecb67df0a6

nagios-4.3.4-7.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.