Bug 675287

Summary: Clustered qpidd leaks semaphores when started other than via the service script.
Product: Red Hat Enterprise MRG Reporter: Alan Conway <aconway>
Component: qpid-cppAssignee: messaging-bugs <messaging-bugs>
Status: CLOSED NOTABUG QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 1.3CC: gsim, jross
Target Milestone: ---Flags: jross: needinfo? (mcressma)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: qpidd is started from the command line or from a script, and not via the "service qpidd start" command. SELinux is in Enforcing mode. Note this does not apply when qpidd is started as a service via "service qpidd start". Consequence: Over many restarts, qpidd consumes semaphores as shown by the "ipcs -s" command. Eventually when the system limit is reached, qpidd will hang during startup with the last message in the log being: notice Initializing CPG The selinux audit log will also show messages like this: type=AVC msg=audit(1296806605.532:30390): avc: denied { destroy } for pid=24489 comm="aisexec" key=893429658 scontext=root:system_r:aisexec_t:s0 tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem Fix: Added a new selinux policy package to allow qpidd to start in contexts other than service start. Result: no leaked semaphores, no problems starting qpidd.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-03 23:11:51 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
seplinux policy package to resolve the problem none

Description Alan Conway 2011-02-04 19:27:35 UTC
Description of problem: 

When run from the command line or test script (and not via the "service qpidd start" command) a clustered qpidd leaks semaphores


Version-Release number of selected component (if applicable): 0:0.7.946106-28.el5


How reproducible: 100%


Steps to Reproduce:

Do this as a non-root user (it works as root also but results will be obscured by other root-owned system semaphores)

[aconway@mrg32 ~]$ ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     

[aconway@mrg32 ~]$ qpidd -d --cluster-name foo --auth no
[aconway@mrg32 ~]$ qpid-latency-test
Sent 1000 msgs through latency-test-1 in 76ms (13157.9 msgs/s) Latency(ms): min=3.022, max=8.996, avg=5.87668
[aconway@mrg32 ~]$ ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x532c3824 1615921154 aconway   600        3         

[aconway@mrg32 ~]$ pkill qpidd
[aconway@mrg32 ~]$ ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x532c3824 1615921154 aconway   600        3         

Actual results: Final ipcs -s shows a semaphore still exits.


Expected results: ipcs -s should show no semaphores


Additional info:

Comment 1 Alan Conway 2011-02-04 19:34:42 UTC
Created attachment 477102 [details]
seplinux policy package to resolve the problem

If qpidd is started other than by the service start command, it runs in a different  selinux context, and selinux is not allowing the semaphore destroy as we can see from the audit log:

[root@mrg-qe-01 audit]# hostname
mrg-qe-01.lab.eng.brq.redhat.com
[root@mrg-qe-01 audit]# grep sem /var/log/audit/audit.log
type=AVC msg=audit(1296806605.532:30390): avc:  denied  { destroy } for  pid=24489 comm="aisexec" key=893429658 scontext=root:system_r:aisexec_t:s0 tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem
type=AVC msg=audit(1296806612.589:30392): avc:  denied  { destroy } for  pid=24489 comm="aisexec" key=1430045791 scontext=root:system_r:aisexec_t:s0 tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem

The attached selinux policy package qpiddevel.pp resolves this problem. It should be installed by the RPM alongside the qpidd.pp policy that we already install as customers will also likely run into this issue.  

Note that this issue does not occur if qpidd is started via the "service start" command, as the existing qpidd.pp covers that case. However customers in development, like ourselves, will want to start qpidd in other contexts for their own testing so we should install qpiddevel.pp as well.

Comment 2 Alan Conway 2011-02-04 19:45:43 UTC
To install the policy package do:
  selinux -i qpiddevel.pp
and then verify the leaked semaphore problem no longer occurs.

Comment 3 Alan Conway 2011-02-04 19:45:43 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: qpidd is started from the command line or from a script, and not via the "service qpidd start" command. SELinux is in Enforcing mode. Note this does not apply when qpidd is started as a service via "service qpidd start".

Consequence: Over many restarts, qpidd consumes semaphores as shown by the "ipcs -s" command. Eventually when the system limit is reached, qpidd will hang during startup with the last message in the log being: 
  notice Initializing CPG
The selinux audit log will also show messages like this:
type=AVC msg=audit(1296806605.532:30390): avc:  denied  { destroy } for  pid=24489 comm="aisexec" key=893429658 scontext=root:system_r:aisexec_t:s0 tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem

Fix: Added a new selinux policy package to allow qpidd to start in contexts other than service start.

Result: no leaked semaphores, no problems starting qpidd.
Fix: added a new policy

Comment 4 Alan Conway 2011-02-04 19:46:48 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -7,5 +7,4 @@
 
 Fix: Added a new selinux policy package to allow qpidd to start in contexts other than service start.
 
-Result: no leaked semaphores, no problems starting qpidd.
+Result: no leaked semaphores, no problems starting qpidd.-Fix: added a new policy

Comment 5 Justin Ross 2011-03-17 19:01:02 UTC
Moving this one out of 2.0.  It's a potential problem for developers, not standard deployments, and the fix raises its own problems.

Comment 6 Justin Ross 2013-02-24 16:49:18 UTC
Did anyone or thing consume the policy change? I can't tell where this got fixed.

Comment 7 Mike McCune 2016-03-28 21:59:03 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions