Bug 485750

Summary: openais-0.80.5-1 hangs in cpg_local_get
Product: Red Hat Enterprise MRG Reporter: Alan Conway <aconway>
Component: qpid-cppAssignee: Justin Ross <jross>
Status: CLOSED CURRENTRELEASE QA Contact: Frantisek Reznicek <freznice>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.1CC: clasohm, davids, dwalsh, esammons, fedora-jasper, freznice, iannis, iboverma, jdigilio, jneedle, mgrepl, ohudlick
Target Milestone: 1.1.1   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-27 20:47:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 486116    
Attachments:
Description Flags
audit log when running selinux in permissive mode with openais started from init script. none

Description Alan Conway 2009-02-16 17:29:24 UTC 
Description of problem:

qpidd is hanging in cpg_local_get during startup, see bug 485689 for more details.


Version-Release number of selected component (if applicable):


How reproducible:

On dell-pesc430-03.rhts.bos.redhat.com the hang occurs reliabily every time. 

On another host mrg10.lab.bos.redhat.com I don't see the hang during startup
but I do see other deadlocks and error, will raise separate BZ.

Steps to Reproduce:

 qpidd --cluster-name foo 

This runs the installed qpidd. I also tried rebuilding qpidd against the 0.80.5 headers, same result.
Comment 1 Steven Dake 2009-02-17 23:36:16 UTC 
Starting aisexec with the init script causes access permission denied on reading configuration files.  Turning SELINUX to permissive shows in the audit log that the request to read the config file has issues.  Also there appear to be other permission issues such as unknown netlink operations.

I am not qualified to fix selinux packaging issues related to our product.

Turning selinux to permissive causes openais to work as expected but qpidd may still have some issues when run from the command line.  This may be a cluster-wide issue though because of the permissions available to openais.

Running without the init script works great even with selinux set to enforcing.
Comment 2 Steven Dake 2009-02-17 23:38:58 UTC 
Created attachment 332316 [details]
audit log when running selinux in permissive mode with openais started from init script.
Comment 3 Jeff Needle 2009-02-18 15:03:03 UTC 
Here's the readers' digest version of the issues in the audit logs.  Looks like leaked file descriptors, an unlabeled directory, and a couple of other minor issues - I've asked Dan to weigh in.

[root@dell-pe2850-01 qpid_test_mnode_cluster]# hostname
dell-pe2850-01.rhts.bos.redhat.com
[root@dell-pe2850-01 qpid_test_mnode_cluster]# uname -a
Linux dell-pe2850-01.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 i686 i386 GNU/Linux
[root@dell-pe2850-01 qpid_test_mnode_cluster]# audit2allow -a


#============= ccs_t ==============
allow ccs_t default_t:file { read getattr };
allow ccs_t tmpfs_t:file { read write };
allow ccs_t unconfined_t:sem { write unix_read unix_write setattr associate read destroy };
allow ccs_t unconfined_t:shm { write unix_read unix_write associate read destroy };

#============= groupadd_t ==============
allow groupadd_t user_home_t:file append;

#============= nscd_t ==============
allow nscd_t user_home_t:file append;

#============= restorecon_t ==============
allow restorecon_t user_home_t:file append;

#============= semanage_t ==============
allow semanage_t user_home_t:file append;





[root@dell-pesc430-03 qpid_test_mnode_cluster]# hostname
dell-pesc430-03.rhts.bos.redhat.com
[root@dell-pesc430-03 qpid_test_mnode_cluster]# uname -a
Linux dell-pesc430-03.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-pesc430-03 qpid_test_mnode_cluster]# audit2allow -a


#============= ccs_t ==============
allow ccs_t default_t:file read;
allow ccs_t tmpfs_t:file { read write };
allow ccs_t unconfined_t:sem { write unix_read unix_write associate read destroy };
allow ccs_t unconfined_t:shm { write unix_read unix_write associate read destroy };

#============= groupadd_t ==============
allow groupadd_t user_home_t:file append;

#============= nscd_t ==============
allow nscd_t user_home_t:file append;

#============= restorecon_t ==============
allow restorecon_t user_home_t:file append;

#============= semanage_t ==============
allow semanage_t user_home_t:file append;
Comment 4 Daniel Walsh 2009-02-18 15:19:57 UTC 
All of the append avcs are caused by your tests redirecting stdout to 

/root/qpid_test_mytps/yum.log

Setup your tests to output to a log file you would have more chance at success 

/var/log/qpid_test_mytps/yum.log

for example, or add a test module to allow this to succeed

allow ccs_t unconfined_t:sem { write unix_read unix_write associate read
destroy };
allow ccs_t unconfined_t:shm { write unix_read unix_write associate read
destroy };

ccs_t is trying to communicate via sem and shm to a process running as unconfined_t,  Looks like someone started a process as the unconfined_t user and expected ccs_t to be able to communicate with it, not a valid test.

allow ccs_t tmpfs_t:file { read write };

This might be related to the unconfined_t process above.

/etc/ais/openais.conf is a mislabled file which was created at / and then moved into /etc/ais directory.  If you are going to move files around you need to verify the label is correct.  restorecon -R -v /etc/ais 

will fix.

While there may be a bug here, I see this as all problems with the test scripts.
Comment 5 Jeff Needle 2009-02-18 18:40:31 UTC 
Still seeing these after cleaning up labels:

#============= ccs_t ==============
allow ccs_t self:capability ipc_owner;
allow ccs_t unconfined_t:sem { unix_read unix_write };
allow ccs_t unconfined_t:shm { unix_read unix_write };


root:system_r:ccs_t             root     19691     1  0 11:52 pts/1    00:00:01 /usr/sbin/aisexec

qpidd hangs on startup:

root:system_r:initrc_t          qpidd    20681     1  0 13:35 ?        00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd --mgmt-enable yes --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test -p 0

Setting to Permissive, the process starts, but we get some additional AVCs:

#============= ccs_t ==============
allow ccs_t initrc_t:sem { unix_read read write unix_write associate };
allow ccs_t initrc_t:shm { unix_read read write unix_write associate };
allow ccs_t tmpfs_t:file { read write };

strace shows it hanging here:

semop(2064434, 0x7fff288c8630, 1
Comment 6 Daniel Walsh 2009-02-18 20:13:21 UTC 
So we either need custom policy for this packaged with your package or we need to wait for the U4 update.

audit2allow -la -M myqpidd
Will generate a polciy module that you can install and make sure it works.
Comment 7 Frantisek Reznicek 2009-02-19 08:53:54 UTC 
Please find below last observations on RHEL 5.3 i386 / x86_64 with SELinux in Enforcing mode and disabled.
Could you possibly review my steps and let me know whether the procedure is correct, please?

Summary is following:
  qpidd in cluster mode doesn't start-up with openais-0.80.5-1/2 on RHEL 5.3 i386/x86_64 with default SELinux enforce mode. (When SELinux disabled then it starts well, but of course this is workaround not solution)


// --------------------------------------------------------------------------
// [1a] RHEL 5.3 i386 qpidd start-up | SELinux in enforce mode (restorecon executed before openais start)
//      qpidd not starting well
// --------------------------------------------------------------------------
[root@dell-pe2850-01 _oat]# hostname
dell-pe2850-01.rhts.bos.redhat.com
[root@dell-pe2850-01 _oat]# uname -a
Linux dell-pe2850-01.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 i686 i386 GNU/Linux
[root@dell-pe2850-01 _oat]# service openais stop
Stopping OpenAIS daemon (aisexec): [  OK  ]
[root@dell-pe2850-01 _oat]#
[root@dell-pe2850-01 _oat]# cat /dev/null > /var/log/audit/audit.log
[root@dell-pe2850-01 _oat]# rm -rf data-test/
[root@dell-pe2850-01 _oat]# restorecon -R -v /etc/ais ; setenforce 1
[root@dell-pe2850-01 _oat]# service openais start
Starting OpenAIS daemon (aisexec): [  OK  ]
[root@dell-pe2850-01 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/rdma.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so
2009-feb-19 03:33:37 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/client/sslconnector.so
2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2009-feb-19 03:33:37 notice Journal "TplStore": Created
2009-feb-19 03:33:37 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl"
2009-feb-19 03:33:37 notice Store module initialized; dir=data-test
2009-feb-19 03:33:37 info > Default files per journal: 8
2009-feb-19 03:33:37 info > Default journal file size: 24 (wpgs)
2009-feb-19 03:33:37 info > Default write cache page size: 32 (Kib)
2009-feb-19 03:33:37 info > Default number of write cache pages: 32
2009-feb-19 03:33:37 info > TPL files per journal: 8
2009-feb-19 03:33:37 info > TPL journal file size: 24 (wpgs)
2009-feb-19 03:33:37 info > TPL write cache page size: 4 (Kib)
2009-feb-19 03:33:37 info > TPL number of write cache pages: 64
2009-feb-19 03:33:37 debug Initialized CPG handle 0x17f345ea00000000

[1]+  Stopped                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pe2850-01 _oat]# bg
[1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pe2850-01 _oat]# !ps
ps auxw | grep qpidd
root     21227  0.4  0.9  57052  9376 pts/0    Sl   03:33   0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
root     21233  0.0  0.0   3912   708 pts/0    S+   03:33   0:00 grep qpidd
[root@dell-pe2850-01 _oat]# !nets
netstat -nlp | grep qpidd
[root@dell-pe2850-01 _oat]# audit2allow -a -l -e


#============= ccs_t ==============
# audit(1235032417.513:181):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023"
#  class="sem" perms="{ unix_read unix_write }"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235032417.513:181): avc:  denied  { unix_read
#   unix_write } for  pid=21223 comm="aisexec" key=96052010
#   scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem"
allow ccs_t unconfined_t:sem { unix_read unix_write };
# audit(1235032417.513:180):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023"
#  class="shm" perms="{ unix_read unix_write }"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235032417.513:180): avc:  denied  { unix_read
#   unix_write } for  pid=21223 comm="aisexec" key=859548187
#   scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=shm"
allow ccs_t unconfined_t:shm { unix_read unix_write };
[root@dell-pe2850-01 _oat]# fg
qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0

[root@dell-pe2850-01 _oat]# jobs
[root@dell-pe2850-01 _oat]# service openais stop
Stopping OpenAIS daemon (aisexec): [  OK  ]

// --------------------------------------------------------------------------
// [1b] RHEL 5.3 i386 qpidd start-up | SELinux disabled mode
//      qpidd starting ok (as expected)
// --------------------------------------------------------------------------

[root@dell-pe2850-01 _oat]# rm -rf data-test/
[root@dell-pe2850-01 _oat]# restorecon -R -v /etc/ais ; setenforce 0
[root@dell-pe2850-01 _oat]# service openais status
aisexec is stopped
[root@dell-pe2850-01 _oat]# service openais start
Starting OpenAIS daemon (aisexec): [  OK  ]
[root@dell-pe2850-01 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/acl.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/xml.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/rdma.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so
2009-feb-19 03:36:10 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/client/sslconnector.so
2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/cluster.so
2009-feb-19 03:36:10 notice Journal "TplStore": Created
2009-feb-19 03:36:10 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl"
2009-feb-19 03:36:10 notice Store module initialized; dir=data-test
2009-feb-19 03:36:10 info > Default files per journal: 8
2009-feb-19 03:36:10 info > Default journal file size: 24 (wpgs)
2009-feb-19 03:36:10 info > Default write cache page size: 32 (Kib)
2009-feb-19 03:36:10 info > Default number of write cache pages: 32
2009-feb-19 03:36:10 info > TPL files per journal: 8
2009-feb-19 03:36:10 info > TPL journal file size: 24 (wpgs)
2009-feb-19 03:36:10 info > TPL write cache page size: 4 (Kib)
2009-feb-19 03:36:10 info > TPL number of write cache pages: 64
2009-feb-19 03:36:10 debug Initialized CPG handle 0x255e971000000000
2009-feb-19 03:36:10 debug 10.16.65.59:21314(INIT) config change: 10.16.65.59:21314
2009-feb-19 03:36:10 info Management not enabled
2009-feb-19 03:36:10 notice SASL disabled: No Authentication Performed
2009-feb-19 03:36:10 notice Listening on TCP port 55582
2009-feb-19 03:36:10 info Policy file not specified. ACL Disabled, no ACL checking being done!
2009-feb-19 03:36:10 info Registered replication exchange
2009-feb-19 03:36:10 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
2009-feb-19 03:36:10 info Registered xml exchange
libibverbs: Fatal: couldn't read uverbs ABI version.
2009-feb-19 03:36:10 info Rdma: Disabled: no rdma devices found
2009-feb-19 03:36:10 notice 10.16.65.59:21314(INIT) joining cluster qpid_mnode_cluster_test with url=amqp:tcp:10.16.65.59:55582
55582
2009-feb-19 03:36:10 notice Broker running
2009-feb-19 03:36:10 debug 10.16.65.59:21314(INIT) cluster-id = 651d0c75-33b6-408e-8a5b-d3f00859ef4d
2009-feb-19 03:36:10 info 10.16.65.59:21314(READY) member update: 10.16.65.59:21314(member)
2009-feb-19 03:36:10 notice 10.16.65.59:21314(READY) first in cluster

[1]+  Stopped                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pe2850-01 _oat]# bg
[1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pe2850-01 _oat]# !ps
ps auxw | grep qpidd
root     21314  0.6  0.9  98468  9388 pts/0    Sl   03:36   0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
root     21324  0.0  0.0   3912   684 pts/0    S+   03:36   0:00 grep qpidd
[root@dell-pe2850-01 _oat]# !nets
netstat -nlp | grep qpidd
tcp        0      0 0.0.0.0:55582               0.0.0.0:*                   LISTEN      21314/qpidd
[root@dell-pe2850-01 _oat]# jobs
[1]+  Running                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pe2850-01 _oat]# kill %1
[root@dell-pe2850-01 _oat]# 2009-feb-19 03:36:30 notice 10.16.65.59:21314(READY) shutting down
2009-feb-19 03:36:30 debug Shutting down CPG
2009-feb-19 03:36:30 notice Shut down
2009-feb-19 03:36:30 debug Journal "TplStore": Destroyed

[1]+  Done                    qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0


// --------------------------------------------------------------------------
// [2a] RHEL 5.3 i386 qpidd start-up | SELinux in enforce mode (restorecon executed before openais start)
//      qpidd not starting well
// --------------------------------------------------------------------------

[root@dell-pesc430-03 _oat]# hostname
dell-pesc430-03.rhts.bos.redhat.com
[root@dell-pesc430-03 _oat]# uname -a
Linux dell-pesc430-03.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-pesc430-03 _oat]# service openais stop
Stopping OpenAIS daemon (aisexec): [FAILED]
[root@dell-pesc430-03 _oat]# service openais status
aisexec is stopped
[root@dell-pesc430-03 _oat]# cat /dev/null > /var/log/audit/audit.log
[root@dell-pesc430-03 _oat]# rm -rf data-test/
[root@dell-pesc430-03 _oat]# restorecon -R -v /etc/ais ; setenforce 1
[root@dell-pesc430-03 _oat]# service openais start
Starting OpenAIS daemon (aisexec): [  OK  ]
[root@dell-pesc430-03 _oat]# audit2allow -a -l -e
<no matches>


[root@dell-pesc430-03 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so
2009-feb-19 03:41:08 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2009-feb-19 03:41:08 debug Initialized CPG handle 0x624154be00000000

[1]+  Stopped                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pesc430-03 _oat]# bg
[1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pesc430-03 _oat]# ps auxw | grep qpidd
root     22820  0.2  0.9 133208  9340 pts/0    Sl   03:41   0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
root     22825  0.0  0.0  61180   728 pts/0    R+   03:41   0:00 grep qpidd
[root@dell-pesc430-03 _oat]# netstat -nlp | grep qpidd
[root@dell-pesc430-03 _oat]# audit2allow -a -l -e


#============= ccs_t ==============
# audit(1235032868.738:3146):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023"
#  class="sem" perms="{ unix_read unix_write }"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235032868.738:3146): avc:  denied  { unix_read
#   unix_write } for  pid=22812 comm="aisexec" key=1570360252
#   scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem"
allow ccs_t unconfined_t:sem { unix_read unix_write };
# audit(1235032868.738:3145):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023"
#  class="shm" perms="{ unix_read unix_write }"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235032868.738:3145): avc:  denied  { unix_read
#   unix_write } for  pid=22812 comm="aisexec" key=1905983932
#   scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=shm"
allow ccs_t unconfined_t:shm { unix_read unix_write };
[root@dell-pesc430-03 _oat]# jobs
[1]+  Running                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pesc430-03 _oat]# kill %1
[root@dell-pesc430-03 _oat]#
[1]+  Terminated              qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pesc430-03 _oat]#
[root@dell-pesc430-03 _oat]# service openais stop
Stopping OpenAIS daemon (aisexec): [  OK  ]

// --------------------------------------------------------------------------
// [2b] RHEL 5.3 x86_64 qpidd start-up | SELinux disabled mode
//      qpidd starting ok (as expected)
// --------------------------------------------------------------------------


[root@dell-pesc430-03 _oat]#
[root@dell-pesc430-03 _oat]# rm -rf data-test/
[root@dell-pesc430-03 _oat]# restorecon -R -v /etc/ais ; setenforce 0
[root@dell-pesc430-03 _oat]# service openais status
aisexec is stopped
[root@dell-pesc430-03 _oat]# service openais start
Starting OpenAIS daemon (aisexec): [  OK  ]
[root@dell-pesc430-03 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so
2009-feb-19 03:42:36 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2009-feb-19 03:42:36 debug Initialized CPG handle 0x43fac70400000000
2009-feb-19 03:42:36 debug 10.16.64.106:22886(INIT) config change: 10.16.64.106:22886
2009-feb-19 03:42:36 notice Journal "TplStore": Created
2009-feb-19 03:42:36 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl"
2009-feb-19 03:42:36 notice Store module initialized; dir=data-test
2009-feb-19 03:42:36 info > Default files per journal: 8
2009-feb-19 03:42:36 info > Default journal file size: 24 (wpgs)
2009-feb-19 03:42:36 info > Default write cache page size: 32 (Kib)
2009-feb-19 03:42:36 info > Default number of write cache pages: 32
2009-feb-19 03:42:36 info > TPL files per journal: 8
2009-feb-19 03:42:36 info > TPL journal file size: 24 (wpgs)
2009-feb-19 03:42:36 info > TPL write cache page size: 4 (Kib)
2009-feb-19 03:42:36 info > TPL number of write cache pages: 64
2009-feb-19 03:42:36 info Management not enabled
2009-feb-19 03:42:36 notice SASL disabled: No Authentication Performed
2009-feb-19 03:42:36 notice Listening on TCP port 55328
libibverbs: Fatal: couldn't read uverbs ABI version.
2009-feb-19 03:42:36 info Rdma: Disabled: no rdma devices found
2009-feb-19 03:42:36 notice 10.16.64.106:22886(INIT) joining cluster qpid_mnode_cluster_test with url=amqp:tcp:10.16.64.106:55328
2009-feb-19 03:42:36 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it.
2009-feb-19 03:42:36 info Registered xml exchange
2009-feb-19 03:42:36 info Registered replication exchange
2009-feb-19 03:42:36 info Policy file not specified. ACL Disabled, no ACL checking being done!
55328
2009-feb-19 03:42:36 notice Broker running
2009-feb-19 03:42:36 debug 10.16.64.106:22886(INIT) cluster-id = 9fb4b2e4-957e-41cd-8792-acd04f7714b6
2009-feb-19 03:42:36 info 10.16.64.106:22886(READY) member update: 10.16.64.106:22886(member)
2009-feb-19 03:42:36 notice 10.16.64.106:22886(READY) first in cluster

[1]+  Stopped                 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pesc430-03 _oat]# bg
[1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 &
[root@dell-pesc430-03 _oat]# !ps
ps auxw | grep qpidd
root     22886  0.8  1.0 165632 10864 pts/0    Sl   03:42   0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
root     22894  0.0  0.0  61180   732 pts/0    S+   03:42   0:00 grep qpidd
[root@dell-pesc430-03 _oat]# !nets
netstat -nlp | grep qpidd
tcp        0      0 0.0.0.0:55328               0.0.0.0:*                   LISTEN      22886/qpidd
[root@dell-pesc430-03 _oat]# kill %1
2009-feb-19 03:43:01 notice 10.16.64.106:22886(READY) shutting down
[root@dell-pesc430-03 _oat]# 2009-feb-19 03:43:01 debug Shutting down CPG
2009-feb-19 03:43:01 notice Shut down
2009-feb-19 03:43:01 debug Journal "TplStore": Destroyed

[1]+  Done                    qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0
[root@dell-pesc430-03 _oat]#        

// eof
Comment 8 Frantisek Reznicek 2009-02-19 16:07:11 UTC 
Hello Daniel,
Could you possibly validate whether below pasted qpidd selinux policy looks good to you (make sense to you).
I verified that this policy allows the qpidd start with openais-0.80-1/2 in SELinux enforce mode.

[root@dell-pe2850-01 _oat]# cat qpiddselinuxpolicy5.te

module qpiddselinuxpolicy5 1.0;

require {
        type unconfined_t;
        type ccs_t;
        type tmpfs_t;
        class sem { unix_read read write unix_write associate };
        class shm { unix_read read write unix_write associate };
        class file { read write };
}

#============= ccs_t ==============
allow ccs_t tmpfs_t:file { read write };
allow ccs_t unconfined_t:sem { unix_read read write unix_write associate };
allow ccs_t unconfined_t:shm { unix_read read write unix_write associate };

Thanks,
Frantisek
Comment 9 Frantisek Reznicek 2009-02-19 17:11:27 UTC 
Corrected data for review are here:

[root@dell-pe2850-01 _oat]# cat qpidd.te

module qpidd 1.0;

require {
        type initrc_t;
        type ccs_t;
        type tmpfs_t;
        class sem { unix_read read write unix_write associate };
        class shm { unix_read read write unix_write associate };
        class file { read write };
}

#============= ccs_t ==============
allow ccs_t tmpfs_t:file { read write };
allow ccs_t initrc_t:sem { unix_read read write unix_write associate };
allow ccs_t initrc_t:shm { unix_read read write unix_write associate };

I can see expected behavior on RHEL 5.3 i386 at the moment. Going to go through x86_64 (and RHEL5.2 i386/x86_64)
Comment 10 Daniel Walsh 2009-02-19 21:40:53 UTC 
This looks good, this bug should be reassigned to another package though.
Comment 11 Frantisek Reznicek 2009-02-20 12:24:53 UTC 
I retested on all used platforms (RHEL 5.2/3 i[36]86/x86_64) for MRG and got to a bit extended one. Could you possibly review once more Daniel, please?

module qpidd 1.0;

require {
        type initrc_t;
        type ccs_t;
        type tmpfs_t;
        class sem { unix_read read write unix_write associate };
        class shm { unix_read read write unix_write associate };
        class file { read write };
        class capability ipc_owner;
}

#============= ccs_t ==============
allow ccs_t tmpfs_t:file { read write };
allow ccs_t initrc_t:sem { unix_read read write unix_write associate };
allow ccs_t initrc_t:shm { unix_read read write unix_write associate };
allow ccs_t self:capability ipc_owner;

Thanks in advance,
Frantisek

P.S. selinux policy from #9 works well on RHEL 5.3 i386 only / the other architectures needs 'allow ccs_t self:capability ipc_owner;' rule in addition.
Comment 12 Daniel Walsh 2009-02-20 13:55:40 UTC 
Looks good.

Try this policy in stead

poliyc_module(qpidd, 1.0)

require {
        type initrc_t;
        type ccs_t;
}

#============= ccs_t ==============
fs_rw_tmpfs_files(ccs_t)
allow ccs_t initrc_t:sem rw_sem_perms;
allow ccs_t initrc_t:shm rw_shm_perms;
allow ccs_t self:capability ipc_owner;

I would like to start writing real qpidd policy soon for RHEL6/F-11
Comment 13 Frantisek Reznicek 2009-02-20 15:05:59 UTC 
yes, tested with your updated one on RHEL 5.3 i386 and x86_64 and both work

after correct qpidd startup I can see:

[root@dell380-2 _oat]# audit2allow -a -l -e

#============= ccs_t ==============
# audit(1235141689.430:117):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:initrc_t:s0"
#  class="sem" perms="destroy"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235141689.430:117): avc:  denied  { destroy } for
#   pid=23092 comm="aisexec" key=1164666434 scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:initrc_t:s0 tclass=sem"
allow ccs_t initrc_t:sem destroy;
# audit(1235141689.430:116):
#  scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:initrc_t:s0"
#  class="shm" perms="destroy"
#  comm="aisexec" exe="" path=""
#  message="type=AVC msg=audit(1235141689.430:116): avc:  denied  { destroy } for
#   pid=23092 comm="aisexec" key=563006265 scontext=root:system_r:ccs_t:s0
#   tcontext=root:system_r:initrc_t:s0 tclass=shm"
Comment 14 Daniel Walsh 2009-02-20 15:52:53 UTC 
Change the rw_*perms to create_*_perms;

Does aisexec run as a daemon?
Comment 15 Alan Conway 2009-02-20 18:20:48 UTC 
The init.d/openais script runs it with the daemon() function from /etc/init.d/functions

Also note the hang only occurs when aisexec is started via the init script. If aisexec is started directly from the command line or a script there's no problem (which is why we didn't detect this earlier)
Comment 16 Daniel Walsh 2009-02-20 19:02:52 UTC 
I wonder if /etc/init.d/openais runs before qpid are the avcs different non existant.  

The avcs are related to whom created the shm and sem, if they are created by openais they would be labeled ccs_t, rather then initrc_t and I believe this would just work.

But since qpid is creating them they are labeled initrc_t, and ccs_t is not allowed access.

Running an app directly from unconfined_t will not transition while running it from a script labeled initrc_exec_t will.  This is why you see a difference between running it via init versus running it directly.
Comment 18 Daniel Walsh 2009-02-25 15:54:44 UTC 
Fixed typo

policy_module(qpidd, 1.0)

require {
        type initrc_t;
        type ccs_t;
}

#============= ccs_t ==============
fs_rw_tmpfs_files(ccs_t)
allow ccs_t initrc_t:sem rw_sem_perms;
allow ccs_t initrc_t:shm rw_shm_perms;
allow ccs_t self:capability ipc_owner;

You can compile this using make -f /usr/share/selinux/devel/Makefile

And ship the pp file, which you want to install in the post install

semodule -i qpidd.pp

Additional Info
http://danwalsh.livejournal.com/24750.html
Comment 19 Justin Ross 2009-02-26 16:58:09 UTC 
The new policy is now packaged with the latest rpm in distcvs.
Comment 20 David Sommerseth 2009-02-27 15:36:03 UTC 
Tested against 0.4.747892-1.el5.x86_64.rpms (https://brewweb.devel.redhat.com/taskinfo?taskID=1705562) and the current candidate release (0.4.744917-1.el5) rpms.

Reproduction:

- Make sure SELinux is Enforced (getenforce)
- Upgrade openais to openais-0.80.5-2.el5
- Configure openais (/etc/ais/openais.conf) and a corresponding network interface.
- Configure qpidd (/etc/qpidd.conf).  Add cluster-name=foo
- Test 1: Try to start qpidd from command line (qpidd -t --cluster-name=foo)
- Test 2: Try to start qpidd via service (service qpidd start)

For the candidate release, both tests failed due to qpidd hanging.

For the brewbuilds qpidd starts perfectly well via the service command (Test 2) but hangs when starting via command line (Test 1).

This is the correct behavior, since the SELinux context will be different when started from command line and via service/init rc.
Comment 21 Jasper Capel 2009-04-29 07:31:48 UTC 
I have encountered this bug on a RHEL5.3 x86_64 clustered node and as I'm "just" using the cluster suite but not qpid, this problem does not seem exclusive to qpid to me. In my situation it manifests as hanging on "Starting fencing..." on starting the cman service.
Comment 22 Justin Ross 2011-06-27 20:47:16 UTC 
Fixed and verified wrt qpid; closing.