Description of problem: qpidd is hanging in cpg_local_get during startup, see bug 485689 for more details. Version-Release number of selected component (if applicable): How reproducible: On dell-pesc430-03.rhts.bos.redhat.com the hang occurs reliabily every time. On another host mrg10.lab.bos.redhat.com I don't see the hang during startup but I do see other deadlocks and error, will raise separate BZ. Steps to Reproduce: qpidd --cluster-name foo This runs the installed qpidd. I also tried rebuilding qpidd against the 0.80.5 headers, same result.
Starting aisexec with the init script causes access permission denied on reading configuration files. Turning SELINUX to permissive shows in the audit log that the request to read the config file has issues. Also there appear to be other permission issues such as unknown netlink operations. I am not qualified to fix selinux packaging issues related to our product. Turning selinux to permissive causes openais to work as expected but qpidd may still have some issues when run from the command line. This may be a cluster-wide issue though because of the permissions available to openais. Running without the init script works great even with selinux set to enforcing.
Created attachment 332316 [details] audit log when running selinux in permissive mode with openais started from init script.
Here's the readers' digest version of the issues in the audit logs. Looks like leaked file descriptors, an unlabeled directory, and a couple of other minor issues - I've asked Dan to weigh in. [root@dell-pe2850-01 qpid_test_mnode_cluster]# hostname dell-pe2850-01.rhts.bos.redhat.com [root@dell-pe2850-01 qpid_test_mnode_cluster]# uname -a Linux dell-pe2850-01.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 i686 i386 GNU/Linux [root@dell-pe2850-01 qpid_test_mnode_cluster]# audit2allow -a #============= ccs_t ============== allow ccs_t default_t:file { read getattr }; allow ccs_t tmpfs_t:file { read write }; allow ccs_t unconfined_t:sem { write unix_read unix_write setattr associate read destroy }; allow ccs_t unconfined_t:shm { write unix_read unix_write associate read destroy }; #============= groupadd_t ============== allow groupadd_t user_home_t:file append; #============= nscd_t ============== allow nscd_t user_home_t:file append; #============= restorecon_t ============== allow restorecon_t user_home_t:file append; #============= semanage_t ============== allow semanage_t user_home_t:file append; [root@dell-pesc430-03 qpid_test_mnode_cluster]# hostname dell-pesc430-03.rhts.bos.redhat.com [root@dell-pesc430-03 qpid_test_mnode_cluster]# uname -a Linux dell-pesc430-03.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [root@dell-pesc430-03 qpid_test_mnode_cluster]# audit2allow -a #============= ccs_t ============== allow ccs_t default_t:file read; allow ccs_t tmpfs_t:file { read write }; allow ccs_t unconfined_t:sem { write unix_read unix_write associate read destroy }; allow ccs_t unconfined_t:shm { write unix_read unix_write associate read destroy }; #============= groupadd_t ============== allow groupadd_t user_home_t:file append; #============= nscd_t ============== allow nscd_t user_home_t:file append; #============= restorecon_t ============== allow restorecon_t user_home_t:file append; #============= semanage_t ============== allow semanage_t user_home_t:file append;
All of the append avcs are caused by your tests redirecting stdout to /root/qpid_test_mytps/yum.log Setup your tests to output to a log file you would have more chance at success /var/log/qpid_test_mytps/yum.log for example, or add a test module to allow this to succeed allow ccs_t unconfined_t:sem { write unix_read unix_write associate read destroy }; allow ccs_t unconfined_t:shm { write unix_read unix_write associate read destroy }; ccs_t is trying to communicate via sem and shm to a process running as unconfined_t, Looks like someone started a process as the unconfined_t user and expected ccs_t to be able to communicate with it, not a valid test. allow ccs_t tmpfs_t:file { read write }; This might be related to the unconfined_t process above. /etc/ais/openais.conf is a mislabled file which was created at / and then moved into /etc/ais directory. If you are going to move files around you need to verify the label is correct. restorecon -R -v /etc/ais will fix. While there may be a bug here, I see this as all problems with the test scripts.
Still seeing these after cleaning up labels: #============= ccs_t ============== allow ccs_t self:capability ipc_owner; allow ccs_t unconfined_t:sem { unix_read unix_write }; allow ccs_t unconfined_t:shm { unix_read unix_write }; root:system_r:ccs_t root 19691 1 0 11:52 pts/1 00:00:01 /usr/sbin/aisexec qpidd hangs on startup: root:system_r:initrc_t qpidd 20681 1 0 13:35 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd --mgmt-enable yes --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test -p 0 Setting to Permissive, the process starts, but we get some additional AVCs: #============= ccs_t ============== allow ccs_t initrc_t:sem { unix_read read write unix_write associate }; allow ccs_t initrc_t:shm { unix_read read write unix_write associate }; allow ccs_t tmpfs_t:file { read write }; strace shows it hanging here: semop(2064434, 0x7fff288c8630, 1
So we either need custom policy for this packaged with your package or we need to wait for the U4 update. audit2allow -la -M myqpidd Will generate a polciy module that you can install and make sure it works.
Please find below last observations on RHEL 5.3 i386 / x86_64 with SELinux in Enforcing mode and disabled. Could you possibly review my steps and let me know whether the procedure is correct, please? Summary is following: qpidd in cluster mode doesn't start-up with openais-0.80.5-1/2 on RHEL 5.3 i386/x86_64 with default SELinux enforce mode. (When SELinux disabled then it starts well, but of course this is workaround not solution) // -------------------------------------------------------------------------- // [1a] RHEL 5.3 i386 qpidd start-up | SELinux in enforce mode (restorecon executed before openais start) // qpidd not starting well // -------------------------------------------------------------------------- [root@dell-pe2850-01 _oat]# hostname dell-pe2850-01.rhts.bos.redhat.com [root@dell-pe2850-01 _oat]# uname -a Linux dell-pe2850-01.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 i686 i386 GNU/Linux [root@dell-pe2850-01 _oat]# service openais stop Stopping OpenAIS daemon (aisexec): [ OK ] [root@dell-pe2850-01 _oat]# [root@dell-pe2850-01 _oat]# cat /dev/null > /var/log/audit/audit.log [root@dell-pe2850-01 _oat]# rm -rf data-test/ [root@dell-pe2850-01 _oat]# restorecon -R -v /etc/ais ; setenforce 1 [root@dell-pe2850-01 _oat]# service openais start Starting OpenAIS daemon (aisexec): [ OK ] [root@dell-pe2850-01 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/acl.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/ssl.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/xml.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/rdma.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so 2009-feb-19 03:33:37 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it. 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/client/sslconnector.so 2009-feb-19 03:33:37 info Loaded Module: /usr/lib/qpid/daemon/cluster.so 2009-feb-19 03:33:37 notice Journal "TplStore": Created 2009-feb-19 03:33:37 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl" 2009-feb-19 03:33:37 notice Store module initialized; dir=data-test 2009-feb-19 03:33:37 info > Default files per journal: 8 2009-feb-19 03:33:37 info > Default journal file size: 24 (wpgs) 2009-feb-19 03:33:37 info > Default write cache page size: 32 (Kib) 2009-feb-19 03:33:37 info > Default number of write cache pages: 32 2009-feb-19 03:33:37 info > TPL files per journal: 8 2009-feb-19 03:33:37 info > TPL journal file size: 24 (wpgs) 2009-feb-19 03:33:37 info > TPL write cache page size: 4 (Kib) 2009-feb-19 03:33:37 info > TPL number of write cache pages: 64 2009-feb-19 03:33:37 debug Initialized CPG handle 0x17f345ea00000000 [1]+ Stopped qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pe2850-01 _oat]# bg [1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pe2850-01 _oat]# !ps ps auxw | grep qpidd root 21227 0.4 0.9 57052 9376 pts/0 Sl 03:33 0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 root 21233 0.0 0.0 3912 708 pts/0 S+ 03:33 0:00 grep qpidd [root@dell-pe2850-01 _oat]# !nets netstat -nlp | grep qpidd [root@dell-pe2850-01 _oat]# audit2allow -a -l -e #============= ccs_t ============== # audit(1235032417.513:181): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023" # class="sem" perms="{ unix_read unix_write }" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235032417.513:181): avc: denied { unix_read # unix_write } for pid=21223 comm="aisexec" key=96052010 # scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem" allow ccs_t unconfined_t:sem { unix_read unix_write }; # audit(1235032417.513:180): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023" # class="shm" perms="{ unix_read unix_write }" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235032417.513:180): avc: denied { unix_read # unix_write } for pid=21223 comm="aisexec" key=859548187 # scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=shm" allow ccs_t unconfined_t:shm { unix_read unix_write }; [root@dell-pe2850-01 _oat]# fg qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pe2850-01 _oat]# jobs [root@dell-pe2850-01 _oat]# service openais stop Stopping OpenAIS daemon (aisexec): [ OK ] // -------------------------------------------------------------------------- // [1b] RHEL 5.3 i386 qpidd start-up | SELinux disabled mode // qpidd starting ok (as expected) // -------------------------------------------------------------------------- [root@dell-pe2850-01 _oat]# rm -rf data-test/ [root@dell-pe2850-01 _oat]# restorecon -R -v /etc/ais ; setenforce 0 [root@dell-pe2850-01 _oat]# service openais status aisexec is stopped [root@dell-pe2850-01 _oat]# service openais start Starting OpenAIS daemon (aisexec): [ OK ] [root@dell-pe2850-01 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/acl.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/ssl.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/xml.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/rdma.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/client/rdmaconnector.so 2009-feb-19 03:36:10 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it. 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/client/sslconnector.so 2009-feb-19 03:36:10 info Loaded Module: /usr/lib/qpid/daemon/cluster.so 2009-feb-19 03:36:10 notice Journal "TplStore": Created 2009-feb-19 03:36:10 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl" 2009-feb-19 03:36:10 notice Store module initialized; dir=data-test 2009-feb-19 03:36:10 info > Default files per journal: 8 2009-feb-19 03:36:10 info > Default journal file size: 24 (wpgs) 2009-feb-19 03:36:10 info > Default write cache page size: 32 (Kib) 2009-feb-19 03:36:10 info > Default number of write cache pages: 32 2009-feb-19 03:36:10 info > TPL files per journal: 8 2009-feb-19 03:36:10 info > TPL journal file size: 24 (wpgs) 2009-feb-19 03:36:10 info > TPL write cache page size: 4 (Kib) 2009-feb-19 03:36:10 info > TPL number of write cache pages: 64 2009-feb-19 03:36:10 debug Initialized CPG handle 0x255e971000000000 2009-feb-19 03:36:10 debug 10.16.65.59:21314(INIT) config change: 10.16.65.59:21314 2009-feb-19 03:36:10 info Management not enabled 2009-feb-19 03:36:10 notice SASL disabled: No Authentication Performed 2009-feb-19 03:36:10 notice Listening on TCP port 55582 2009-feb-19 03:36:10 info Policy file not specified. ACL Disabled, no ACL checking being done! 2009-feb-19 03:36:10 info Registered replication exchange 2009-feb-19 03:36:10 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it. 2009-feb-19 03:36:10 info Registered xml exchange libibverbs: Fatal: couldn't read uverbs ABI version. 2009-feb-19 03:36:10 info Rdma: Disabled: no rdma devices found 2009-feb-19 03:36:10 notice 10.16.65.59:21314(INIT) joining cluster qpid_mnode_cluster_test with url=amqp:tcp:10.16.65.59:55582 55582 2009-feb-19 03:36:10 notice Broker running 2009-feb-19 03:36:10 debug 10.16.65.59:21314(INIT) cluster-id = 651d0c75-33b6-408e-8a5b-d3f00859ef4d 2009-feb-19 03:36:10 info 10.16.65.59:21314(READY) member update: 10.16.65.59:21314(member) 2009-feb-19 03:36:10 notice 10.16.65.59:21314(READY) first in cluster [1]+ Stopped qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pe2850-01 _oat]# bg [1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pe2850-01 _oat]# !ps ps auxw | grep qpidd root 21314 0.6 0.9 98468 9388 pts/0 Sl 03:36 0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 root 21324 0.0 0.0 3912 684 pts/0 S+ 03:36 0:00 grep qpidd [root@dell-pe2850-01 _oat]# !nets netstat -nlp | grep qpidd tcp 0 0 0.0.0.0:55582 0.0.0.0:* LISTEN 21314/qpidd [root@dell-pe2850-01 _oat]# jobs [1]+ Running qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pe2850-01 _oat]# kill %1 [root@dell-pe2850-01 _oat]# 2009-feb-19 03:36:30 notice 10.16.65.59:21314(READY) shutting down 2009-feb-19 03:36:30 debug Shutting down CPG 2009-feb-19 03:36:30 notice Shut down 2009-feb-19 03:36:30 debug Journal "TplStore": Destroyed [1]+ Done qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 // -------------------------------------------------------------------------- // [2a] RHEL 5.3 i386 qpidd start-up | SELinux in enforce mode (restorecon executed before openais start) // qpidd not starting well // -------------------------------------------------------------------------- [root@dell-pesc430-03 _oat]# hostname dell-pesc430-03.rhts.bos.redhat.com [root@dell-pesc430-03 _oat]# uname -a Linux dell-pesc430-03.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [root@dell-pesc430-03 _oat]# service openais stop Stopping OpenAIS daemon (aisexec): [FAILED] [root@dell-pesc430-03 _oat]# service openais status aisexec is stopped [root@dell-pesc430-03 _oat]# cat /dev/null > /var/log/audit/audit.log [root@dell-pesc430-03 _oat]# rm -rf data-test/ [root@dell-pesc430-03 _oat]# restorecon -R -v /etc/ais ; setenforce 1 [root@dell-pesc430-03 _oat]# service openais start Starting OpenAIS daemon (aisexec): [ OK ] [root@dell-pesc430-03 _oat]# audit2allow -a -l -e <no matches> [root@dell-pesc430-03 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so 2009-feb-19 03:41:08 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it. 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/xml.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so 2009-feb-19 03:41:08 info Loaded Module: /usr/lib64/qpid/daemon/acl.so 2009-feb-19 03:41:08 debug Initialized CPG handle 0x624154be00000000 [1]+ Stopped qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pesc430-03 _oat]# bg [1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pesc430-03 _oat]# ps auxw | grep qpidd root 22820 0.2 0.9 133208 9340 pts/0 Sl 03:41 0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 root 22825 0.0 0.0 61180 728 pts/0 R+ 03:41 0:00 grep qpidd [root@dell-pesc430-03 _oat]# netstat -nlp | grep qpidd [root@dell-pesc430-03 _oat]# audit2allow -a -l -e #============= ccs_t ============== # audit(1235032868.738:3146): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023" # class="sem" perms="{ unix_read unix_write }" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235032868.738:3146): avc: denied { unix_read # unix_write } for pid=22812 comm="aisexec" key=1570360252 # scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=sem" allow ccs_t unconfined_t:sem { unix_read unix_write }; # audit(1235032868.738:3145): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:unconfined_t:s0-s0:c0.c1023" # class="shm" perms="{ unix_read unix_write }" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235032868.738:3145): avc: denied { unix_read # unix_write } for pid=22812 comm="aisexec" key=1905983932 # scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:unconfined_t:s0-s0:c0.c1023 tclass=shm" allow ccs_t unconfined_t:shm { unix_read unix_write }; [root@dell-pesc430-03 _oat]# jobs [1]+ Running qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pesc430-03 _oat]# kill %1 [root@dell-pesc430-03 _oat]# [1]+ Terminated qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pesc430-03 _oat]# [root@dell-pesc430-03 _oat]# service openais stop Stopping OpenAIS daemon (aisexec): [ OK ] // -------------------------------------------------------------------------- // [2b] RHEL 5.3 x86_64 qpidd start-up | SELinux disabled mode // qpidd starting ok (as expected) // -------------------------------------------------------------------------- [root@dell-pesc430-03 _oat]# [root@dell-pesc430-03 _oat]# rm -rf data-test/ [root@dell-pesc430-03 _oat]# restorecon -R -v /etc/ais ; setenforce 0 [root@dell-pesc430-03 _oat]# service openais status aisexec is stopped [root@dell-pesc430-03 _oat]# service openais start Starting OpenAIS daemon (aisexec): [ OK ] [root@dell-pesc430-03 _oat]# qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/rdma.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/client/rdmaconnector.so 2009-feb-19 03:42:36 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it. 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/xml.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so 2009-feb-19 03:42:36 info Loaded Module: /usr/lib64/qpid/daemon/acl.so 2009-feb-19 03:42:36 debug Initialized CPG handle 0x43fac70400000000 2009-feb-19 03:42:36 debug 10.16.64.106:22886(INIT) config change: 10.16.64.106:22886 2009-feb-19 03:42:36 notice Journal "TplStore": Created 2009-feb-19 03:42:36 debug Journal "TplStore": Journal directory = "data-test/rhm/tpl/"; Base file name = "tpl" 2009-feb-19 03:42:36 notice Store module initialized; dir=data-test 2009-feb-19 03:42:36 info > Default files per journal: 8 2009-feb-19 03:42:36 info > Default journal file size: 24 (wpgs) 2009-feb-19 03:42:36 info > Default write cache page size: 32 (Kib) 2009-feb-19 03:42:36 info > Default number of write cache pages: 32 2009-feb-19 03:42:36 info > TPL files per journal: 8 2009-feb-19 03:42:36 info > TPL journal file size: 24 (wpgs) 2009-feb-19 03:42:36 info > TPL write cache page size: 4 (Kib) 2009-feb-19 03:42:36 info > TPL number of write cache pages: 64 2009-feb-19 03:42:36 info Management not enabled 2009-feb-19 03:42:36 notice SASL disabled: No Authentication Performed 2009-feb-19 03:42:36 notice Listening on TCP port 55328 libibverbs: Fatal: couldn't read uverbs ABI version. 2009-feb-19 03:42:36 info Rdma: Disabled: no rdma devices found 2009-feb-19 03:42:36 notice 10.16.64.106:22886(INIT) joining cluster qpid_mnode_cluster_test with url=amqp:tcp:10.16.64.106:55328 2009-feb-19 03:42:36 info SSL plugin not enabled, you must set --qpid-ssl-cert-db to enable it. 2009-feb-19 03:42:36 info Registered xml exchange 2009-feb-19 03:42:36 info Registered replication exchange 2009-feb-19 03:42:36 info Policy file not specified. ACL Disabled, no ACL checking being done! 55328 2009-feb-19 03:42:36 notice Broker running 2009-feb-19 03:42:36 debug 10.16.64.106:22886(INIT) cluster-id = 9fb4b2e4-957e-41cd-8792-acd04f7714b6 2009-feb-19 03:42:36 info 10.16.64.106:22886(READY) member update: 10.16.64.106:22886(member) 2009-feb-19 03:42:36 notice 10.16.64.106:22886(READY) first in cluster [1]+ Stopped qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pesc430-03 _oat]# bg [1]+ qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 & [root@dell-pesc430-03 _oat]# !ps ps auxw | grep qpidd root 22886 0.8 1.0 165632 10864 pts/0 Sl 03:42 0:00 qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 root 22894 0.0 0.0 61180 732 pts/0 S+ 03:42 0:00 grep qpidd [root@dell-pesc430-03 _oat]# !nets netstat -nlp | grep qpidd tcp 0 0 0.0.0.0:55328 0.0.0.0:* LISTEN 22886/qpidd [root@dell-pesc430-03 _oat]# kill %1 2009-feb-19 03:43:01 notice 10.16.64.106:22886(READY) shutting down [root@dell-pesc430-03 _oat]# 2009-feb-19 03:43:01 debug Shutting down CPG 2009-feb-19 03:43:01 notice Shut down 2009-feb-19 03:43:01 debug Journal "TplStore": Destroyed [1]+ Done qpidd --mgmt-enable no --auth no --log-enable debug+ --cluster-name qpid_mnode_cluster_test --data-dir data-test -p 0 [root@dell-pesc430-03 _oat]# // eof
Hello Daniel, Could you possibly validate whether below pasted qpidd selinux policy looks good to you (make sense to you). I verified that this policy allows the qpidd start with openais-0.80-1/2 in SELinux enforce mode. [root@dell-pe2850-01 _oat]# cat qpiddselinuxpolicy5.te module qpiddselinuxpolicy5 1.0; require { type unconfined_t; type ccs_t; type tmpfs_t; class sem { unix_read read write unix_write associate }; class shm { unix_read read write unix_write associate }; class file { read write }; } #============= ccs_t ============== allow ccs_t tmpfs_t:file { read write }; allow ccs_t unconfined_t:sem { unix_read read write unix_write associate }; allow ccs_t unconfined_t:shm { unix_read read write unix_write associate }; Thanks, Frantisek
Corrected data for review are here: [root@dell-pe2850-01 _oat]# cat qpidd.te module qpidd 1.0; require { type initrc_t; type ccs_t; type tmpfs_t; class sem { unix_read read write unix_write associate }; class shm { unix_read read write unix_write associate }; class file { read write }; } #============= ccs_t ============== allow ccs_t tmpfs_t:file { read write }; allow ccs_t initrc_t:sem { unix_read read write unix_write associate }; allow ccs_t initrc_t:shm { unix_read read write unix_write associate }; I can see expected behavior on RHEL 5.3 i386 at the moment. Going to go through x86_64 (and RHEL5.2 i386/x86_64)
This looks good, this bug should be reassigned to another package though.
I retested on all used platforms (RHEL 5.2/3 i[36]86/x86_64) for MRG and got to a bit extended one. Could you possibly review once more Daniel, please? module qpidd 1.0; require { type initrc_t; type ccs_t; type tmpfs_t; class sem { unix_read read write unix_write associate }; class shm { unix_read read write unix_write associate }; class file { read write }; class capability ipc_owner; } #============= ccs_t ============== allow ccs_t tmpfs_t:file { read write }; allow ccs_t initrc_t:sem { unix_read read write unix_write associate }; allow ccs_t initrc_t:shm { unix_read read write unix_write associate }; allow ccs_t self:capability ipc_owner; Thanks in advance, Frantisek P.S. selinux policy from #9 works well on RHEL 5.3 i386 only / the other architectures needs 'allow ccs_t self:capability ipc_owner;' rule in addition.
Looks good. Try this policy in stead poliyc_module(qpidd, 1.0) require { type initrc_t; type ccs_t; } #============= ccs_t ============== fs_rw_tmpfs_files(ccs_t) allow ccs_t initrc_t:sem rw_sem_perms; allow ccs_t initrc_t:shm rw_shm_perms; allow ccs_t self:capability ipc_owner; I would like to start writing real qpidd policy soon for RHEL6/F-11
yes, tested with your updated one on RHEL 5.3 i386 and x86_64 and both work after correct qpidd startup I can see: [root@dell380-2 _oat]# audit2allow -a -l -e #============= ccs_t ============== # audit(1235141689.430:117): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:initrc_t:s0" # class="sem" perms="destroy" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235141689.430:117): avc: denied { destroy } for # pid=23092 comm="aisexec" key=1164666434 scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:initrc_t:s0 tclass=sem" allow ccs_t initrc_t:sem destroy; # audit(1235141689.430:116): # scontext="root:system_r:ccs_t:s0" tcontext="root:system_r:initrc_t:s0" # class="shm" perms="destroy" # comm="aisexec" exe="" path="" # message="type=AVC msg=audit(1235141689.430:116): avc: denied { destroy } for # pid=23092 comm="aisexec" key=563006265 scontext=root:system_r:ccs_t:s0 # tcontext=root:system_r:initrc_t:s0 tclass=shm"
Change the rw_*perms to create_*_perms; Does aisexec run as a daemon?
The init.d/openais script runs it with the daemon() function from /etc/init.d/functions Also note the hang only occurs when aisexec is started via the init script. If aisexec is started directly from the command line or a script there's no problem (which is why we didn't detect this earlier)
I wonder if /etc/init.d/openais runs before qpid are the avcs different non existant. The avcs are related to whom created the shm and sem, if they are created by openais they would be labeled ccs_t, rather then initrc_t and I believe this would just work. But since qpid is creating them they are labeled initrc_t, and ccs_t is not allowed access. Running an app directly from unconfined_t will not transition while running it from a script labeled initrc_exec_t will. This is why you see a difference between running it via init versus running it directly.
Fixed typo policy_module(qpidd, 1.0) require { type initrc_t; type ccs_t; } #============= ccs_t ============== fs_rw_tmpfs_files(ccs_t) allow ccs_t initrc_t:sem rw_sem_perms; allow ccs_t initrc_t:shm rw_shm_perms; allow ccs_t self:capability ipc_owner; You can compile this using make -f /usr/share/selinux/devel/Makefile And ship the pp file, which you want to install in the post install semodule -i qpidd.pp Additional Info http://danwalsh.livejournal.com/24750.html
The new policy is now packaged with the latest rpm in distcvs.
Tested against 0.4.747892-1.el5.x86_64.rpms (https://brewweb.devel.redhat.com/taskinfo?taskID=1705562) and the current candidate release (0.4.744917-1.el5) rpms. Reproduction: - Make sure SELinux is Enforced (getenforce) - Upgrade openais to openais-0.80.5-2.el5 - Configure openais (/etc/ais/openais.conf) and a corresponding network interface. - Configure qpidd (/etc/qpidd.conf). Add cluster-name=foo - Test 1: Try to start qpidd from command line (qpidd -t --cluster-name=foo) - Test 2: Try to start qpidd via service (service qpidd start) For the candidate release, both tests failed due to qpidd hanging. For the brewbuilds qpidd starts perfectly well via the service command (Test 2) but hangs when starting via command line (Test 1). This is the correct behavior, since the SELinux context will be different when started from command line and via service/init rc.
I have encountered this bug on a RHEL5.3 x86_64 clustered node and as I'm "just" using the cluster suite but not qpid, this problem does not seem exclusive to qpid to me. In my situation it manifests as hanging on "Starting fencing..." on starting the cman service.
Fixed and verified wrt qpid; closing.