Bug 1284672

Summary: /usr/bin/clustercheck returned 1 instead of one of 0
Product: Red Hat OpenStack Reporter: Derek Higgins <derekh>
Component: openstack-selinuxAssignee: Ryan Hallisey <rhallise>
Status: CLOSED WORKSFORME QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: lhh, mburns, mgrepl, rhel-osp-director-maint, srevivo, yeylon
Target Milestone: z3Keywords: ZStream
Target Release: 7.0 (Kilo)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-10 15:06:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Derek Higgins 2015-11-23 21:01:42 UTC
While trying to deploy an overcloud (on virt), I get a failed deployment with the following errors in the os-collect-config logs

Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: [2015-11-23 13:13:54,276] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: /Stage[main]/Main/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: [2015-11-23 13:13:54,277] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/aca4a8ed-6613-421f-83dc-93833673886a.pp. [6]



I also see a number of selinux AVC's

type=AVC msg=audit(1448300161.461:81): avc:  denied  { read } for  pid=572 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.484:82): avc:  denied  { read } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.484:83): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.505:84): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.587:85): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300609.868:128): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8777 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:129): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8776 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:130): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9292 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:glance_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:131): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9191 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:glance_registry_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:132): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=1993 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:133): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8004 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:134): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8000 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:soundd_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:135): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8003 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:136): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=3306 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:137): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9696 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:neutron_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:138): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8773 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:139): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8775 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:neutron_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:140): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=6080 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:geneve_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:141): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8774 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:osapi_compute_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:142): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=6379 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:redis_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file
type=AVC msg=audit(1448300621.547:157): avc:  denied  { read } for  pid=31659 comm="mysqld_safe" name="cores" dev="sda2" ino=26693278 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:object_r:cluster_var_lib_t:s0 tclass=dir


package versions

dnsmasq-2.66-14.el7_1.x86_64
dnsmasq-utils-2.66-14.el7_1.x86_64
galera-25.3.5-7.el7ost.x86_64
haproxy-1.5.14-3.el7.x86_64
libselinux-2.2.2-6.el7.x86_64
libselinux-devel-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
libselinux-ruby-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
mariadb-5.5.44-2.el7.x86_64
mariadb-devel-5.5.44-2.el7.x86_64
mariadb-galera-common-5.5.42-1.el7ost.x86_64
mariadb-galera-server-5.5.42-1.el7ost.x86_64
mariadb-libs-5.5.44-2.el7.x86_64
openstack-neutron-2015.1.2-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.1.38-1.el7ost.noarch
openstack-neutron-common-2015.1.2-2.el7ost.noarch
openstack-neutron-lbaas-2015.1.2-1.el7ost.noarch
openstack-neutron-metering-agent-2015.1.2-2.el7ost.noarch
openstack-neutron-ml2-2015.1.2-2.el7ost.noarch
openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch
openstack-puppet-modules-2015.1.8-30.el7ost.noarch
openstack-selinux-0.6.46-1.el7ost.noarch
python-neutron-2015.1.2-2.el7ost.noarch
python-neutronclient-2.4.0-2.el7ost.noarch
python-neutron-lbaas-2015.1.2-1.el7ost.noarch
selinux-policy-3.13.1-60.el7.noarch
selinux-policy-targeted-3.13.1-60.el7.noarch

openstack-tripleo-heat-templates-0.8.6-81.el7ost.noarch

Comment 1 Ryan Hallisey 2015-11-23 21:18:46 UTC
These are all fixed with the 'haproxy_connect_any' bool which is already turned on.  Verify that with `getsebool haproxy_connect_any`.

allow haproxy_t geneve_port_t:tcp_socket name_bind;
allow haproxy_t glance_port_t:tcp_socket name_bind;
allow haproxy_t glance_registry_port_t:tcp_socket name_bind;
allow haproxy_t mysqld_port_t:tcp_socket name_bind;
allow haproxy_t neutron_port_t:tcp_socket name_bind;
allow haproxy_t osapi_compute_port_t:tcp_socket name_bind;
allow haproxy_t redis_port_t:tcp_socket name_bind;
allow haproxy_t soundd_port_t:tcp_socket name_bind;
allow haproxy_t unreserved_port_t:tcp_socket name_bind;

These don't look related.
allow NetworkManager_t var_run_t:file read;
allow dhcpc_t var_run_t:file { read write };

These are likely the culprit
allow mysqld_safe_t cluster_tmp_t:file write;
allow mysqld_safe_t cluster_var_lib_t:dir read;

Comment 2 Lon Hohberger 2015-11-24 15:25:18 UTC
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.

type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.

type=AVC msg=audit(1448300621.547:157): avc:  denied  { read } for  pid=31659 comm="mysqld_safe" name="cores" dev="sda2" ino=26693278 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:object_r:cluster_var_lib_t:s0 tclass=dir

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.

Comment 3 Lon Hohberger 2015-11-24 15:25:59 UTC
These AVCs are allowed by this boolean, and 0.6.47 does not change anything behavior-wise:

daemons_enable_cluster_mode --> on

Comment 4 Lon Hohberger 2015-11-24 15:27:17 UTC
These are set to on when openstack-selinux is installed.

Comment 5 Lon Hohberger 2015-11-24 15:30:28 UTC
All of those AVCs except the first handful (which do not appear relevant) are handled by 0.6.46.

Comment 6 Lon Hohberger 2015-11-24 15:31:23 UTC
It's almost as if openstack-selinux wasn't installed, or was installed after the AVCs were generated.

Comment 7 Ryan Hallisey 2015-11-24 15:32:57 UTC
ignore 0.6.47.  I was testing incorrectly.  I don't what the issue here is thought.

Comment 8 Ryan Hallisey 2015-11-24 15:35:10 UTC
type=AVC msg=audit(1448357287.289:77): avc:  denied  { read } for  pid=574 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.289:77): avc:  denied  { open } for  pid=574 comm="NetworkManager" path="/run/dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.290:78): avc:  denied  { getattr } for  pid=574 comm="NetworkManager" path="/run/dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.290:79): avc:  denied  { signal } for  pid=574 comm="NetworkManager" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process
type=AVC msg=audit(1448357287.291:80): avc:  denied  { signull } for  pid=574 comm="NetworkManager" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process
type=AVC msg=audit(1448357287.295:81): avc:  denied  { unlink } for  pid=574 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file


Few others reported by derekh.

Comment 9 Lon Hohberger 2015-11-24 21:32:15 UTC
Something's setting up NetworkManager and/or files incorrectly here.

I do think the AVCs Derek noted are valid - just they would not happen with openstack-selinux properly installed.  I have to wonder if the image has a broken setup or something?

Comment 10 Derek Higgins 2015-12-08 18:49:05 UTC
It was an image I generated myself, if others aren't seeing a problem I'm happy to close this based on the assumption my image may have had problems.