This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1284672 - /usr/bin/clustercheck returned 1 instead of one of 0
/usr/bin/clustercheck returned 1 instead of one of 0
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux (Show other bugs)
7.0 (Kilo)
x86_64 Linux
unspecified Severity high
: z3
: 7.0 (Kilo)
Assigned To: Ryan Hallisey
yeylon@redhat.com
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-23 16:01 EST by Derek Higgins
Modified: 2016-04-18 02:52 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-10 10:06:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Derek Higgins 2015-11-23 16:01:42 EST
While trying to deploy an overcloud (on virt), I get a failed deployment with the following errors in the os-collect-config logs

Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: [2015-11-23 13:13:54,276] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: /Stage[main]/Main/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov 23 13:13:54 overcloud-controller-0.localdomain os-collect-config[4905]: [2015-11-23 13:13:54,277] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/aca4a8ed-6613-421f-83dc-93833673886a.pp. [6]



I also see a number of selinux AVC's

type=AVC msg=audit(1448300161.461:81): avc:  denied  { read } for  pid=572 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.484:82): avc:  denied  { read } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.484:83): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.505:84): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300161.587:85): avc:  denied  { write } for  pid=10069 comm="dhclient" name="dhclient-br-ex.pid" dev="tmpfs" ino=31679 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448300609.868:128): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8777 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:129): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8776 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:130): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9292 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:glance_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:131): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9191 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:glance_registry_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:132): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=1993 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:133): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8004 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:134): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8000 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:soundd_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:135): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8003 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.868:136): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=3306 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:137): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=9696 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:neutron_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:138): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8773 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:139): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8775 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:neutron_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:140): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=6080 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:geneve_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:141): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=8774 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:osapi_compute_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300609.869:142): avc:  denied  { name_bind } for  pid=30620 comm="haproxy" src=6379 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:redis_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file
type=AVC msg=audit(1448300621.547:157): avc:  denied  { read } for  pid=31659 comm="mysqld_safe" name="cores" dev="sda2" ino=26693278 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:object_r:cluster_var_lib_t:s0 tclass=dir


package versions

dnsmasq-2.66-14.el7_1.x86_64
dnsmasq-utils-2.66-14.el7_1.x86_64
galera-25.3.5-7.el7ost.x86_64
haproxy-1.5.14-3.el7.x86_64
libselinux-2.2.2-6.el7.x86_64
libselinux-devel-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
libselinux-ruby-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
mariadb-5.5.44-2.el7.x86_64
mariadb-devel-5.5.44-2.el7.x86_64
mariadb-galera-common-5.5.42-1.el7ost.x86_64
mariadb-galera-server-5.5.42-1.el7ost.x86_64
mariadb-libs-5.5.44-2.el7.x86_64
openstack-neutron-2015.1.2-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.1.38-1.el7ost.noarch
openstack-neutron-common-2015.1.2-2.el7ost.noarch
openstack-neutron-lbaas-2015.1.2-1.el7ost.noarch
openstack-neutron-metering-agent-2015.1.2-2.el7ost.noarch
openstack-neutron-ml2-2015.1.2-2.el7ost.noarch
openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch
openstack-puppet-modules-2015.1.8-30.el7ost.noarch
openstack-selinux-0.6.46-1.el7ost.noarch
python-neutron-2015.1.2-2.el7ost.noarch
python-neutronclient-2.4.0-2.el7ost.noarch
python-neutron-lbaas-2015.1.2-1.el7ost.noarch
selinux-policy-3.13.1-60.el7.noarch
selinux-policy-targeted-3.13.1-60.el7.noarch

openstack-tripleo-heat-templates-0.8.6-81.el7ost.noarch
Comment 1 Ryan Hallisey 2015-11-23 16:18:46 EST
These are all fixed with the 'haproxy_connect_any' bool which is already turned on.  Verify that with `getsebool haproxy_connect_any`.

allow haproxy_t geneve_port_t:tcp_socket name_bind;
allow haproxy_t glance_port_t:tcp_socket name_bind;
allow haproxy_t glance_registry_port_t:tcp_socket name_bind;
allow haproxy_t mysqld_port_t:tcp_socket name_bind;
allow haproxy_t neutron_port_t:tcp_socket name_bind;
allow haproxy_t osapi_compute_port_t:tcp_socket name_bind;
allow haproxy_t redis_port_t:tcp_socket name_bind;
allow haproxy_t soundd_port_t:tcp_socket name_bind;
allow haproxy_t unreserved_port_t:tcp_socket name_bind;

These don't look related.
allow NetworkManager_t var_run_t:file read;
allow dhcpc_t var_run_t:file { read write };

These are likely the culprit
allow mysqld_safe_t cluster_tmp_t:file write;
allow mysqld_safe_t cluster_var_lib_t:dir read;
Comment 2 Lon Hohberger 2015-11-24 10:25:18 EST
type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.

type=AVC msg=audit(1448300619.049:156): avc:  denied  { write } for  pid=30790 comm="mysqld_safe" path="/tmp/tmp.3eZRnSANSZ" dev="sda2" ino=26429760 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:cluster_tmp_t:s0 tclass=file

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.

type=AVC msg=audit(1448300621.547:157): avc:  denied  { read } for  pid=31659 comm="mysqld_safe" name="cores" dev="sda2" ino=26693278 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:object_r:cluster_var_lib_t:s0 tclass=dir

	Was caused by:
		Unknown - would be allowed by active policy
		Possible mismatch between this policy and the one under which the audit message was generated.

		Possible mismatch between current in-memory boolean settings vs. permanent ones.
Comment 3 Lon Hohberger 2015-11-24 10:25:59 EST
These AVCs are allowed by this boolean, and 0.6.47 does not change anything behavior-wise:

daemons_enable_cluster_mode --> on
Comment 4 Lon Hohberger 2015-11-24 10:27:17 EST
These are set to on when openstack-selinux is installed.
Comment 5 Lon Hohberger 2015-11-24 10:30:28 EST
All of those AVCs except the first handful (which do not appear relevant) are handled by 0.6.46.
Comment 6 Lon Hohberger 2015-11-24 10:31:23 EST
It's almost as if openstack-selinux wasn't installed, or was installed after the AVCs were generated.
Comment 7 Ryan Hallisey 2015-11-24 10:32:57 EST
ignore 0.6.47.  I was testing incorrectly.  I don't what the issue here is thought.
Comment 8 Ryan Hallisey 2015-11-24 10:35:10 EST
type=AVC msg=audit(1448357287.289:77): avc:  denied  { read } for  pid=574 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.289:77): avc:  denied  { open } for  pid=574 comm="NetworkManager" path="/run/dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.290:78): avc:  denied  { getattr } for  pid=574 comm="NetworkManager" path="/run/dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file
type=AVC msg=audit(1448357287.290:79): avc:  denied  { signal } for  pid=574 comm="NetworkManager" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process
type=AVC msg=audit(1448357287.291:80): avc:  denied  { signull } for  pid=574 comm="NetworkManager" scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process
type=AVC msg=audit(1448357287.295:81): avc:  denied  { unlink } for  pid=574 comm="NetworkManager" name="dhclient-br-ex.pid" dev="tmpfs" ino=30502 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file


Few others reported by derekh.
Comment 9 Lon Hohberger 2015-11-24 16:32:15 EST
Something's setting up NetworkManager and/or files incorrectly here.

I do think the AVCs Derek noted are valid - just they would not happen with openstack-selinux properly installed.  I have to wonder if the image has a broken setup or something?
Comment 10 Derek Higgins 2015-12-08 13:49:05 EST
It was an image I generated myself, if others aren't seeing a problem I'm happy to close this based on the assumption my image may have had problems.

Note You need to log in before you can comment on or make changes to this bug.