Bug 1312358

Summary:	Introspection fails due to selinux blocking swift objects upload
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	rhosp-director	Assignee:	Angus Thomas <athomas>
Status:	CLOSED WORKSFORME	QA Contact:	Arik Chernetsky <achernet>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	8.0 (Liberty)	CC:	dbecker, dtantsur, jcoufal, mburns, mcornea, morazi, rhel-osp-director-maint, vcojot
Target Milestone:	---	Keywords:	Reopened
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-10-14 14:26:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-02-26 14:20:48 UTC

Description of problem:
Introspection fails due to selinux blocking swift objects upload:

Introspection for UUID 0dd8e783-b9dc-4043-bfba-c762206140cf finished with error: Swift failed to create object inspector_data-0dd8e783-b9dc-4043-bfba-c762206140cf in container ironic-inspector. Error was: Object PUT failed: http://192.0.2.1:8080/v1/AUTH_28644b9f0bb542b49d243182b843c09c/ironic-inspector/inspector_data-0dd8e783-b9dc-4043-bfba-c762206140cf 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

/var/log/audit/audit.log:

type=AVC msg=audit(1456495310.564:3125): avc:  denied  { name_connect } for  pid=27249 comm="swift-object-se" dest=49159 scontext=system_u:system_r:swift_t:s0 tcontext=system_u:object_r:virt_migration_port_t:s0 tclass=tcp_socket


Version-Release number of selected component (if applicable):
Latest OSPd8 puddle
openstack-selinux-0.6.55-1.el7ost.noarch
selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-devel-3.13.1-60.el7_2.3.noarch

How reproducible:
once

Steps to Reproduce:
1. Deploy undercloud
2. Register nodes
3. Run bulk introspection

Actual results:
Introspecion fails 

Expected results:
Introspection succeeds.

Comment 1 Mike Burns 2016-02-26 14:29:42 UTC

Please provide full audit log from a run in permissive mode.

Comment 2 Marius Cornea 2016-02-26 17:09:39 UTC

I wasn't able to reproduce this, tried it several times. If the provided info is not enough to indicate a problem I guess we are good to close this as not a bug.

Comment 3 Vincent S. Cojot 2016-03-23 22:47:30 UTC

Hi everyone,
I'm currently seeing this on the latest puddle (20160318/22) and I -do- have SELinux in permissive mode.. Any hints?

[root@instack ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

[stack@instack ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: d20a5089-904d-403c-81b2-11d850d04a64
Starting introspection of node: 06c3cfca-3df3-41d9-b219-d1172176e15f
Starting introspection of node: fd88ffcf-709b-4a43-ba4d-47cf71be8d0a
Starting introspection of node: cce5b077-87a5-41a0-a84b-e0e1c71cd1a5
Starting introspection of node: 95c6b687-5192-4080-8bd8-2a602e6bd0ee
Starting introspection of node: df76017e-7119-4996-bc72-e373fe92be4b
Starting introspection of node: f1516d92-2ae9-4c96-9e71-1ea249f5a83c
Starting introspection of node: 671a5a53-97d1-4c88-b93d-e0a7615a6e6c
Starting introspection of node: a70406b4-982e-4651-8f76-c570e5dfa066
Starting introspection of node: 9e44805f-2d4c-46ff-ad44-6fdaa0d695d0
Starting introspection of node: 56ff00a9-1d78-419c-af79-0780d1791275
Starting introspection of node: df83654d-a4b4-4367-b8b4-b31296cb49da
Waiting for introspection to finish...
Introspection for UUID 06c3cfca-3df3-41d9-b219-d1172176e15f finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID cce5b077-87a5-41a0-a84b-e0e1c71cd1a5 finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID a70406b4-982e-4651-8f76-c570e5dfa066 finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID fd88ffcf-709b-4a43-ba4d-47cf71be8d0a finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID 9e44805f-2d4c-46ff-ad44-6fdaa0d695d0 finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID 95c6b687-5192-4080-8bd8-2a602e6bd0ee finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID f1516d92-2ae9-4c96-9e71-1ea249f5a83c finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID df76017e-7119-4996-bc72-e373fe92be4b finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID 671a5a53-97d1-4c88-b93d-e0a7615a6e6c finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID 56ff00a9-1d78-419c-af79-0780d1791275 finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently
Introspection for UUID df83654d-a4b4-4367-b8b4-b31296cb49da finished with error: Swift failed to create container ironic-inspector. Error was: Container PUT failed: http://10.20.0.2:8080/v1/AUTH_a2491c4cc5184f5a96daea9ebc861b79/ironic-inspector 503 Service Unavailable  [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

Comment 4 Vincent S. Cojot 2016-03-23 22:48:08 UTC

[root@instack ~]# rpm -aq openstack\*|sort
openstack-aodh-api-1.1.2-1.el7ost.noarch
openstack-aodh-common-1.1.2-1.el7ost.noarch
openstack-aodh-evaluator-1.1.2-1.el7ost.noarch
openstack-aodh-listener-1.1.2-1.el7ost.noarch
openstack-aodh-notifier-1.1.2-1.el7ost.noarch
openstack-ceilometer-alarm-5.0.2-2.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-glance-11.0.1-4.el7ost.noarch
openstack-heat-api-5.0.1-3.el7ost.noarch
openstack-heat-api-cfn-5.0.1-3.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-3.el7ost.noarch
openstack-heat-common-5.0.1-3.el7ost.noarch
openstack-heat-engine-5.0.1-3.el7ost.noarch
openstack-heat-templates-0-0.1.20151019.el7ost.noarch
openstack-ironic-api-4.2.2-4.el7ost.noarch
openstack-ironic-common-4.2.2-4.el7ost.noarch
openstack-ironic-conductor-4.2.2-4.el7ost.noarch
openstack-ironic-inspector-2.2.5-1.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-neutron-7.0.1-13.el7ost.noarch
openstack-neutron-common-7.0.1-13.el7ost.noarch
openstack-neutron-ml2-7.0.1-13.el7ost.noarch
openstack-neutron-openvswitch-7.0.1-13.el7ost.noarch
openstack-nova-api-12.0.2-2.el7ost.noarch
openstack-nova-cert-12.0.2-2.el7ost.noarch
openstack-nova-common-12.0.2-2.el7ost.noarch
openstack-nova-compute-12.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.2-2.el7ost.noarch
openstack-nova-scheduler-12.0.2-2.el7ost.noarch
openstack-puppet-modules-7.0.15-1.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-tripleo-common-0.3.0-3.el7ost.noarch
openstack-tripleo-heat-templates-0.8.12-2.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.12-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch

Comment 5 Mike Burns 2016-04-07 21:11:06 UTC

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 Dmitry Tantsur 2016-10-14 14:26:35 UTC

Hi! IIRC this was a problem in one of the puddles, that mysteriously disappeared later on. Please reopen this bug if you see it again.