1608829 – ilo-pxe could not be loaded, [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'

Bug 1608829 - ilo-pxe could not be loaded, [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'

Summary: ilo-pxe could not be loaded, [Errno 13] Permission denied: '/var/lib/ironic/h...

Keywords:
Status:	CLOSED DUPLICATE of bug 1608529
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-ironic
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	mlammon
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-26 11:09 UTC by Filip Hubík
Modified:	2018-08-03 18:36 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-07-26 12:07:36 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
/var/log/containers/ironic/ironic-conductor.log (2.26 MB, text/plain) 2018-07-26 11:09 UTC, Filip Hubík	no flags	Details
View All

Description Filip Hubík 2018-07-26 11:09:42 UTC

Created attachment 1470698 [details]
/var/log/containers/ironic/ironic-conductor.log

Description of problem:

Overcloud nodes are stuck in BUILD state during overcloud deployment:

+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 850c0ce0-5b9b-45d9-8477-4487daad956a | compute-0    | BUILD  | ctlplane=192.168.24.11 | overcloud-full | compute    |
| b61714a5-3fdc-45c5-b65b-a54738777e82 | controller-0 | BUILD  | ctlplane=192.168.24.12 | overcloud-full | controller |
| d6050da7-f790-4228-b627-89be1c8f9c6a | ceph-0       | BUILD  | ctlplane=192.168.24.7  | overcloud-full | ceph       |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+

/var/log/containers/ironic/ironic-conductor.log reports error:

DEBUG oslo_concurrency.lockutils [req-f4ccfb57-5c95-4cc4-b09a-a5b15c8c7050 - - - - -] Lock "extension_manager" released by "ironic.common.driver_factory._init_extension_manager" :: held 0.011s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
ERROR oslo_service.service [req-f4ccfb57-5c95-4cc4-b09a-a5b15c8c7050 - - - - -] Error starting thread.: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'.
ERROR oslo_service.service Traceback (most recent call last):
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 792, in run_service
ERROR oslo_service.service     service.start()
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/rpc_service.py", line 54, in start
ERROR oslo_service.service     self.manager.init_host(admin_context)
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/conductor/base_manager.py", line 140, in init_host
ERROR oslo_service.service     self._collect_periodic_tasks(admin_context)
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/conductor/base_manager.py", line 268, in _collect_periodic_tasks
ERROR oslo_service.service     for ifaces in driver_factory.all_interfaces().values():
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 271, in all_interfaces
ERROR oslo_service.service     return {iface: interfaces(iface) for iface in _INTERFACE_LOADERS}
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 271, in <dictcomp>
ERROR oslo_service.service     return {iface: interfaces(iface) for iface in _INTERFACE_LOADERS}
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 262, in interfaces
ERROR oslo_service.service     return _get_all_drivers(_INTERFACE_LOADERS[interface_type]())
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 323, in __init__
ERROR oslo_service.service     self.__class__._init_extension_manager()
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
ERROR oslo_service.service     return f(*args, **kwargs)
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 393, in _init_extension_manager
ERROR oslo_service.service     on_missing_entrypoints_callback=missing_callback))
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__
ERROR oslo_service.service     verify_requirements)
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins
ERROR oslo_service.service     self._on_load_failure_callback(self, ep, err)
ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 378, in _catch_driver_not_found
ERROR oslo_service.service     raise exception.DriverLoadError(driver=ep.name, reason=exc)
ERROR oslo_service.service DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'.
ERROR oslo_service.service

$ ls -lahZ /var/lib/ironic/httpboot/
drwxr-xr-x. 42422 42422 unconfined_u:object_r:var_lib_t:s0 .
drwxr-xr-x. 42422 42422 unconfined_u:object_r:var_lib_t:s0 ..
drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0   2e005e19-b2ef-4ee5-b9f2-99d62856b328
drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0   78e7549a-8020-49b6-9679-0fd1e630bce2
-rwxr-xr-x. root  root  unconfined_u:object_r:var_lib_t:s0 agent.kernel
-rw-r--r--. root  root  unconfined_u:object_r:var_lib_t:s0 agent.ramdisk
drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0   b5db2f3c-42b0-4d4d-9261-c332ce30f329
-rw-r--r--. 42422 42422 system_u:object_r:var_lib_t:s0   boot.ipxe
-rw-r--r--. 42422 42422 system_u:object_r:var_lib_t:s0   inspector.ipxe
drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0   pxelinux.cfg

$ cat /var/log/audit/audit.log | grep -i denied
type=AVC msg=audit(1532596659.036:616): avc:  denied  { net_broadcast } for  pid=13715 comm="ovs-vswitchd" capability=11  scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability

Version-Release number of selected component (if applicable):
OSP14, containerized UC+OC, openstack-tripleo-heat-templates-9.0.0-0.20180717094150.el7ost.noarch (built 2018-07-25 18:08:27), puddle 2018-07-25.2

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP14 1:1:1:1 using InfraRed
2. OC deploy step is stuck until times out

Additional info:
python2-ironic-inspector-client-3.2.0-0.20180529111245.6048afc.el7ost.noarch
puppet-ironic-13.1.1-0.20180716160409.d8195a7.el7ost.noarch
python2-ironicclient-2.4.0-0.20180626143850.a49938d.el7ost.noarch
openstack-tripleo-heat-templates-9.0.0-0.20180717094150.el7ost.noarch

Comment 1 Filip Hubík 2018-07-26 11:25:59 UTC

As same as per https://bugzilla.redhat.com/show_bug.cgi?id=1608529 , possibly related upstream patch seems to be present in deployment - https://code.engineering.redhat.com/gerrit/#/c/145094/ .

Comment 3 Bogdan Dobrelya 2018-07-26 12:07:36 UTC


*** This bug has been marked as a duplicate of bug 1608529 ***

Note You need to log in before you can comment on or make changes to this bug.