Bug 1608829
| Summary: | ilo-pxe could not be loaded, [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe' | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Filip Hubík <fhubik> | ||||
| Component: | openstack-ironic | Assignee: | RHOS Maint <rhos-maint> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | mlammon | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 14.0 (Rocky) | CC: | bdobreli, bfournie, mburns, mcornea, srevivo | ||||
| Target Milestone: | --- | Keywords: | Automation, AutomationBlocker | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-07-26 12:07:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
As same as per https://bugzilla.redhat.com/show_bug.cgi?id=1608529 , possibly related upstream patch seems to be present in deployment - https://code.engineering.redhat.com/gerrit/#/c/145094/ . *** This bug has been marked as a duplicate of bug 1608529 *** |
Created attachment 1470698 [details] /var/log/containers/ironic/ironic-conductor.log Description of problem: Overcloud nodes are stuck in BUILD state during overcloud deployment: +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 850c0ce0-5b9b-45d9-8477-4487daad956a | compute-0 | BUILD | ctlplane=192.168.24.11 | overcloud-full | compute | | b61714a5-3fdc-45c5-b65b-a54738777e82 | controller-0 | BUILD | ctlplane=192.168.24.12 | overcloud-full | controller | | d6050da7-f790-4228-b627-89be1c8f9c6a | ceph-0 | BUILD | ctlplane=192.168.24.7 | overcloud-full | ceph | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ /var/log/containers/ironic/ironic-conductor.log reports error: DEBUG oslo_concurrency.lockutils [req-f4ccfb57-5c95-4cc4-b09a-a5b15c8c7050 - - - - -] Lock "extension_manager" released by "ironic.common.driver_factory._init_extension_manager" :: held 0.011s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 ERROR oslo_service.service [req-f4ccfb57-5c95-4cc4-b09a-a5b15c8c7050 - - - - -] Error starting thread.: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'. ERROR oslo_service.service Traceback (most recent call last): ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 792, in run_service ERROR oslo_service.service service.start() ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/rpc_service.py", line 54, in start ERROR oslo_service.service self.manager.init_host(admin_context) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/conductor/base_manager.py", line 140, in init_host ERROR oslo_service.service self._collect_periodic_tasks(admin_context) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/conductor/base_manager.py", line 268, in _collect_periodic_tasks ERROR oslo_service.service for ifaces in driver_factory.all_interfaces().values(): ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 271, in all_interfaces ERROR oslo_service.service return {iface: interfaces(iface) for iface in _INTERFACE_LOADERS} ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 271, in <dictcomp> ERROR oslo_service.service return {iface: interfaces(iface) for iface in _INTERFACE_LOADERS} ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 262, in interfaces ERROR oslo_service.service return _get_all_drivers(_INTERFACE_LOADERS[interface_type]()) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 323, in __init__ ERROR oslo_service.service self.__class__._init_extension_manager() ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner ERROR oslo_service.service return f(*args, **kwargs) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 393, in _init_extension_manager ERROR oslo_service.service on_missing_entrypoints_callback=missing_callback)) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__ ERROR oslo_service.service verify_requirements) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins ERROR oslo_service.service self._on_load_failure_callback(self, ep, err) ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/ironic/common/driver_factory.py", line 378, in _catch_driver_not_found ERROR oslo_service.service raise exception.DriverLoadError(driver=ep.name, reason=exc) ERROR oslo_service.service DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe'. ERROR oslo_service.service $ ls -lahZ /var/lib/ironic/httpboot/ drwxr-xr-x. 42422 42422 unconfined_u:object_r:var_lib_t:s0 . drwxr-xr-x. 42422 42422 unconfined_u:object_r:var_lib_t:s0 .. drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0 2e005e19-b2ef-4ee5-b9f2-99d62856b328 drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0 78e7549a-8020-49b6-9679-0fd1e630bce2 -rwxr-xr-x. root root unconfined_u:object_r:var_lib_t:s0 agent.kernel -rw-r--r--. root root unconfined_u:object_r:var_lib_t:s0 agent.ramdisk drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0 b5db2f3c-42b0-4d4d-9261-c332ce30f329 -rw-r--r--. 42422 42422 system_u:object_r:var_lib_t:s0 boot.ipxe -rw-r--r--. 42422 42422 system_u:object_r:var_lib_t:s0 inspector.ipxe drwxr-xr-x. 42422 42422 system_u:object_r:var_lib_t:s0 pxelinux.cfg $ cat /var/log/audit/audit.log | grep -i denied type=AVC msg=audit(1532596659.036:616): avc: denied { net_broadcast } for pid=13715 comm="ovs-vswitchd" capability=11 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability Version-Release number of selected component (if applicable): OSP14, containerized UC+OC, openstack-tripleo-heat-templates-9.0.0-0.20180717094150.el7ost.noarch (built 2018-07-25 18:08:27), puddle 2018-07-25.2 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP14 1:1:1:1 using InfraRed 2. OC deploy step is stuck until times out Additional info: python2-ironic-inspector-client-3.2.0-0.20180529111245.6048afc.el7ost.noarch puppet-ironic-13.1.1-0.20180716160409.d8195a7.el7ost.noarch python2-ironicclient-2.4.0-0.20180626143850.a49938d.el7ost.noarch openstack-tripleo-heat-templates-9.0.0-0.20180717094150.el7ost.noarch