Description of problem: ----------------------- Minor update of RHOS-11 undercloud. After it's rebooted openstack-ironic-inspector-dnsmasq.service fails to start: systemctl status openstack-ironic-inspector-dnsmasq.service ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-03-01 12:08:33 EST; 17min ago Process: 838 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=2) Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Mar 01 12:08:33 undercloud-0.redhat.local dnsmasq[838]: dnsmasq: unknown interface br-ctlplane Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. Starting it manually works without problems Version-Release number of selected component (if applicable): ------------------------------------------------------------- puppet-ironic-10.3.0-0.20170218234537.6e78d67.el7ost.noarch python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch openstack-ironic-conductor-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch openstack-ironic-common-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch openstack-ironic-api-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch Steps to Reproduce: ------------------- 1. Install RHOS-11 (2017-02-24.2) 2. Setup latest repos 3. Update undercloud: openstack undercloud upgrade 4. Reboot undercloud 5. Assert service's status Additional info: ---------------- Virtual setup: 3controller + 1compute + 3ceph
We get the same behavior on new Ocata installations, with out and upgrade or updates. + some packages information: [root@undercloud-0 ~]# rpm -qa | grep ironic python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch openstack-ironic-common-7.0.1-0.20170301202959.91540cd.el7ost.noarch puppet-ironic-10.3.0-1.el7ost.noarch python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch openstack-ironic-api-7.0.1-0.20170301202959.91540cd.el7ost.noarch openstack-ironic-conductor-7.0.1-0.20170301202959.91540cd.el7ost.noarch python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch [root@undercloud-0 ~]# rpm -qa | grep dns ldns-1.6.16-10.el7.x86_64 dnsmasq-2.66-21.el7.x86_64 dnsmasq-utils-2.66-21.el7.x86_64 python-dns-1.12.0-2.20150617git465785f.el7.noarch
Apr 03 12:47:36 nuc0-undercloud.nucstack network[1161]: Bringing up interface br-ctlplane: [ OK ] Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: FAILED to start up Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: unknown interface br-ctlplane Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: dnsmasq: unknown interface br-ctlplane Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch. Apr 03 12:47:25 nuc0-undercloud.nucstack nm-dispatcher[964]: req:3 'connectivity-change': start running ordered scripts... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting Open vSwitch... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch Forwarding Unit. Apr 03 12:47:25 nuc0-undercloud.nucstack ovs-ctl[1004]: Enabling remote OVSDB managers [ OK ] In short inspector wanted br-ctlplane but it wasn't there yet as network service started later. Changing /etc/systemd/system/openstack-ironic-inspector-dnsmasq.service to After=network.target fixes the issue - packaging issue
RDO master patch proposed: https://review.rdoproject.org/r/#/c/6133/. Could you please check if it's correct?
proposed backport to ocata-rdo branch
Verified: Environment: openstack-ironic-inspector The reported issue no longer reproduces. See below [stack@undercloud-0 ~]$ rpm -qa | grep openstack-ironic-inspector openstack-ironic-inspector-5.0.0-3.el7ost.noarch 1. deployed latest octata (11) 2. check current status of service openstack-ironic-inspector-dnsmasq [root@undercloud-0 ~]# su - stack Last login: Tue Apr 25 17:59:18 EDT 2017 on pts/0 [stack@undercloud-0 ~]$ . stackrc [stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-04-25 16:41:01 EDT; 1h 19min ago Main PID: 28168 (dnsmasq) CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service └─28168 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf 3. reboot undercloud-0 domain from hypervisor [root@sealusa3 ~]# virsh reboot undercloud-0 Domain undercloud-0 is being rebooted [root@sealusa3 ~]# virsh console undercloud-0 Connected to domain undercloud-0 Escape character is ^] [ OK ] Stopped OpenStack ceilometer notification agent. [ OK ] Stopped OpenStack Nova Compute Server. [ OK ] Stopped OpenStack Neutron DHCP Agent. . . . Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64 undercloud-0 login: [ 25.516311] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready [ 25.557248] device tap2e883534-88 left promiscuous mode [ 128.124606] IPVS: Creating netns size=2040 id=2 [ 128.245378] device tap2e883534-88 entered promiscuous mode Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64 4. check service after reboot [stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-04-25 18:03:47 EDT; 4min 14s ago Process: 1420 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS) Main PID: 1429 (dnsmasq) CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service └─1429 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf
Specific Environment Verified: ^^^^ openstack-ironic-inspector-5.0.0-3.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245