Bug 1428024

Summary: openstack-ironic-inspector-dnsmasq.service fails to start after undercloud reboot
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-ironic-inspectorAssignee: Dmitry Tantsur <dtantsur>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: athomas, dtantsur, jschluet, lbezdick, mburns, mcornea, rhel-osp-director-maint, slinaber, srevivo
Target Milestone: rcKeywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-inspector-5.0.0-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1452610 1452615 (view as bug list) Environment:
Last Closed: 2017-05-17 20:03:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1452610, 1452615    

Description Yurii Prokulevych 2017-03-01 17:44:44 UTC
Description of problem:
-----------------------
Minor update of RHOS-11 undercloud. After it's rebooted openstack-ironic-inspector-dnsmasq.service fails to start:

systemctl status openstack-ironic-inspector-dnsmasq.service
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2017-03-01 12:08:33 EST; 17min ago
  Process: 838 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=2)

Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Mar 01 12:08:33 undercloud-0.redhat.local dnsmasq[838]: dnsmasq: unknown interface br-ctlplane
Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.

Starting it manually works without problems


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
puppet-ironic-10.3.0-0.20170218234537.6e78d67.el7ost.noarch
python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch
python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch
openstack-ironic-conductor-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch
openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch
openstack-ironic-common-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch
openstack-ironic-api-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch
python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch


Steps to Reproduce:
-------------------
1. Install RHOS-11 (2017-02-24.2)
2. Setup latest repos
3. Update undercloud:
     openstack undercloud upgrade 
4. Reboot undercloud
5. Assert service's status


Additional info:
----------------
Virtual setup: 3controller + 1compute + 3ceph

Comment 2 Raviv Bar-Tal 2017-03-12 09:58:50 UTC
We get the same behavior on new Ocata installations, with out and upgrade or updates.

+ some packages information:

[root@undercloud-0 ~]# rpm -qa | grep ironic
python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch
openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch
openstack-ironic-common-7.0.1-0.20170301202959.91540cd.el7ost.noarch
puppet-ironic-10.3.0-1.el7ost.noarch
python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch
openstack-ironic-api-7.0.1-0.20170301202959.91540cd.el7ost.noarch
openstack-ironic-conductor-7.0.1-0.20170301202959.91540cd.el7ost.noarch
python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch

[root@undercloud-0 ~]# rpm -qa | grep dns
ldns-1.6.16-10.el7.x86_64
dnsmasq-2.66-21.el7.x86_64
dnsmasq-utils-2.66-21.el7.x86_64
python-dns-1.12.0-2.20150617git465785f.el7.noarch

Comment 3 Lukas Bezdicka 2017-04-03 10:59:05 UTC
Apr 03 12:47:36 nuc0-undercloud.nucstack network[1161]: Bringing up interface br-ctlplane:  [  OK  ]

Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: FAILED to start up
Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: unknown interface br-ctlplane
Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: dnsmasq: unknown interface br-ctlplane
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch.
Apr 03 12:47:25 nuc0-undercloud.nucstack nm-dispatcher[964]: req:3 'connectivity-change': start running ordered scripts...
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting Open vSwitch...
Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch Forwarding Unit.
Apr 03 12:47:25 nuc0-undercloud.nucstack ovs-ctl[1004]: Enabling remote OVSDB managers [  OK  ]


In short inspector wanted br-ctlplane but it wasn't there yet as network service started later.


Changing /etc/systemd/system/openstack-ironic-inspector-dnsmasq.service to After=network.target fixes the issue - packaging issue

Comment 4 Dmitry Tantsur 2017-04-03 13:37:38 UTC
RDO master patch proposed: https://review.rdoproject.org/r/#/c/6133/. Could you please check if it's correct?

Comment 5 Jon Schlueter 2017-04-03 20:45:58 UTC
proposed backport to ocata-rdo branch

Comment 8 mlammon 2017-04-25 22:18:38 UTC
Verified:
Environment:  openstack-ironic-inspector

The reported issue no longer reproduces. See below

[stack@undercloud-0 ~]$ rpm -qa | grep openstack-ironic-inspector
openstack-ironic-inspector-5.0.0-3.el7ost.noarch

1. deployed latest octata (11) 
2. check current status of service openstack-ironic-inspector-dnsmasq

[root@undercloud-0 ~]# su - stack
Last login: Tue Apr 25 17:59:18 EDT 2017 on pts/0
[stack@undercloud-0 ~]$ . stackrc
[stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-04-25 16:41:01 EDT; 1h 19min ago
 Main PID: 28168 (dnsmasq)
   CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
           └─28168 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf

3. reboot undercloud-0 domain from hypervisor
[root@sealusa3 ~]# virsh reboot undercloud-0
Domain undercloud-0 is being rebooted

[root@sealusa3 ~]# virsh console undercloud-0
Connected to domain undercloud-0
Escape character is ^]
[  OK  ] Stopped OpenStack ceilometer notification agent.
[  OK  ] Stopped OpenStack Nova Compute Server.
[  OK  ] Stopped OpenStack Neutron DHCP Agent.
.
.
.
Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64

undercloud-0 login: [   25.516311] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   25.557248] device tap2e883534-88 left promiscuous mode
[  128.124606] IPVS: Creating netns size=2040 id=2
[  128.245378] device tap2e883534-88 entered promiscuous mode

Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64


4. check service after reboot
[stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-04-25 18:03:47 EDT; 4min 14s ago
  Process: 1420 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS)
 Main PID: 1429 (dnsmasq)
   CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
           └─1429 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf

Comment 9 mlammon 2017-04-25 22:22:01 UTC
Specific Environment Verified: ^^^^
openstack-ironic-inspector-5.0.0-3.el7ost.noarch

Comment 10 errata-xmlrpc 2017-05-17 20:03:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245