Bug 1428024
| Summary: | openstack-ironic-inspector-dnsmasq.service fails to start after undercloud reboot | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yurii Prokulevych <yprokule> | |
| Component: | openstack-ironic-inspector | Assignee: | Dmitry Tantsur <dtantsur> | |
| Status: | CLOSED ERRATA | QA Contact: | mlammon | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 11.0 (Ocata) | CC: | athomas, dtantsur, jschluet, lbezdick, mburns, mcornea, rhel-osp-director-maint, slinaber, srevivo | |
| Target Milestone: | rc | Keywords: | Triaged | |
| Target Release: | 11.0 (Ocata) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-ironic-inspector-5.0.0-3.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1452610 1452615 (view as bug list) | Environment: | ||
| Last Closed: | 2017-05-17 20:03:54 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1452610, 1452615 | |||
We get the same behavior on new Ocata installations, with out and upgrade or updates. + some packages information: [root@undercloud-0 ~]# rpm -qa | grep ironic python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch openstack-ironic-common-7.0.1-0.20170301202959.91540cd.el7ost.noarch puppet-ironic-10.3.0-1.el7ost.noarch python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch openstack-ironic-api-7.0.1-0.20170301202959.91540cd.el7ost.noarch openstack-ironic-conductor-7.0.1-0.20170301202959.91540cd.el7ost.noarch python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch [root@undercloud-0 ~]# rpm -qa | grep dns ldns-1.6.16-10.el7.x86_64 dnsmasq-2.66-21.el7.x86_64 dnsmasq-utils-2.66-21.el7.x86_64 python-dns-1.12.0-2.20150617git465785f.el7.noarch Apr 03 12:47:36 nuc0-undercloud.nucstack network[1161]: Bringing up interface br-ctlplane: [ OK ] Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: FAILED to start up Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: unknown interface br-ctlplane Apr 03 12:47:25 nuc0-undercloud.nucstack dnsmasq[1093]: dnsmasq: unknown interface br-ctlplane Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch. Apr 03 12:47:25 nuc0-undercloud.nucstack nm-dispatcher[964]: req:3 'connectivity-change': start running ordered scripts... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Starting Open vSwitch... Apr 03 12:47:25 nuc0-undercloud.nucstack systemd[1]: Started Open vSwitch Forwarding Unit. Apr 03 12:47:25 nuc0-undercloud.nucstack ovs-ctl[1004]: Enabling remote OVSDB managers [ OK ] In short inspector wanted br-ctlplane but it wasn't there yet as network service started later. Changing /etc/systemd/system/openstack-ironic-inspector-dnsmasq.service to After=network.target fixes the issue - packaging issue RDO master patch proposed: https://review.rdoproject.org/r/#/c/6133/. Could you please check if it's correct? proposed backport to ocata-rdo branch Verified:
Environment: openstack-ironic-inspector
The reported issue no longer reproduces. See below
[stack@undercloud-0 ~]$ rpm -qa | grep openstack-ironic-inspector
openstack-ironic-inspector-5.0.0-3.el7ost.noarch
1. deployed latest octata (11)
2. check current status of service openstack-ironic-inspector-dnsmasq
[root@undercloud-0 ~]# su - stack
Last login: Tue Apr 25 17:59:18 EDT 2017 on pts/0
[stack@undercloud-0 ~]$ . stackrc
[stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-04-25 16:41:01 EDT; 1h 19min ago
Main PID: 28168 (dnsmasq)
CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
└─28168 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf
3. reboot undercloud-0 domain from hypervisor
[root@sealusa3 ~]# virsh reboot undercloud-0
Domain undercloud-0 is being rebooted
[root@sealusa3 ~]# virsh console undercloud-0
Connected to domain undercloud-0
Escape character is ^]
[ OK ] Stopped OpenStack ceilometer notification agent.
[ OK ] Stopped OpenStack Nova Compute Server.
[ OK ] Stopped OpenStack Neutron DHCP Agent.
.
.
.
Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64
undercloud-0 login: [ 25.516311] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[ 25.557248] device tap2e883534-88 left promiscuous mode
[ 128.124606] IPVS: Creating netns size=2040 id=2
[ 128.245378] device tap2e883534-88 entered promiscuous mode
Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel 3.10.0-514.16.1.el7.x86_64 on an x86_64
4. check service after reboot
[stack@undercloud-0 ~]$ systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-04-25 18:03:47 EDT; 4min 14s ago
Process: 1420 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS)
Main PID: 1429 (dnsmasq)
CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
└─1429 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf
Specific Environment Verified: ^^^^ openstack-ironic-inspector-5.0.0-3.el7ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245 |
Description of problem: ----------------------- Minor update of RHOS-11 undercloud. After it's rebooted openstack-ironic-inspector-dnsmasq.service fails to start: systemctl status openstack-ironic-inspector-dnsmasq.service ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2017-03-01 12:08:33 EST; 17min ago Process: 838 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=2) Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Mar 01 12:08:33 undercloud-0.redhat.local dnsmasq[838]: dnsmasq: unknown interface br-ctlplane Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Mar 01 12:08:33 undercloud-0.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. Starting it manually works without problems Version-Release number of selected component (if applicable): ------------------------------------------------------------- puppet-ironic-10.3.0-0.20170218234537.6e78d67.el7ost.noarch python-ironic-inspector-client-1.11.0-0.20170208193115.481a92e.el7ost.noarch python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch openstack-ironic-conductor-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch openstack-ironic-inspector-5.0.1-0.20170214181727.babc2b6.el7ost.noarch openstack-ironic-common-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch openstack-ironic-api-7.0.1-0.20170223011831.a13ea4f.el7ost.noarch python-ironicclient-1.11.0-0.20170208194603.f1f10cb.el7ost.noarch Steps to Reproduce: ------------------- 1. Install RHOS-11 (2017-02-24.2) 2. Setup latest repos 3. Update undercloud: openstack undercloud upgrade 4. Reboot undercloud 5. Assert service's status Additional info: ---------------- Virtual setup: 3controller + 1compute + 3ceph