Bug 1468390 - osp10z3 order of operations problem with ironic inspector and br-ctlplane
osp10z3 order of operations problem with ironic inspector and br-ctlplane
Status: CLOSED DUPLICATE of bug 1452610
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: James Slagle
Arik Chernetsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-06 19:08 EDT by Andreas Karis
Modified: 2018-02-06 13:41 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-22 16:36:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sosreport --all-logs (13.18 MB, application/x-xz)
2017-07-21 19:36 EDT, Andreas Karis
no flags Details

  None (edit)
Description Andreas Karis 2017-07-06 19:08:25 EDT
The following happens after the reboot of the undercloud during a minor upgrade. The same happens for any new installation of the undercloud. opesntack-ironic-inspector-dnsmasq fails because br-ctlplane is not up:

ack@director ~]$ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"
  UNIT                                       LOAD   ACTIVE     SUB     JOB   DESCRIPTION
  neutron-dhcp-agent.service                 loaded active     running       OpenStack Neutron DHCP Agent
  neutron-openvswitch-agent.service          loaded active     running       OpenStack Neutron Open vSwitch Agent
  neutron-ovs-cleanup.service                loaded active     exited        OpenStack Neutron Open vSwitch Cleanup Utility
  neutron-server.service                     loaded active     running       OpenStack Neutron Server
  openstack-aodh-evaluator.service           loaded active     running       OpenStack Alarm evaluator service
  openstack-aodh-listener.service            loaded active     running       OpenStack Alarm listener service
  openstack-aodh-notifier.service            loaded active     running       OpenStack Alarm notifier service
  openstack-ceilometer-central.service       loaded active     running       OpenStack ceilometer central agent
  openstack-ceilometer-collector.service     loaded active     running       OpenStack ceilometer collection service
  openstack-ceilometer-notification.service  loaded active     running       OpenStack ceilometer notification agent
  openstack-glance-api.service               loaded active     running       OpenStack Image Service (code-named Glance) API server
  openstack-glance-registry.service          loaded active     running       OpenStack Image Service (code-named Glance) Registry server
  openstack-heat-api-cfn.service             loaded active     running       Openstack Heat CFN-compatible API Service
  openstack-heat-api.service                 loaded active     running       OpenStack Heat API Service
  openstack-heat-engine.service              loaded active     running       Openstack Heat Engine Service
  openstack-ironic-api.service               loaded active     running       OpenStack Ironic API service
  openstack-ironic-conductor.service         loaded active     running       OpenStack Ironic Conductor service
● openstack-ironic-inspector-dnsmasq.service loaded failed     failed        PXE boot dnsmasq service for Ironic Inspector
  openstack-ironic-inspector.service         loaded active     running       Hardware introspection service for OpenStack Ironic
  openstack-mistral-api.service              loaded active     running       Mistral API Server
  openstack-mistral-engine.service           loaded active     running       Mistral Engine Server
  openstack-mistral-executor.service         loaded active     running       Mistral Executor Server
  openstack-nova-api.service                 loaded active     running       OpenStack Nova API Server
  openstack-nova-cert.service                loaded active     running       OpenStack Nova Cert Server
  openstack-nova-compute.service             loaded activating start   start OpenStack Nova Compute Server
  openstack-nova-conductor.service           loaded active     running       OpenStack Nova Conductor Server
  openstack-nova-scheduler.service           loaded active     running       OpenStack Nova Scheduler Server
  openstack-swift-account-reaper.service     loaded active     running       OpenStack Object Storage (swift) - Account Reaper
  openstack-swift-account.service            loaded active     running       OpenStack Object Storage (swift) - Account Server
  openstack-swift-container-updater.service  loaded active     running       OpenStack Object Storage (swift) - Container Updater
  openstack-swift-container.service          loaded active     running       OpenStack Object Storage (swift) - Container Server
  openstack-swift-object-updater.service     loaded active     running       OpenStack Object Storage (swift) - Object Updater
  openstack-swift-object.service             loaded active     running       OpenStack Object Storage (swift) - Object Server
  openstack-swift-proxy.service              loaded active     running       OpenStack Object Storage (swift) - Proxy Server
  openstack-zaqar.service                    loaded active     running       OpenStack Message Queuing Service (code-named Zaqar) Server
  openstack-zaqar@1.service                  loaded active     running       OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
  openvswitch.service                        loaded active     exited        Open vSwitch

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

37 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[stack@director ~]$ sudo systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-07-06 19:51:14 EDT; 1min 27s ago
  Process: 1443 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=2)

Jul 06 19:51:14 director.customer.poc systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 06 19:51:14 director.customer.poc dnsmasq[1443]: dnsmasq: unknown interface br-ctlplane
Jul 06 19:51:14 director.customer.poc systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Jul 06 19:51:14 director.customer.poc systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Jul 06 19:51:14 director.customer.poc systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Jul 06 19:51:14 director.customer.poc systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
[stack@director ~]$ sudo systemctl restart openstack-ironic-inspector-dnsmasq
[stack@director ~]$ sudo systemctl status openstack-ironic-inspector-dnsmasq
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-07-06 19:53:06 EDT; 5s ago
  Process: 15348 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS)
 Main PID: 15350 (dnsmasq)
   CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service
           └─15350 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf

Jul 06 19:53:06 director.customer.poc systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 06 19:53:06 director.customer.poc dnsmasq[15350]: started, version 2.66 DNS disabled
Jul 06 19:53:06 director.customer.poc dnsmasq[15350]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Jul 06 19:53:06 director.customer.poc dnsmasq-dhcp[15350]: DHCP, IP range 10.10.159.150 -- 10.10.159.239, lease time 2m
Jul 06 19:53:06 director.customer.poc systemd[1]: Started PXE boot dnsmasq service for Ironic Inspector.
Comment 1 Bob Fournier 2017-07-21 14:30:47 EDT
Andreas - any chance of getting an sosreport? Is one available through the referenced support ticket that would show this? Thanks.
Comment 2 Andreas Karis 2017-07-21 14:33:57 EDT
Hi Bob,

I think all new installations with OSP 10z3 show this issue, at least all that I saw. I'm redeploying my lab until I get to this stage and will provide a sosreport.

- Andreas
Comment 3 Andreas Karis 2017-07-21 19:33:17 EDT
[akaris@wks-akaris ~]$ ssh root@10.10.179.85
Last login: Fri Jul 21 19:30:04 2017 from 10.17.17.204
################################### 
akaris' reproducer 
###################################
[root@rhospbl-4 ~]# ssh undercloud
Last login: Fri Jul 21 19:30:14 2017 from uchost.default.redhat.local
[root@undercloud-4 ~]# system^C
[root@undercloud-4 ~]# ^C
[root@undercloud-4 ~]# systemctl list-units | grep fail
● docker-storage-setup.service                                                loaded failed failed    Docker Storage Setup
● NetworkManager-wait-online.service                                          loaded failed failed    Network Manager Wait Online
● openstack-ironic-inspector-dnsmasq.service                                  loaded failed failed    PXE boot dnsmasq service for Ironic Inspector
● ovirt-guest-agent.service                                                   loaded failed failed    oVirt Guest Agent
[root@undercloud-4 ~]# systemctl status openstack-ironic-inspector-dnsmasq -l
● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-07-21 19:01:05 EDT; 29min ago

Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
[root@undercloud-4 ~]# journalctl -u !$
journalctl -u -l
-- No entries --
[root@undercloud-4 ~]# journalctl -u openstack-ironic-inspector-dnsmasq
-- Logs begin at Fri 2017-07-21 18:14:52 EDT, end at Fri 2017-07-21 19:31:07 EDT. --
Jul 21 18:46:06 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq[20664]: started, version 2.66 DNS disabled
Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq[20664]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq-dhcp[20664]: DHCP, IP range 192.0.2.100 -- 192.0.2.120, lease time 2m
Jul 21 18:46:06 undercloud-4.default.redhat.local systemd[1]: Started PXE boot dnsmasq service for Ironic Inspector.
Jul 21 18:51:32 undercloud-4.default.redhat.local systemd[1]: Stopping PXE boot dnsmasq service for Ironic Inspector...
Jul 21 18:51:32 undercloud-4.default.redhat.local systemd[1]: Stopped PXE boot dnsmasq service for Ironic Inspector.
-- Reboot --
Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: dnsmasq: unknown interface br-ctlplane
Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
-- Reboot --
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector...
Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector.
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state.
Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed.
Comment 4 Andreas Karis 2017-07-21 19:33:48 EDT
[root@undercloud-4 ~]# journalctl | grep -i br-ctlplane | grep -i unknown | grep -i interfa
Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: dnsmasq: unknown interface br-ctlplane
Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: unknown interface br-ctlplane
Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane
Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: unknown interface br-ctlplane
Comment 5 Andreas Karis 2017-07-21 19:36 EDT
Created attachment 1302568 [details]
sosreport --all-logs
Comment 6 Bob Fournier 2017-07-21 21:55:35 EDT
Thanks for the sospreport Andreas.  It appears that this is a known issue that has been resolved - see https://bugzilla.redhat.com/show_bug.cgi?id=1428024

There is a backport request for OSP-10 here - https://bugzilla.redhat.com/show_bug.cgi?id=1452610

I think it makes sense to mark this as a duplicate 1452610. 

Let me know what you think.
Comment 7 Andreas Karis 2017-07-22 16:36:27 EDT

*** This bug has been marked as a duplicate of bug 1452610 ***
Comment 8 Andreas Karis 2017-07-22 16:37:12 EDT
Thanks!

Note You need to log in before you can comment on or make changes to this bug.