Bug 1468390
| Summary: | osp10z3 order of operations problem with ironic inspector and br-ctlplane | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Andreas Karis <akaris> | ||||
| Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 10.0 (Newton) | CC: | akaris, akarlsso, aschultz, bfournie, kholden, mburns, mcornea, mlammon, rhel-osp-director-maint | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-07-22 20:36:27 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Andreas - any chance of getting an sosreport? Is one available through the referenced support ticket that would show this? Thanks. Hi Bob, I think all new installations with OSP 10z3 show this issue, at least all that I saw. I'm redeploying my lab until I get to this stage and will provide a sosreport. - Andreas [akaris@wks-akaris ~]$ ssh root.179.85 Last login: Fri Jul 21 19:30:04 2017 from 10.17.17.204 ################################### akaris' reproducer ################################### [root@rhospbl-4 ~]# ssh undercloud Last login: Fri Jul 21 19:30:14 2017 from uchost.default.redhat.local [root@undercloud-4 ~]# system^C [root@undercloud-4 ~]# ^C [root@undercloud-4 ~]# systemctl list-units | grep fail ● docker-storage-setup.service loaded failed failed Docker Storage Setup ● NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online ● openstack-ironic-inspector-dnsmasq.service loaded failed failed PXE boot dnsmasq service for Ironic Inspector ● ovirt-guest-agent.service loaded failed failed oVirt Guest Agent [root@undercloud-4 ~]# systemctl status openstack-ironic-inspector-dnsmasq -l ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2017-07-21 19:01:05 EDT; 29min ago Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. [root@undercloud-4 ~]# journalctl -u !$ journalctl -u -l -- No entries -- [root@undercloud-4 ~]# journalctl -u openstack-ironic-inspector-dnsmasq -- Logs begin at Fri 2017-07-21 18:14:52 EDT, end at Fri 2017-07-21 19:31:07 EDT. -- Jul 21 18:46:06 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq[20664]: started, version 2.66 DNS disabled Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq[20664]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Jul 21 18:46:06 undercloud-4.default.redhat.local dnsmasq-dhcp[20664]: DHCP, IP range 192.0.2.100 -- 192.0.2.120, lease time 2m Jul 21 18:46:06 undercloud-4.default.redhat.local systemd[1]: Started PXE boot dnsmasq service for Ironic Inspector. Jul 21 18:51:32 undercloud-4.default.redhat.local systemd[1]: Stopping PXE boot dnsmasq service for Ironic Inspector... Jul 21 18:51:32 undercloud-4.default.redhat.local systemd[1]: Stopped PXE boot dnsmasq service for Ironic Inspector. -- Reboot -- Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: dnsmasq: unknown interface br-ctlplane Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Jul 21 18:53:22 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. -- Reboot -- Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Jul 21 19:01:05 undercloud-4.default.redhat.local systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. [root@undercloud-4 ~]# journalctl | grep -i br-ctlplane | grep -i unknown | grep -i interfa Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: dnsmasq: unknown interface br-ctlplane Jul 21 18:53:22 undercloud-4.default.redhat.local dnsmasq[873]: unknown interface br-ctlplane Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: dnsmasq: unknown interface br-ctlplane Jul 21 19:01:05 undercloud-4.default.redhat.local dnsmasq[845]: unknown interface br-ctlplane Created attachment 1302568 [details]
sosreport --all-logs
Thanks for the sospreport Andreas. It appears that this is a known issue that has been resolved - see https://bugzilla.redhat.com/show_bug.cgi?id=1428024 There is a backport request for OSP-10 here - https://bugzilla.redhat.com/show_bug.cgi?id=1452610 I think it makes sense to mark this as a duplicate 1452610. Let me know what you think. *** This bug has been marked as a duplicate of bug 1452610 *** Thanks! |
The following happens after the reboot of the undercloud during a minor upgrade. The same happens for any new installation of the undercloud. opesntack-ironic-inspector-dnsmasq fails because br-ctlplane is not up: ack@director ~]$ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*" UNIT LOAD ACTIVE SUB JOB DESCRIPTION neutron-dhcp-agent.service loaded active running OpenStack Neutron DHCP Agent neutron-openvswitch-agent.service loaded active running OpenStack Neutron Open vSwitch Agent neutron-ovs-cleanup.service loaded active exited OpenStack Neutron Open vSwitch Cleanup Utility neutron-server.service loaded active running OpenStack Neutron Server openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service openstack-aodh-listener.service loaded active running OpenStack Alarm listener service openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service openstack-ceilometer-central.service loaded active running OpenStack ceilometer central agent openstack-ceilometer-collector.service loaded active running OpenStack ceilometer collection service openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent openstack-glance-api.service loaded active running OpenStack Image Service (code-named Glance) API server openstack-glance-registry.service loaded active running OpenStack Image Service (code-named Glance) Registry server openstack-heat-api-cfn.service loaded active running Openstack Heat CFN-compatible API Service openstack-heat-api.service loaded active running OpenStack Heat API Service openstack-heat-engine.service loaded active running Openstack Heat Engine Service openstack-ironic-api.service loaded active running OpenStack Ironic API service openstack-ironic-conductor.service loaded active running OpenStack Ironic Conductor service ● openstack-ironic-inspector-dnsmasq.service loaded failed failed PXE boot dnsmasq service for Ironic Inspector openstack-ironic-inspector.service loaded active running Hardware introspection service for OpenStack Ironic openstack-mistral-api.service loaded active running Mistral API Server openstack-mistral-engine.service loaded active running Mistral Engine Server openstack-mistral-executor.service loaded active running Mistral Executor Server openstack-nova-api.service loaded active running OpenStack Nova API Server openstack-nova-cert.service loaded active running OpenStack Nova Cert Server openstack-nova-compute.service loaded activating start start OpenStack Nova Compute Server openstack-nova-conductor.service loaded active running OpenStack Nova Conductor Server openstack-nova-scheduler.service loaded active running OpenStack Nova Scheduler Server openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server openstack-zaqar.service loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server openstack-zaqar loaded active running OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1 openvswitch.service loaded active exited Open vSwitch LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. JOB = Pending job for the unit. 37 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [stack@director ~]$ sudo systemctl status openstack-ironic-inspector-dnsmasq ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-07-06 19:51:14 EDT; 1min 27s ago Process: 1443 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=2) Jul 06 19:51:14 director.customer.poc systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 06 19:51:14 director.customer.poc dnsmasq[1443]: dnsmasq: unknown interface br-ctlplane Jul 06 19:51:14 director.customer.poc systemd[1]: openstack-ironic-inspector-dnsmasq.service: control process exited, code=exited status=2 Jul 06 19:51:14 director.customer.poc systemd[1]: Failed to start PXE boot dnsmasq service for Ironic Inspector. Jul 06 19:51:14 director.customer.poc systemd[1]: Unit openstack-ironic-inspector-dnsmasq.service entered failed state. Jul 06 19:51:14 director.customer.poc systemd[1]: openstack-ironic-inspector-dnsmasq.service failed. [stack@director ~]$ sudo systemctl restart openstack-ironic-inspector-dnsmasq [stack@director ~]$ sudo systemctl status openstack-ironic-inspector-dnsmasq ● openstack-ironic-inspector-dnsmasq.service - PXE boot dnsmasq service for Ironic Inspector Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-inspector-dnsmasq.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-07-06 19:53:06 EDT; 5s ago Process: 15348 ExecStart=/sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf (code=exited, status=0/SUCCESS) Main PID: 15350 (dnsmasq) CGroup: /system.slice/openstack-ironic-inspector-dnsmasq.service └─15350 /sbin/dnsmasq --conf-file=/etc/ironic-inspector/dnsmasq.conf Jul 06 19:53:06 director.customer.poc systemd[1]: Starting PXE boot dnsmasq service for Ironic Inspector... Jul 06 19:53:06 director.customer.poc dnsmasq[15350]: started, version 2.66 DNS disabled Jul 06 19:53:06 director.customer.poc dnsmasq[15350]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Jul 06 19:53:06 director.customer.poc dnsmasq-dhcp[15350]: DHCP, IP range 10.10.159.150 -- 10.10.159.239, lease time 2m Jul 06 19:53:06 director.customer.poc systemd[1]: Started PXE boot dnsmasq service for Ironic Inspector.