Bug 1479386

Summary: Ironic introspection loops over ipxe.efi for UEFI boot_mode
Product: Red Hat OpenStack Reporter: PURANDHAR SAIRAM MANNIDI <pmannidi>
Component: puppet-ironicAssignee: Bob Fournier <bfournie>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: bfournie, dcain, dondavis, dwojewod, dyasny, ipilcher, jbiao, jjoyce, jschluet, lruzicka, mlammon, racedoro, slinaber, stphilli, tvignaud
Target Milestone: z6Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-ironic-9.5.0-2.el7ost Doc Type: Bug Fix
Doc Text:
Originally, the DHCP server configuration file for Ironic Inspector was not set up properly to handle hosts that were using UEFI and iPXE. Consequently, any such host might fail to boot during Ironic Introspection. The problem has been fixed and the `/etc/ironic-inspector/dnsmasq.conf` has been updated to handle this situation. As a result, hosts using UEFI and iPXE can properly boot during Ironic Introspection.
Story Points: ---
Clone Of:
: 1489192 1489546 (view as bug list) Environment:
Last Closed: 2017-11-15 13:45:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1489192, 1489546    
Attachments:
Description Flags
screenshot of loop none

Description PURANDHAR SAIRAM MANNIDI 2017-08-08 13:19:42 UTC
Description of problem:

Ironic introspection loops over ipxe.efi for UEFI boot_mode

Version-Release number of selected component (if applicable):
RH OSP 10

How reproducible:
Always

Steps to Reproduce:
1. Start the ironic node introspection
2. Node loops at PXE booting after downloading ipxe.efi
3.

Actual results:
Node loops at PXE booting by downloading ipxe.efi

Expected results:
Node should continue with introspection process.

Additional info:
To reproduce the issue in lab, I added dhcp-match=set:efi,option:client-arch,9 as well.

Comment 6 PURANDHAR SAIRAM MANNIDI 2017-08-08 14:09:33 UTC
The fix is changing
dhcp-boot=tag:efi,ipxe.efi
to
dhcp-boot=tag:efi,tag:!ipxe,ipxe.efi

Comment 8 Bob Fournier 2017-08-31 21:12:48 UTC
upstream patch proposed - https://review.openstack.org/#/c/499750/

Comment 9 Bob Fournier 2017-09-11 18:49:37 UTC
Newton upstream patch - https://review.openstack.org/#/c/502365/

Comment 10 Bob Fournier 2017-09-19 18:22:41 UTC
*** Bug 1464523 has been marked as a duplicate of this bug. ***

Comment 11 stphilli 2017-10-12 19:59:29 UTC
I have another customer that seems to be hitting this issue attaching screenshot of where their remote console is looping on introspection with UEFI.

Comment 12 stphilli 2017-10-12 20:00:56 UTC
Created attachment 1337961 [details]
screenshot of loop

Comment 13 Bob Fournier 2017-10-12 20:05:43 UTC
stphilli - its likely, would have to see a tcpdump of the dhcp transactions to know for sure.  We recommend making the change to /etc/ironic-inspector/dnsmasq.conf as shown above.  The fix has merged upstream and should be in next 10.z release.

Comment 14 PURANDHAR SAIRAM MANNIDI 2017-10-13 01:36:40 UTC
stphilli it looks like a networking issue to me. PXE boot itself didn't start in the first place and this bug resolves PXE boot loop.

Comment 20 mlammon 2017-11-01 16:04:05 UTC
installed latest osp10 with puddle 2017-10-30.3 on 11/1/2017
This bug has been verified

environment:
[stack@undercloud ~]$ rpm -qa | grep puppet-ironic-
puppet-ironic-9.5.0-2.el7ost.noarch

dnsmasq.conf file appears updated
 
[stack@undercloud ~]$ sudo cat /etc/ironic-inspector/dnsmasq.conf
port=0
interface=br-ctlplane
bind-interfaces
dhcp-range=192.168.0.100,192.168.0.120,29
dhcp-sequential-ip
dhcp-match=ipxe,175
dhcp-match=set:efi,option:client-arch,7
dhcp-match=set:efi,option:client-arch,9
# Client is already running iPXE; move to next stage of chainloading
dhcp-boot=tag:ipxe,http://192.168.0.1:8088/inspector.ipxe
# Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader
dhcp-boot=tag:efi,tag:!ipxe,ipxe.efi
# Client is running PXE over BIOS; send BIOS version of iPXE chainloader
dhcp-boot=undionly.kpxe,localhost.localdomain,192.168.0.1


[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Started Mistral Workflow. Execution ID: a97a2b8f-be30-43c6-9b43-1e5776be905d
Waiting for introspection to finish...
Introspection for UUID a3002942-c5c7-4d23-aba8-aa4cee4adf17 finished successfully.
Introspection for UUID 34ac3467-6ce6-4191-bfdc-13a2163e4b39 finished successfully.
Introspection completed.
Setting manageable nodes to available...
Started Mistral Workflow. Execution ID: bcdbcbb2-99a7-4dea-b7fc-5499b31514c0

[stack@undercloud ~]$ openstack baremetal introspection bulk status
+--------------------------------------+----------+-------+
| Node UUID                            | Finished | Error |
+--------------------------------------+----------+-------+
| 34ac3467-6ce6-4191-bfdc-13a2163e4b39 | True     | None  |
| a3002942-c5c7-4d23-aba8-aa4cee4adf17 | True     | None  |
+--------------------------------------+----------+-------+
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| 34ac3467-6ce6-4191-bfdc-13a2163e4b39 | None | None          | power off   | available          | False       |
| a3002942-c5c7-4d23-aba8-aa4cee4adf17 | None | None          | power off   | available          | False       |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+

Comment 22 Bob Fournier 2017-11-02 14:54:55 UTC
I made 2 minor edits to doc text, otherwise it looks fine.
- changed iPxE to iPXE
- removed "over PXE"

Comment 25 errata-xmlrpc 2017-11-15 13:45:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3231