Description of problem: Cisco UCS 200-M3 blades will not properly boot over the network from IPMI. The boot order does not get set to PXE properly. This works fine on the M4 blades. Version-Release number of selected component (if applicable): ironic in OSP 8 (all versions) How reproducible: all the time Steps to Reproduce: 1. deploy a new overcloud or nova boot a single node # Issue: # Actions executed by ipmi_pxetool: ~~~ ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmplMByEN raw 0x00 0x08 0x03 0x08 ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpP4vH3V chassis bootdev pxe options=persistent ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpyrGWa4 power off ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpQ__g09 power on ~~~ These actions work fine for UCS M4. We manually ran these commands (excluding the raw command, which disables a specific timeout) and it turns out that `chassis bootdev pxe` on the M3 will overwrite the boot order which is specified by the UCS profile. At the same time, it will not force the server to pxe boot. We also tried `ipmitool ... chassis bootparam set bootflag force_pxe`, but this did not help neither. In order to revert to the UCS profile, we ran `ipmitool ... chassis bootparam set bootflag none` (in order to verify the bootflag settings, one can run `ipmitool ... chassis booparam get 5`) # Analysis: # Difference between M3 and M4 via `mc info`: B200 M3 ~~~ [root@lplospiudirc1 ironic]# ipmitool -I lanplus -H 192.168.1.100 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -P '...' mc info Device ID : 32 Device Revision : 0 Firmware Revision : 2.02 IPMI Version : 2.0 Manufacturer ID : 5771 Manufacturer Name : Unknown (0x168B) Product ID : 9 (0x0009) Product Name : Unknown (0x9) Device Available : yes Provides Device SDRs : yes Additional Device Support : Sensor Device SDR Repository Device SEL Device FRU Inventory Device IPMB Event Receiver IPMB Event Generator Aux Firmware Rev Info : 0x00 0x03 0xa8 0xa1 ~~~ B200 M4 ~~~ [stack@lplosp7deploydir1 ~]$ ipmitool -I lanplus -H 192.168.1.101 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -P '...' mc info Device ID : 32 Device Revision : 0 Firmware Revision : 2.02 IPMI Version : 2.0 Manufacturer ID : 5771 Manufacturer Name : Unknown (0x168B) Product ID : 24 (0x0018) Product Name : Unknown (0x18) Device Available : yes Provides Device SDRs : yes Additional Device Support : Sensor Device SDR Repository Device SEL Device FRU Inventory Device IPMB Event Receiver IPMB Event Generator Aux Firmware Rev Info : 0x00 0x03 0xe3 0x7c ~~~ The code which sets `bootdev pxe` is in /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipmitool.py ~~~ (...) def set_boot_device(self, task, device, persistent=False): """Set the boot device for the task's node. Set the boot device to use on next reboot of the node. :param task: a task from TaskManager. :param device: the boot device, one of :mod:`ironic.common.boot_devices`. :param persistent: Boolean value. True if the boot device will persist to all future boots, False if not. Default: False. :raises: InvalidParameterValue if an invalid boot device is specified :raises: MissingParameterValue if required ipmi parameters are missing. :raises: IPMIFailure on an error from ipmitool. """ if device not in self.get_supported_boot_devices(task): raise exception.InvalidParameterValue(_( "Invalid boot device %s specified.") % device) # note(JayF): IPMI spec indicates unless you send these raw bytes the # boot device setting times out after 60s. Since it's possible it # could be >60s before a node is rebooted, we should always send them. # This mimics pyghmi's current behavior, and the "option=timeout" # setting on newer ipmitool binaries. timeout_disable = "0x00 0x08 0x03 0x08" send_raw(task, timeout_disable) if task.node.driver_info.get('ipmi_force_boot_device', False): driver_utils.force_persistent_boot(task, device, persistent) # Reset persistent to False, in case of BMC does not support # persistent or we do not have admin rights. persistent = False cmd = "chassis bootdev %s" % device if persistent: cmd = cmd + " options=persistent" driver_info = _parse_driver_info(task.node) try: out, err = _exec_ipmitool(driver_info, cmd) except (exception.PasswordFileFailedToCreate, processutils.ProcessExecutionError) as e: LOG.warning(_LW('IPMI set boot device failed for node %(node)s ' 'when executing "ipmitool %(cmd)s". ' 'Error: %(error)s'), {'node': driver_info['uuid'], 'cmd': cmd, 'error': e}) raise exception.IPMIFailure(cmd=cmd) (...) ~~~ Workaround: ~~~ We modified this to: /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipmitool.py ~~~ def set_boot_device(self, task, device, persistent=False): """Set the boot device for the task's node. Set the boot device to use on next reboot of the node. :param task: a task from TaskManager. :param device: the boot device, one of :mod:`ironic.common.boot_devices`. :param persistent: Boolean value. True if the boot device will persist to all future boots, False if not. Default: False. :raises: InvalidParameterValue if an invalid boot device is specified :raises: MissingParameterValue if required ipmi parameters are missing. :raises: IPMIFailure on an error from ipmitool. """ if device not in self.get_supported_boot_devices(task): raise exception.InvalidParameterValue(_( "Invalid boot device %s specified.") % device) # note(JayF): IPMI spec indicates unless you send these raw bytes the # boot device setting times out after 60s. Since it's possible it # could be >60s before a node is rebooted, we should always send them. # This mimics pyghmi's current behavior, and the "option=timeout" # setting on newer ipmitool binaries. timeout_disable = "0x00 0x08 0x03 0x08" send_raw(task, timeout_disable) if task.node.driver_info.get('ipmi_force_boot_device', False): driver_utils.force_persistent_boot(task, device, persistent) # Reset persistent to False, in case of BMC does not support # persistent or we do not have admin rights. persistent = False cmd = "chassis bootdev %s" % device if persistent: cmd = cmd + " options=persistent" driver_info = _parse_driver_info(task.node) try: LOG.warning('dwb') # out, err = _exec_ipmitool(driver_info, cmd) except (exception.PasswordFileFailedToCreate, processutils.ProcessExecutionError) as e: LOG.warning(_LW('IPMI set boot device failed for node %(node)s ' 'when executing "ipmitool %(cmd)s". ' 'Error: %(error)s'), {'node': driver_info['uuid'], 'cmd': cmd, 'error': e}) raise exception.IPMIFailure(cmd=cmd) ~~~ This stop ironic from sending the `chassis bootdev pxe` ipmi command.
We are going to contact Cisco and figure out why 'chassis bootdev pxe' is not working on the M3 ==> is there a workaround / different IPMI command that we need to send? *) with the conclusions from Cisco, we can hopefully make changes to the ipmi_pxetool: ==> either give users a choice not to force the `bootdev pxe` ==> detect the vendor / chassis version and do not force pxe boot if it's an M3 / respectively use the correct command as provided by Cisco
(In reply to Andreas Karis from comment #1) > We are going to contact Cisco and figure out why 'chassis bootdev pxe' is > not working on the M3 > ==> is there a workaround / different IPMI command that we need to send? > *) with the conclusions from Cisco, we can hopefully make changes to the > ipmi_pxetool: There's a specific driver for Cisco UCS [0], have you tried to use it ? > ==> either give users a choice not to force the `bootdev pxe` If we don't set the boot device we would require to ask the operator to change the boot device manually before starting deployment. That would be a manual step right ? By your report it seems to be a bug in the M3 model, it should not overwrite the boot order in the profile. > ==> detect the vendor / chassis version and do not force pxe boot if it's an > M3 / respectively use the correct command as provided by Cisco So, the ipmitool driver is suppose to be a generic driver. We shouldn't add commands for specific hardware models, that why I think you should try to the UCS driver for Ironic [0], because that's a specific driver for Cisco Hardware and we could have this type of code. ... Please lemme know if using the UCS driver [0] works for you. [0] http://docs.openstack.org/developer/ironic/drivers/ucs.html
Also, can you please check if the firmware is up-to-date ? This behavior/bug might have been fixed on newer releases.
We are trying to get clarification from Cisco as well. About the UCS driver .. we are a bit reluctant to use it, because the UCS driver tends to generate a lot of other issues (call it bad experience or "the burnt child dreads the fire"). We will talk this through and may try to use it. I will get back to you as soon as I have further information.
(In reply to Andreas Karis from comment #4) > We are trying to get clarification from Cisco as well. About the UCS driver > .. we are a bit reluctant to use it, because the UCS driver tends to > generate a lot of other issues (call it bad experience or "the burnt child > dreads the fire"). We will talk this through and may try to use it. > > I will get back to you as soon as I have further information. Thank you, I will wait for an update on this then (will leave the NEEDINFO flag active). Also, good feedback on the UCS driver. If you can elaborate it a bit more I can raise the problem upstream and fix it or at least let the Ironic Cisco developers aware of it. Thanks again, Lucas
From one of our consultants about the UCS driver: "From my experience, you can't import nodes with the UCS driver on OSPd 7/8. I went through a rotation of parameters it wanted / wanted removed, until I got back to where I started from. I believe the parsing util for the syntax looks for both pm attributes and the ucs attributes, and complains if they are both there or missing. I know that sounds confusing, but I left the exercise with a headache, and not able to import a node." Still trying to get hands on that info from Cisco.
(In reply to Andreas Karis from comment #6) > From one of our consultants about the UCS driver: > > "From my experience, you can't import nodes with the UCS driver on OSPd 7/8. > I went through a rotation of parameters it wanted / wanted removed, until I > got back to where I started from. I believe the parsing util for the syntax > looks for both pm attributes and the ucs attributes, and complains if they > are both there or missing. I know that sounds confusing, but I left the > exercise with a headache, and not able to import a node." > > Still trying to get hands on that info from Cisco. Oh I see, I think he's talking about https://bugzilla.redhat.com/show_bug.cgi?id=1290338
Coming back to this: ~~~ > ==> either give users a choice not to force the `bootdev pxe` If we don't set the boot device we would require to ask the operator to change the boot device manually before starting deployment. That would be a manual step right ? ~~~ Would it be possible to give users a choice when using the ipmi_pxe driver to simply power on/off the machines, and _not_ set the boot order? In that case, users would be required to manually set their machines to pxe boot, and the ipmi_pxe driver would only take care of power on/off, and it wouldn't touch user defined settings. I can create an RFE for that, as it looks like a newish feature.
Hi all! The bootdev behavior is likely a bug in the firmware, I'd suggest reaching out to Cisco with it. I'm converting this bug to an RFE, but I'm not sure it will be accepted upstream, as it's going to be a workaround for a firmware bug. P.S. Please do not set NEEDINFO to get attention to your bug, this is not what this field is for.
Hi, No problem, thank you, and thanks for the advice! - Andreas
Proposing upstream in https://storyboard.openstack.org/#!/story/2003203, will report back when it's accepted or rejected.
Per https://bugzilla.redhat.com/show_bug.cgi?id=1627041#c7 core_puddle_version: RHOS_TRUNK-15.0-RHEL-8-20190722.n.1 python3-tripleoclient.noarch 11.5.1-0.20190719020420.bffda01.el8ost openstack baremetal node list --fields name management_interface +--------------+----------------------+ | Name | Management Interface | +--------------+----------------------+ | ceph-0 | noop | | ceph-1 | ipmitool | | ceph-2 | ipmitool | | compute-0 | noop | | compute-1 | ipmitool | | controller-0 | noop | | controller-1 | ipmitool | | controller-2 | ipmitool | | ironic-0 | ipmitool | | ironic-1 | ipmitool | +--------------+----------------------+ or i in compute controller ceph ; do for j in 0 1 2; do echo "$i-$j"; virsh dumpxml $i-$j |grep -A 4 '<os>'; echo; done; done compute-0 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='network'/> <boot dev='hd'/> </os> compute-1 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> controller-0 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='network'/> <boot dev='hd'/> </os> controller-1 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> controller-2 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> ceph-0 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='network'/> <boot dev='hd'/> </os> ceph-1 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> ceph-2 <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> overcloud_depoy.sh ran successfully and instances were deployed to the overcloud successfully:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811