Bug 1379010

Description of problem:
Cisco UCS 200-M3 blades will not properly boot over the network from IPMI. The boot order does not get set to PXE properly. This works fine on the M4 blades.

Version-Release number of selected component (if applicable):
ironic in OSP 8 (all versions)

How reproducible:
all the time

Steps to Reproduce:
1. deploy a new overcloud or nova boot a single node

# Issue: #

Actions executed by ipmi_pxetool:
~~~
ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmplMByEN raw 0x00 0x08 0x03 0x08
ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpP4vH3V chassis bootdev pxe options=persistent
ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpyrGWa4 power off
ipmitool -I lanplus -H 172.31.20.69 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -f /tmp/tmpQ__g09 power on
~~~

These actions work fine for UCS M4. We manually ran these commands (excluding the raw command, which disables a specific timeout) and it turns out that `chassis bootdev pxe` on the M3 will overwrite the boot order which is specified by the UCS profile. At the same time, it will not force the server to pxe boot. We also tried `ipmitool ... chassis bootparam set bootflag force_pxe`, but this did not help neither. In order to revert to the UCS profile, we ran `ipmitool ... chassis bootparam set bootflag none` (in order to verify the bootflag settings, one can run `ipmitool ... chassis booparam get 5`)

# Analysis: #
 
Difference between M3 and M4 via `mc info`:

B200 M3
~~~
[root@lplospiudirc1 ironic]# ipmitool -I lanplus -H 192.168.1.100 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -P '...' mc info
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 2.02
IPMI Version              : 2.0
Manufacturer ID           : 5771
Manufacturer Name         : Unknown (0x168B)
Product ID                : 9 (0x0009)
Product Name              : Unknown (0x9)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
Aux Firmware Rev Info     : 
    0x00
    0x03
    0xa8
    0xa1
~~~

B200 M4
~~~
[stack@lplosp7deploydir1 ~]$ ipmitool -I lanplus -H 192.168.1.101 -L ADMINISTRATOR -U osp7ipmi -R 3 -N 5 -P '...' mc info
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 2.02
IPMI Version              : 2.0
Manufacturer ID           : 5771
Manufacturer Name         : Unknown (0x168B)
Product ID                : 24 (0x0018)
Product Name              : Unknown (0x18)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
Aux Firmware Rev Info     : 
    0x00
    0x03
    0xe3
    0x7c
~~~

The code which sets `bootdev pxe` is in
/usr/lib/python2.7/site-packages/ironic/drivers/modules/ipmitool.py
~~~
(...)
def set_boot_device(self, task, device, persistent=False):
        """Set the boot device for the task's node.

        Set the boot device to use on next reboot of the node.

        :param task: a task from TaskManager.
        :param device: the boot device, one of
                       :mod:`ironic.common.boot_devices`.
        :param persistent: Boolean value. True if the boot device will
                           persist to all future boots, False if not.
                           Default: False.
        :raises: InvalidParameterValue if an invalid boot device is specified
        :raises: MissingParameterValue if required ipmi parameters are missing.
        :raises: IPMIFailure on an error from ipmitool.

        """
        if device not in self.get_supported_boot_devices(task):
            raise exception.InvalidParameterValue(_(
                "Invalid boot device %s specified.") % device)

        # note(JayF): IPMI spec indicates unless you send these raw bytes the
        # boot device setting times out after 60s. Since it's possible it
        # could be >60s before a node is rebooted, we should always send them.
        # This mimics pyghmi's current behavior, and the "option=timeout"
        # setting on newer ipmitool binaries.
        timeout_disable = "0x00 0x08 0x03 0x08"
        send_raw(task, timeout_disable)

        if task.node.driver_info.get('ipmi_force_boot_device', False):
            driver_utils.force_persistent_boot(task,
                                               device,
                                               persistent)
            # Reset persistent to False, in case of BMC does not support
            # persistent or we do not have admin rights.
            persistent = False

        cmd = "chassis bootdev %s" % device
        if persistent:
            cmd = cmd + " options=persistent"
        driver_info = _parse_driver_info(task.node)
        try:
            out, err = _exec_ipmitool(driver_info, cmd)
        except (exception.PasswordFileFailedToCreate,
                processutils.ProcessExecutionError) as e:
            LOG.warning(_LW('IPMI set boot device failed for node %(node)s '
                            'when executing "ipmitool %(cmd)s". '
                            'Error: %(error)s'),
                        {'node': driver_info['uuid'], 'cmd': cmd, 'error': e})
            raise exception.IPMIFailure(cmd=cmd)
(...)
~~~

Workaround:
~~~
We modified this to:
/usr/lib/python2.7/site-packages/ironic/drivers/modules/ipmitool.py
~~~
def set_boot_device(self, task, device, persistent=False):
        """Set the boot device for the task's node.

        Set the boot device to use on next reboot of the node.

        :param task: a task from TaskManager.
        :param device: the boot device, one of
                       :mod:`ironic.common.boot_devices`.
        :param persistent: Boolean value. True if the boot device will
                           persist to all future boots, False if not.
                           Default: False.
        :raises: InvalidParameterValue if an invalid boot device is specified
        :raises: MissingParameterValue if required ipmi parameters are missing.
        :raises: IPMIFailure on an error from ipmitool.

        """
        if device not in self.get_supported_boot_devices(task):
            raise exception.InvalidParameterValue(_(
                "Invalid boot device %s specified.") % device)

        # note(JayF): IPMI spec indicates unless you send these raw bytes the
        # boot device setting times out after 60s. Since it's possible it
        # could be >60s before a node is rebooted, we should always send them.
        # This mimics pyghmi's current behavior, and the "option=timeout"
        # setting on newer ipmitool binaries.
        timeout_disable = "0x00 0x08 0x03 0x08"
        send_raw(task, timeout_disable)

        if task.node.driver_info.get('ipmi_force_boot_device', False):
            driver_utils.force_persistent_boot(task,
                                               device,
                                               persistent)
            # Reset persistent to False, in case of BMC does not support
            # persistent or we do not have admin rights.
            persistent = False

        cmd = "chassis bootdev %s" % device
        if persistent:
            cmd = cmd + " options=persistent"
        driver_info = _parse_driver_info(task.node)
        try:
            LOG.warning('dwb')
#            out, err = _exec_ipmitool(driver_info, cmd)
        except (exception.PasswordFileFailedToCreate,
                processutils.ProcessExecutionError) as e:
            LOG.warning(_LW('IPMI set boot device failed for node %(node)s '
                            'when executing "ipmitool %(cmd)s". '
                            'Error: %(error)s'),
                        {'node': driver_info['uuid'], 'cmd': cmd, 'error': e})
            raise exception.IPMIFailure(cmd=cmd)
~~~

This stop ironic from sending the `chassis bootdev pxe` ipmi command.