Bug 1206361

Summary: IPMI commands fail for Dell iDRAC 7
Product: Red Hat Satellite Reporter: Brandon Dunne <bdunne>
Component: ProvisioningAssignee: Lukas Zapletal <lzap>
Status: CLOSED CURRENTRELEASE QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: bbuckingham, bdunne, bkearney, cwelton, lzap, mmccune, ohadlevy, sghai
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-12 13:58:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
BMC interface configured successfully none

Description Brandon Dunne 2015-03-26 21:19:55 UTC
Description of problem:
Yesterday we ran into IMPI issues with Foreman while trying to provision a Dell system.  The Dell system is running with iDRAC7 (firmware version is 1.57.57 - Build 04) and we have confirmed that IPMI over LAN is enabled.

At first we were getting a nil value back for the system power state which lead us to dig into Foreman, the Foreman smart proxy and ultimately the RubyIMPI gem.  Foreman can use either ipmitool or freeipmi to talk to the systems.  We tested with both tools and experienced the same failure.

Running the IPMI tools directly on the command line we were able to confirm that:
* ipmitool requires the "-I lanplus" parameter
* freeipmi requires the "-D LAN_2_0 " parameter

The RubyIPMI gem tries to identify failed calls and retry with a different interface type but relies on evaluating error messages coming back from the command.  In our case the message returned does not match any of the known error messages and therefore IPMI v2 calls are never attempted and the call returns nil.

Foreman is using RubyIPMI version 0.7.0 (October 17, 2013).  We also checked the last version (0.9.2 - March 19, 2015) which has the same issue.

We have an older Dell system (with iDRAC 6) where the Foreman IPMI functionality works.  While the root cause is that the newer Dell system is not supporting older IPMI v1.5 commands it seems like Foreman, and ultimately RubyIPMI, needs to be updated to properly detect and handle this scenario.


Version-Release number of selected component (if applicable):
Foreman 1.5.3

Expected results:
IPMI v2.0 only controllers should also be supported

Comment 1 RHEL Program Management 2015-03-26 21:23:16 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 8 Lukas Zapletal 2015-04-10 09:38:09 UTC
Thanks, can you share with me your Satellite Capsule Foreman-Proxy configuration? That should be in the /etc/foreman-proxy/settings.d/bmc.yml.

Comment 9 Lukas Zapletal 2015-04-10 10:24:22 UTC
Okay we have merged this patch which allows users to specify additional options. That solves the issue:

http://projects.theforeman.org/issues/7543

Please confirm and I will backport this patch. The patch is one-liner, you can test it directly if you want. Would be appreciated as I struggle finding iDRAC7 host (got one in beaker but I have no BMC credentials unfortunately). Thanks!

Comment 10 Lukas Zapletal 2015-04-10 11:30:56 UTC
Sorry I just realized that we have added Proxy API struct to pass in BMC options. This could work for you only in case you use the Proxy API directly from your script or something.

The Foreman GUI was not yet modified and we need to think about how to implement this feature. Perhaps using a Hardware model resource we could define additional BMC options per host or hostgroup.

Comment 11 Lukas Zapletal 2015-04-13 07:34:25 UTC
Ok so the dirty workaround is to upgrade rubygem-rubyipmi to version 0.10+ which defaults to lan20 driver which works. The RPM will appear here soon: http://yum.theforeman.org/nightly/el7/x86_64/ Drawback: machines that require "lan" driver will no longer work anymore.

I think we should build this into 6.1 GA as well since it will introduce some regressions (older hardware will not work anymore). As part of 6.1.1 (new BZ) I recommend to release new feature - ability to specify IPMI driver per Hardware model. It's a bigger change for GA, but doable as an feature errata (new field for Hardware type and a small patch in Foreman and Proxy).

Comment 12 Lukas Zapletal 2015-04-13 07:57:17 UTC
Note: Default driver for "PowerEdge R720" hardware model is "lan20".

Comment 13 Lukas Zapletal 2015-04-13 08:03:54 UTC
Note: Relevant smart-proxy patches that must be in to enable foreman UI changes:

http://projects.theforeman.org/issues/10116
http://projects.theforeman.org/issues/7543

Comment 14 Ohad Levy 2015-05-13 11:46:54 UTC
lzap - can you please send merge requests to the proxy (to set lan20 as default) and suggest a documentation update (on how to use, change for previous setting etc) to reflect this change?

thanks,
Ohad

Comment 18 Sachin Ghai 2015-06-11 09:37:39 UTC
Hey Guys, Please provide steps to verify this bz. From comment 5, it seems we need to test it with specific HW Dell iDRAC7 ?

Comment 19 Lukas Zapletal 2015-06-12 11:27:45 UTC
Please just verify there are no regressions in the BMC features (power on, off, reboot). Brandon, can you test the API changes if it works for you?

Comment 21 Lukas Zapletal 2015-06-15 11:22:47 UTC
*** Bug 1231685 has been marked as a duplicate of this bug. ***

Comment 23 Sachin Ghai 2015-06-16 09:08:43 UTC
Verified with sat6.1 GA snap8 compose3.

[root@cloud-XX ~]# rpm -qa | grep ipmi
ipmitool-1.8.13-8.el7_1.x86_64
rubygem-rubyipmi-0.10.0-1.el7sat.noarch
[root@cloud-qe-22 ~]# 

BMC interface is correctly configured. I can see the interface details after provisioning the host. Please see the screenshot for verification.

Comment 24 Sachin Ghai 2015-06-16 09:09:23 UTC
Created attachment 1039311 [details]
BMC interface configured successfully

Comment 25 Sachin Ghai 2015-06-16 10:08:39 UTC
As per comment23 verified this with snap8 compose3 and successfully tested BMC configuration with default and isolated capsule. Moving this to verified.

Comment 26 Bryan Kearney 2015-08-11 13:32:29 UTC
This bug is slated to be released with Satellite 6.1.

Comment 27 Bryan Kearney 2015-08-12 13:58:05 UTC
This bug was fixed in version 6.1.1 of Satellite which was released on 12 August, 2015.