Bug 1877143

Summary: With latest Dell f/w unable to use ipmitool to power nodes
Product: Red Hat Enterprise Linux 8 Reporter: Steve Reichard <sreichar>
Component: ipmitoolAssignee: Vaclav Dolezal <vdolezal>
Status: CLOSED NOTABUG QA Contact: Rachel Sibley <rasibley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.2CC: pibanezr, rvr
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-21 07:21:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steve Reichard 2020-09-08 22:15:39 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Steve Reichard 2020-09-09 11:26:10 UTC 
hit enter too quick

I have a baremetal cluster that has failed to install using IPI

TASK [node-prep : Create list of hosts that are going to be powered off] ***********************************************************
Wednesday 02 September 2020  23:48:16 +0000 (0:00:00.195)       0:01:07.745 *** 
changed: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] => (item=rna1-master-0)
changed: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] => (item=rna1-master-1)
changed: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] => (item=rna1-master-2)

TASK [node-prep : Power off hosts] *************************************************************************************************
Wednesday 02 September 2020  23:48:17 +0000 (0:00:00.131)       0:01:07.877 *** 
ok: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] => (item=rna1-master-0)
FAILED - RETRYING: Power off hosts (10 retries left).
FAILED - RETRYING: Power off hosts (9 retries left).
FAILED - RETRYING: Power off hosts (8 retries left).
FAILED - RETRYING: Power off hosts (7 retries left).
FAILED - RETRYING: Power off hosts (6 retries left).
FAILED - RETRYING: Power off hosts (5 retries left).
FAILED - RETRYING: Power off hosts (4 retries left).
FAILED - RETRYING: Power off hosts (3 retries left).
FAILED - RETRYING: Power off hosts (2 retries left).
FAILED - RETRYING: Power off hosts (1 retries left).
[WARNING]: The value False (type bool) in a string field was converted to 'False' (type string). If this does not look like what
you expect, quote the entire value to ensure it does not change.
failed: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] (item=rna1-master-1) => {"ansible_loop_var": "item", "attempts": 10, "changed": false, "item": "rna1-master-1", "msg": "Insufficient resources to create new session (wait for existing sessions to timeout)"}
ok: [rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com] => (item=rna1-master-2)

PLAY RECAP *************************************************************************************************************************
rna1-worker-0.rna1.cloud.lab.eng.bos.redhat.com : ok=62   changed=15   unreachable=0    failed=1    skipped=42   rescued=0    ignored=0   

Wednesday 02 September 2020  23:49:11 +0000 (0:00:54.726)       0:02:02.603 *** 
=============================================================================== 
node-prep : Power off hosts ------------------------------------------------------------------------------------------------ 54.73s
node-prep : Get all the firmware results from all the hosts ---------------------------------------------------------------- 25.89s
node-prep : Reload baremetal bridge and slave interfaces -------------------------------------------------------------------- 6.25s
node-prep : Get all the chassis results from all the hosts ------------------------------------------------------------------ 5.10s
Gathering Facts ------------------------------------------------------------------------------------------------------------- 4.99s
node-prep : Install required packages --------------------------------------------------------------------------------------- 3.52s
node-prep : Confirm whether or not internet connectivity on provisioner host ------------------------------------------------ 1.68s
node-prep : Gather the rpm package facts ------------------------------------------------------------------------------------ 1.40s
node-prep : Enable and restart Services ------------------------------------------------------------------------------------- 0.88s
node-prep : Enable HTTP for firewalld --------------------------------------------------------------------------------------- 0.70s
node-prep : Delete eno1 due to modify nmcli bug ----------------------------------------------------------------------------- 0.64s
node-prep : Regather Network Facts ------------------------------------------------------------------------------------------ 0.63s
node-prep : Add "{{ ansible_user }}" user to libvirt group and get ssh key setup -------------------------------------------- 0.63s
node-prep : Define Storage Pool for default --------------------------------------------------------------------------------- 0.63s
node-prep : Get the provision connection name ------------------------------------------------------------------------------- 0.55s
node-prep : Clear config dir (if any, in case this is a re-run) ------------------------------------------------------------- 0.55s
node-prep : Enable Services (firewalld) ------------------------------------------------------------------------------------- 0.54s
node-prep : Disconnect "provisioning" connection ---------------------------------------------------------------------------- 0.53s
node-prep : Create Bridge slave on eno1 ipv4 -------------------------------------------------------------------------------- 0.50s
node-prep : Create Bridge slave on eno2 ------------------------------------------------------------------------------------- 0.49s
[kni@rna1-worker-0 ansible-ipi-install]$


ipmitool will allow me to get the power status

[kni@rna1-worker-0 ansible-ipi-install]$ ipmitool -I lanplus -H 10.19.143.113 -U root -P XXXX power status
Chassis Power is off
[kni@rna1-worker-0 ansible-ipi-install]$ 

But does not let me change power

[kni@rna1-worker-0 ansible-ipi-install]$ ipmitool -I lanplus -H 10.19.143.113 -U root -P XXXX power off
Set Chassis Power Control to Down/Off failed: Command not supported in present state
[kni@rna1-worker-0 ansible-ipi-install]$ ipmitool -vv -I lanplus -H 10.19.143.113 -U root -P 100Mgmt- power off

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x80 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x81 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x82 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x83 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x84 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x85 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x54
>>    data    : 0x0e 0x00 0x86 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0
Using best available cipher suite 17

>> SENDING AN OPEN SESSION REQUEST

<<OPEN SESSION RESPONSE
<<  Message tag                        : 0x00
<<  RMCP+ status                       : no errors
<<  Maximum privilege level            : admin
<<  Console Session ID                 : 0xa0a2a3a4
<<  BMC Session ID                     : 0x02001100
<<  Negotiated authenticatin algorithm : hmac_sha256
<<  Negotiated integrity algorithm     : sha256_128
<<  Negotiated encryption algorithm    : aes_cbc_128

>> Console generated random number (16 bytes)
 51 54 6c 44 05 7a c4 25 29 b9 0f 35 9b c8 60 dc
>> SENDING A RAKP 1 MESSAGE

<<RAKP 2 MESSAGE
<<  Message tag                   : 0x00
<<  RMCP+ status                  : no errors
<<  Console Session ID            : 0xa0a2a3a4
<<  BMC random number             : 0xb72e8697d1d830a5bf44c53320f50349
<<  BMC GUID                      : 0x44454c4c590010338051b6c04f325732
<<  Key exchange auth code [sha256]: 0x6405af55eeffebd3a3e6283ae4ee635f964901448db77e74db2e446eb26f3129

session integrity key input (38 bytes)
 51 54 6c 44 05 7a c4 25 29 b9 0f 35 9b c8 60 dc
 b7 2e 86 97 d1 d8 30 a5 bf 44 c5 33 20 f5 03 49
 14 04 72 6f 6f 74
Generated session integrity key (32 bytes)
 bd 8a 92 e3 8b d6 55 91 e0 34 cb 30 af b0 c5 95
 cb 02 e3 46 55 25 41 43 2e 36 1f 8d c9 9a 8a 23
Generated K1 (32 bytes)
 31 f3 eb 17 aa 07 56 84 ae f3 b5 b0 87 59 b9 cd
 db 22 c4 62 b4 13 d3 4a 68 a4 58 d1 23 f4 9a 43
Generated K2 (32 bytes)
 03 39 ba c2 72 52 83 d2 8d 28 9b dd ad 2d e2 06
 03 a6 7e 7b 48 c2 7c ea 4e 7b 9c cc 51 c5 58 e8
>> SENDING A RAKP 3 MESSAGE

<<RAKP 4 MESSAGE
<<  Message tag                   : 0x00
<<  RMCP+ status                  : no errors
<<  Console Session ID            : 0xa0a2a3a4
<<  Key exchange auth code [sha256]: 0xa60b9f5a846f1efcf1b39f4f2a2a16d6

IPMIv2 / RMCP+ SESSION OPENED SUCCESSFULLY


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x3b
>>    data    : 0x04 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0
Set Session Privilege Level to ADMINISTRATOR


>> Sending IPMI command payload
>>    netfn   : 0x2c
>>    command : 0x3e
>>    data    : 0x00 0x02 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0
IPM Controller is not HPM.2 compatible

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x01
>>    data    : 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
Iana: 674
Running Get PICMG Properties my_addr 0x20, transit 0, target 0x20

>> Sending IPMI command payload
>>    netfn   : 0x2c
>>    command : 0x00
>>    data    : 0x00 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
Error response 0xc1 from Get PICMG Properities
Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20

>> Sending IPMI command payload
>>    netfn   : 0x2c
>>    command : 0x00
>>    data    : 0x03 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
Invalid completion code received: Invalid command
Acquire IPMB address
Discovered IPMB address 0x0
Interface address: my_addr 0x20 transit 0:0 target 0x20:0 ipmb_target 0


>> Sending IPMI command payload
>>    netfn   : 0x00
>>    command : 0x02
>>    data    : 0x00 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
Set Chassis Power Control to Down/Off failed: Command not supported in present state

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x3c
>>    data    : 0x00 0x11 0x00 0x02 

Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 0
Closed Session 02001100

[kni@rna1-worker-0 ansible-ipi-install]$





ipmitool version:
[root@rna1-worker-0 ~]# rpm -qa | grep ipmi
ipmitool-1.8.18-14.el8.x86_64
[root@rna1-worker-0 ~]# 

Idrac fw:
BIOS Version	2.8.1
iDRAC Firmware Version	4.22.00.00