Created attachment 1455664 [details] sosreport from director Description of problem: OSP13 deployment is failing with error "Failed to deploy: IPMI call failed: power on" setup info:- USed Ingrared to setup "3cntrl + 3ceph + 2comp" environment. deployment-script:- ~~~ (undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ --log-file overcloud_deployment_45.log ~~~ deployment Error:- nova-compute.log ~~~ 2018-06-30 11:33:02.457 32697 ERROR nova.compute.manager [instance: bdecc25d-b0c9-4cb4-8d91-6dbbfb4ee263] InstanceDeployFailure: Failed to provision instance bdecc25d-b0c9-4cb4-8d91-6dbbfb4ee263: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:03.082 32697 ERROR oslo.service.loopingcall [req-0f642f01-02c8-4f49-84df-633423eb6997 17f47b7d59ce46f7b8838966c6a773d2 a0c375151c524d5d80e1a4954ac05a22 - default default] Fixed interval looping call 'nova.virt.ironic.driver.IronicDriver._wait_for_active' failed: InstanceDeployFailure: Failed to provision instance 784fdeed-5cad-4e22-b98c-6a1bb237a185: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:03.082 32697 ERROR oslo.service.loopingcall InstanceDeployFailure: Failed to provision instance 784fdeed-5cad-4e22-b98c-6a1bb237a185: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:03.083 32697 ERROR nova.virt.ironic.driver [req-0f642f01-02c8-4f49-84df-633423eb6997 17f47b7d59ce46f7b8838966c6a773d2 a0c375151c524d5d80e1a4954ac05a22 - default default] Error deploying instance 784fdeed-5cad-4e22-b98c-6a1bb237a185 on baremetal node a962fe9d-fba2-4c2b-901b-b2cca7cb2d6e.: InstanceDeployFailure: Failed to provision instance 784fdeed-5cad-4e22-b98c-6a1bb237a185: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:03.083 32697 ERROR nova.compute.manager [req-0f642f01-02c8-4f49-84df-633423eb6997 17f47b7d59ce46f7b8838966c6a773d2 a0c375151c524d5d80e1a4954ac05a22 - default default] [instance: 784fdeed-5cad-4e22-b98c-6a1bb237a185] Instance failed to spawn: InstanceDeployFailure: Failed to provision instance 784fdeed-5cad-4e22-b98c-6a1bb237a185: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:03.083 32697 ERROR nova.compute.manager [instance: 784fdeed-5cad-4e22-b98c-6a1bb237a185] InstanceDeployFailure: Failed to provision instance 784fdeed-5cad-4e22-b98c-6a1bb237a185: Failed to deploy: IPMI call failed: power on. 2018-06-30 11:33:12.005 32697 DEBUG nova.compute.utils [req-4dbc2f1a-8fea-4d15-b041-70d23078a41a b9e51299a421407db396f7c6ddb0a2c7 de7265cd4f884c4b9a2747d0ca2a3550 - default default] [instance: bdecc25d-b0c9-4cb4-8d91-6dbbfb4ee263] Failed to provision instance bdecc25d-b0c9-4cb4-8d91-6dbbfb4ee263: Failed to deploy: IPMI call failed: power on. notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:330 ~~~ installation-log ~~~ Stack overcloud CREATE_FAILED overcloud.Controller.1.Controller: resource_type: OS::TripleO::ControllerServer physical_resource_id: 83ade00a-3592-4d75-b6c9-749ba0710938 status: CREATE_FAILED status_reason: | ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance 83ade00a-3592-4d75-b6c9-749ba0710938 aborted: Failure prepping block device., Code: 500" ~~~ Version-Release number of selected component (if applicable): OSP13 How reproducible: Every time Steps to Reproduce: Deploy and provision the overcloud using IR Actual results: Installation is failing as it seem not able to power on the baremetal node ~~ IPMI call failed: power on. ~~ Expected results: Installation should work Additional info: sosreport from the undercloud is attached
Created attachment 1455665 [details] md5chksum
As noted, this is due to IPM failures. ironic-conductor.log 2018-06-30 11:31:52.868 21311 DEBUG oslo_concurrency.processutils [req-63307f66-8f8f-4194-b447-84aa9e30a8dd f078d48713ee49f08ae5a5e3487f1730 de7265cd4f884c4b9a2747d0ca2a3550 - default default] u'ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpaj7eBW power on' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457 2018-06-30 11:31:52.868 21311 ERROR ironic.drivers.modules.ipmitool [req-63307f66-8f8f-4194-b447-84aa9e30a8dd f078d48713ee49f08ae5a5e3487f1730 de7265cd4f884c4b9a2747d0ca2a3550 - default default] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpaj7eBW power on" for node 1caf294f-e25e-4693-9154-384eebed914e. Error: Unexpected error while running command. Command: ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpaj7eBW power on Exit code: 1 Stdout: u'' Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command. 2018-06-30 11:31:52.869 21311 WARNING ironic.drivers.modules.ipmitool [req-63307f66-8f8f-4194-b447-84aa9e30a8dd f078d48713ee49f08ae5a5e3487f1730 de7265cd4f884c4b9a2747d0ca2a3550 - default default] IPMI power action power on failed for node 1caf294f-e25e-4693-9154-384eebed914e with error: Unexpected error while running command. Command: ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpaj7eBW power on Exit code: 1 Stdout: u'' Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n'.: ProcessExecutionError: Unexpected error while running command. Are you using baremetal nodes or vbmc? If vbmc, this is due to the RHEL libvirt issue identified in https://bugzilla.redhat.com/show_bug.cgi?id=1576464 If baremetal, there's an issue with the hardware not responding to IPMI commands. Please check that the IPMI address and username/password are correct and that the hardware has updated firmware. There is not much Ironic can do is the node is not responding to IPMI.
Thanks Sufiyan. As you are using VBMC this is the RHEL libvirt issue for which a fix should be available in recent RHEL 7.4 and 7.5 releases. You will need to install a recent release with the fix. I'm marking as a dup against the 7.5 bug as it looks like you are running 7.5 $ cat etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) *** This bug has been marked as a duplicate of bug 1581364 ***