Bug 1206360 - instack-undercloud: after enabling the pxe_drac driver for a host, running instack-ironic-deployment --discover-nodes gets stuck.
Summary: instack-undercloud: after enabling the pxe_drac driver for a host, running i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ga
: Director
Assignee: James Slagle
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-26 21:17 UTC by Alexander Chuzhoy
Modified: 2015-08-05 13:51 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-05 13:51:04 UTC
Target Upstream Version:


Attachments (Terms of Use)
openstack-ironic-conductor.log (170.53 KB, text/x-vhdl)
2015-03-27 15:43 UTC, Alexander Chuzhoy
no flags Details
openstack-ironic-discoverd.log (107.83 KB, text/x-vhdl)
2015-03-27 15:44 UTC, Alexander Chuzhoy
no flags Details
ironic-conductor.log (6.55 KB, text/plain)
2015-03-27 17:45 UTC, Imre Farkas
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Alexander Chuzhoy 2015-03-26 21:17:45 UTC
instack-undercloud: after enabling the pxe_drac driver for a host, running  instack-ironic-deployment --discover-nodes gets stuck.

Checking the console of a host - I see that nothing happens, the host doesn't boot.


Steps to reproduce:
1. Install undercloud.
2. append pxe_drac to the "enabled_drivers=pxe_ipmitool,pxe_ssh" line
3. sudo systemctl restart openstack-ironic-conductor
4.    ironic node-update <nodeID> add driver='pxe_drac'
    ironic node-update <nodeID> add driver_info/drac_password='<password>'
    ironic node-update <nodeID> add driver_info/drac_username='<user
    ironic node-update <nodeID> add driver_info/drac_host='<host>'`

5.  instack-ironic-deployment --discover-nodes


Result:
Polling discoverd for discovery results ...
gets stuck. the hosts aren't booted.

Expected result:
The discovery should complete.

Comment 3 Dmitry Tantsur 2015-03-27 08:38:47 UTC
Hi! Please provide output (or last relevant lines) of $ sudo journalctl -b -u openstack-ironic-discoverd

Comment 4 Imre Farkas 2015-03-27 09:43:55 UTC
Failed to reproduce based on the steps provided.

+1 for logs, it would be really useful if you could always attach all relevant logs, including discoverd and ironic in this case.

Comment 5 Alexander Chuzhoy 2015-03-27 13:29:25 UTC
 sudo journalctl -b -u openstack-ironic-discoverd:
-- Logs begin at Thu 2015-03-26 15:35:55 EDT, end at Fri 2015-03-27 09:28:08 EDT. --

Comment 6 Alexander Chuzhoy 2015-03-27 15:43:53 UTC
Created attachment 1007363 [details]
openstack-ironic-conductor.log

Comment 7 Alexander Chuzhoy 2015-03-27 15:44:34 UTC
Created attachment 1007364 [details]
openstack-ironic-discoverd.log

Comment 8 Dmitry Tantsur 2015-03-27 16:34:16 UTC
Looks like discovery ended successfully in your case, maybe you should wait a bit more?

Comment 9 Imre Farkas 2015-03-27 17:45:07 UTC
Created attachment 1007408 [details]
ironic-conductor.log

I managed to track this down. It's caused by driver inconsistency in Ironic. Eg. DRAC raised an error when a reboot was requested while the node is powered off. This has been discussed upstream and the desired behaviour is that the driver powers on the node in this case.

One important step to make this issue reproducible is that the node must be powered off before discovery.

Comment 10 Alexander Chuzhoy 2015-03-27 19:10:27 UTC
Reply to comment #8, the log includes subsequent runs.
Comment #9 explains the issue.

Comment 11 Imre Farkas 2015-03-30 13:44:22 UTC
Upstream patch: https://review.openstack.org/#/c/168902/

Comment 17 Alexander Chuzhoy 2015-06-18 16:15:41 UTC
Verified:
Environment:
openstack-ironic-common-2015.1.0-4.el7ost.noarch
openstack-ironic-discoverd-1.1.0-3.el7ost.noarch
python-ironic-discoverd-1.1.0-3.el7ost.noarch
python-ironicclient-0.5.1-5.el7ost.noarch
openstack-ironic-conductor-2015.1.0-4.el7ost.noarch
openstack-ironic-api-2015.1.0-4.el7ost.noarch
instack-undercloud-2.1.2-1.el7ost.noarch


1. These are the enabled drivers now by default:
enabled_drivers=pxe_ipmitool,pxe_ssh,pxe_drac

2. The unified CLI command to discover the nodes is now:
openstack baremetal introspection bulk start


Result:
Discovery for UUID <UUID> finished successfully.

Comment 19 errata-xmlrpc 2015-08-05 13:51:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.