Bug 1225226

Summary: Nodes discovery fails when instackenv.json file contains nodes mac addresses.
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: instack-undercloudAssignee: Dmitry Tantsur <dtantsur>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: apevec, calfonso, dtantsur, lhh, mburns, oblaut, rhel-osp-director-maint, sasha, sclewis, scohen
Target Milestone: gaKeywords: Triaged
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 13:51:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ironic logs
none
more ironic logs none

Description Marius Cornea 2015-05-26 21:20:15 UTC
Created attachment 1030183 [details]
ironic logs

Description of problem:
Nodes discovery fails when instackenv.json file contains the nodes mac addresses.

Version-Release number of selected component (if applicable):
openstack-ironic-conductor-2015.1.0-2.el7ost.noarch
python-ironicclient-0.5.1-5.el7ost.noarch
openstack-ironic-api-2015.1.0-2.el7ost.noarch
openstack-ironic-discoverd-1.1.0-1.el7ost.noarch
python-ironic-discoverd-1.1.0-1.el7ost.noarch
openstack-ironic-common-2015.1.0-2.el7ost.noarch

How reproducible:
Poodle installation.

Steps to Reproduce:
1. Deploy undercloud
2. instack-ironic-deployment --nodes-json instackenv.json --register-nodes
3. instack-ironic-deployment --discover-nodes

Actual results:

[stack@bldr16cc09 ~]$ instack-ironic-deployment --discover-nodes
Preparing for deployment...
  Discovering nodes.
    Sending node ID bda62f11-c51d-4dc2-9678-8568b0df2b52 to discoverd for discovery ... DONE.
    Sending node ID 320892bc-8a4b-466d-82e4-46e6cda9be4d to discoverd for discovery ... DONE.
    Sending node ID 1ef01a1a-e985-4c42-8a2a-fa9493927595 to discoverd for discovery ... DONE.
   Polling discoverd for discovery results ... 
   Result for node bda62f11-c51d-4dc2-9678-8568b0df2b52 is ... DISCOVERED.
Node bda62f11-c51d-4dc2-9678-8568b0df2b52 is locked by host localhost.localdomain, please retry after the current operation is completed. (HTTP 409)

Expected results:
Nodes hardware parameters get discovered.

Additional info:
It looks that if I remove the mac addresses from the instackenv.json file the 'port with mac address already exits' error message disappears but discovery still fails for some of the nodes. I'm attaching relevant logs and steps I did.

Comment 3 Dmitry Tantsur 2015-05-27 07:57:07 UTC
It's definitely not a discoverd bug. Either my retrying solution in ironicclient is incomplete, or we should tune a couple of things in instack-undercloud. I'll have a look.

Comment 4 Dmitry Tantsur 2015-05-27 09:34:24 UTC
Something like https://review.gerrithub.io/#/c/234406/ should help

Comment 5 Marius Cornea 2015-05-27 10:11:10 UTC
Tested the patch and nodes got discovered with provision state set to available this time. Thanks! 

One thing I noticed is that errors still show up in the openstack-ironic-discoverd log, maybe showing those with warning severity would be better:

ERROR:ironicclient.common.http:Error contacting Ironic server: A port with MAC address d8:d3:85:be:01:0c already exists. (HTTP 409). Attempt 6 of 6
ERROR:ironicclient.common.http:Error contacting Ironic server: Node 5394855d-0423-40a4-b86a-45ebfd09abe1 can not be updated while a state transition is in progress. (HTTP 409). Attempt 6 of 6

Comment 8 Ofer Blaut 2015-06-10 14:39:18 UTC
Created attachment 1037336 [details]
more ironic logs

Comment 10 Alexander Chuzhoy 2015-06-10 17:02:58 UTC
If I remove the MAC addresses from the instackenv.json file prior to registering the nodes, then only one node gets registered upon running "instack-ironic-deployment --nodes-json instackenv.json --register-nodes".

I actually had to re-run the above command 4 times to see 4 nodes in the output from "ironic node-list", where one node was added after every run of the command to the list.

Comment 12 errata-xmlrpc 2015-08-05 13:51:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549