Bug 1296678

Summary: rhel-osp-director: 8.0 bulk introspecting 8 nodes with default range (192.0.2.100,192.0.2.120), the last node doesn't get an IP and causes the introspection to stuck.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Hugh Brock <hbrock>
Status: CLOSED CURRENTRELEASE QA Contact: Shai Revivo <srevivo>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: dtantsur, jcoufal, mburns, mcornea, rhel-osp-director-maint
Target Milestone: ---   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-14 15:43:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2016-01-07 20:05:09 UTC 
rhel-osp-director: 8.0 bulk introspecting 8 nodes with default range (192.0.2.100,192.0.2.120), the last node doesn't get an IP and causes the introspection to stuck.


Environment:
openstack-ironic-conductor-4.2.2-3.el7ost.noarch
python-ironic-inspector-client-1.2.0-5.el7ost.noarch
dnsmasq-2.66-14.el7_1.x86_64
openstack-ironic-inspector-2.2.2-1.el7ost.noarch
openstack-ironic-api-4.2.2-3.el7ost.noarch
python-ironicclient-0.8.1-1.el7ost.noarch
openstack-ironic-common-4.2.2-3.el7ost.noarch
dnsmasq-utils-2.66-14.el7_1.x86_64


Steps to reproduce:
Attempt to bulk introspect 8 nodes or more with the default introspection range.

Result:
The introspection doesn't finish.
Checking the console of the node - it doesn't get an IP.
Looking at the logs:
Jan 07 20:31:56 instack.localdomain dnsmasq-dhcp[30022]: DHCPNAK(br-ctlplane) 192.0.2.105 00:ed:dc:54:1d:42 address in use
Jan 07 20:31:57 instack.localdomain dnsmasq-dhcp[30022]: DHCPNAK(br-ctlplane) 192.0.2.105 00:ed:dc:54:1d:42 address in use
Jan 07 20:31:57 instack.localdomain dnsmasq-dhcp[30022]: DHCPNAK(br-ctlplane) 192.0.2.105 00:ed:dc:54:1d:42 address in use


Note: Enlarged the range and restarted the openstack-ironic-inspector.service and openstack-ironic-inspector.service services. Restarted the introspection - completed successfully. Reduced the range to default and restarted the same services - the introspection got stuck again with the same issue.


Expected result:
Bulk introspection completed successfully for all nodes as long as their count doesn't go above the range.
Comment 3 Mike Burns 2016-04-07 21:03:37 UTC 
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 5 Dmitry Tantsur 2016-10-14 15:43:06 UTC 
Hi! I believe this problem is fixed now that we're using dhcp-sequential-ip for introspection dnsmasq (see https://github.com/openstack/puppet-ironic/commit/fc2496a4ef7f123bf4cdb3cb1ea7c32ce8c23262). Please feel free to reopen the bug if you can reproduce it with the latest bits.