Bug 1567917

Summary: Scale out failed ironic node introspection timed out, mixed versions upgrade
Product: Red Hat OpenStack Reporter: Ronnie Rasouli <rrasouli>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: agurenko, aschultz, bfournie, mburns, mlammon, rhel-osp-director-maint, sasha
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-23 14:13:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
iptables rules
none
ironic conductor log none

Description Ronnie Rasouli 2018-04-16 12:30:12 UTC
Created attachment 1422426 [details]
iptables rules

Description of problem:

performed mixed versions stage upgrade, undercloud upgrade to RHOS13 and scaled out 1 extra compute node, the scale out of that node timed out.

node did not received dhcp, then the scale out stuck 

06:59:49.626034 52:54:00:0f:20:b7 > Broadcast, ethertype IPv4 (0x0800), length 438: 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:0f:20:b7, length 396




Version-Release number of selected component (if applicable):
openstack-ironic-api-10.1.2-2.el7ost.noarch
openstack-ironic-conductor-10.1.2-2.el7ost.noarch
openstack-ironic-common-10.1.2-2.el7ost.noarch

instack-undercloud-8.4.0-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. deploy RHOS12 
2. upgrade the undercloud
3. perform scale out for additional compute node

Actual results:
Scale out timeout after 1 hour

Expected results:


Additional info:

var/log/ironic-inspector/ironic-inspector.log:198:2018-04-16 02:58:18.160 27985 DEBUG ironic_inspector.firewall [-] Ignoring failed iptables ('-D', 'INPUT', '-i', 'br-ctlplane', '-p', 'udp', '--dport', '67', '-j', 'ironic-inspector'): iptables v1.4.21: Couldn't load target `ironic-inspector':No such file or directory. . Try `iptables -h' or 'iptables...

Comment 1 Ronnie Rasouli 2018-04-16 12:36:26 UTC
Created attachment 1422427 [details]
ironic conductor log

Comment 3 Ronnie Rasouli 2018-04-17 08:00:02 UTC
Disabling ipatbles does the trick - the ironic introspection pass

Comment 4 Bob Fournier 2018-04-23 14:13:34 UTC
This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1570386, patches are upstream.

*** This bug has been marked as a duplicate of bug 1570386 ***