Bug 1577364

Summary: FFU: ironic-inspector firewall rule is missing on the undercloud after upgrade
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Gurenko Alex <agurenko>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: bfournie, dbecker, hjensas, lbezdick, mburns, morazi, sgolovat
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-24 14:18:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
undercloud upgrade logs
none
/etc/puppet/hieradata/puppet-stack-config.yaml
none
iptables none

Description Marius Cornea 2018-05-11 19:30:05 UTC
Description of problem:
FFU: ironic-inspector firewall rule is missing on the undercloud after upgrade.

Version-Release number of selected component (if applicable):
instack-undercloud-8.4.1-4.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10
2. Upgrade to OSP13 via fast forward upgrade procedure
3. Check iptables rules on the undercloud

Actual results:
No rule allowing connections to tcp port 5050:

(undercloud) [stack@undercloud-0 ~]$ sudo iptables -nL | grep 5050
(undercloud) [stack@undercloud-0 ~]$ sudo grep 5050 /etc/sysconfig/iptables
(undercloud) [stack@undercloud-0 ~]$ sudo grep ironic /etc/sysconfig/iptables
-A INPUT -p tcp -m multiport --dports 6385,13385 -m comment --comment "135 ironic ipv4" -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m multiport --dports 6385,13385 -m comment --comment "135 ironic ipv4" -m state --state NEW -j ACCEPT

Expected results:
An iptables rule allowing connection to tcp port 5050 is present.

Additional info:

Any introspection attempts post upgrade fail because the nodes cannot reach the undercloud on port 5050.

Comment 1 Marius Cornea 2018-05-11 19:38:29 UTC
Wokraround: add the rule manually by:

sudo iptables -I INPUT -p tcp -m multiport --dports 5050 -m state --state NEW -m comment --comment "137 ironic-inspector ipv4" -j ACCEPT

Comment 2 Marius Cornea 2018-05-11 19:40:19 UTC
Created attachment 1435110 [details]
undercloud upgrade logs

Attaching undercloud upgrade outputs.

Comment 3 Marius Cornea 2018-05-11 21:12:25 UTC
Created attachment 1435125 [details]
/etc/puppet/hieradata/puppet-stack-config.yaml

Attaching /etc/puppet/hieradata/puppet-stack-config.yaml

Comment 4 Marius Cornea 2018-05-11 21:12:43 UTC
Created attachment 1435126 [details]
iptables

/etc/sysconfig/iptables

Comment 6 Harald Jensås 2018-05-14 13:15:30 UTC
I was not able to reproduce this using upstream bits.
Are we consistently seeing this downstream?

Comment 7 Marius Cornea 2018-05-14 13:33:03 UTC
(In reply to Harald Jensås from comment #6)
> I was not able to reproduce this using upstream bits.
> Are we consistently seeing this downstream?

Yes, on an environment upgraded yesterday:

[stack@undercloud-0 ~]$ sudo grep 5050 /etc/sysconfig/iptables
[stack@undercloud-0 ~]$ sudo iptables -nL | grep 5050
[stack@undercloud-0 ~]$

Comment 8 Bob Fournier 2018-05-14 14:16:40 UTC
Marius - next time you upgrade, can you check that rule before and after 11 upgrade, after 12 upgrade etc. so we can see if its getting dropped prior to 13 upgrade?

Comment 9 Marius Cornea 2018-05-14 18:32:02 UTC
(In reply to Bob Fournier from comment #8)
> Marius - next time you upgrade, can you check that rule before and after 11
> upgrade, after 12 upgrade etc. so we can see if its getting dropped prior to
> 13 upgrade?

After upgrade to OSP12:

[root@undercloud-0 stack]# grep 5050 /etc/sysconfig/iptables
-A INPUT -p tcp -m multiport --dports 5050 -m comment --comment "137 ironic-inspector ipv4" -m state --state NEW -j ACCEPT
[root@undercloud-0 stack]# iptables -nL | grep 5050
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 5050 /* 137 ironic-inspector ipv4 */ state NEW


After upgrade to OSP13:

[root@undercloud-0 stack]# grep 5050 /etc/sysconfig/iptables
[root@undercloud-0 stack]# iptables -nL | grep 5050
[root@undercloud-0 stack]#

Comment 11 Sergii Golovatiuk 2018-05-24 14:18:24 UTC
I have tried to reproduce the issue. However, I see the port on all release. I also got confirmation from Bob Fournier who cannot reproduce the bug either. Feel free to open it when issue appears again.