Bug 2074541

Summary: [OSP17] composable deployment with separate database node and haproxy on controllers does not create correct IPTables rules
Product: Red Hat OpenStack Reporter: Mikolaj Ciecierski <mciecier>
Component: openstack-tripleo-heat-templatesAssignee: Takashi Kajinami <tkajinam>
Status: CLOSED ERRATA QA Contact: Khomesh Thakre <kthakre>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 17.0 (Wallaby)CC: cjeanner, dabarzil, drosenfe, kthakre, lmiccini, mburns, ramishra
Target Milestone: AlphaKeywords: Triaged
Target Release: 17.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-14.3.1-0.20220529222435.4631b4f.el9ost tripleo-ansible-3.3.1-0.20220513220922.3267287.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:20:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mikolaj Ciecierski 2022-04-12 12:57:15 UTC 
Description of problem:

Overcloud deployment using composable roles(database,messaging,networker, controllerOpenstack,compute) hangs on `Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3` 

The step_3 starts *_api_db_sync containers. They are timing out because they can't connect to mysql. It seems there is no iptables rules on controller nodes to allow database traffic for haproxy(running on controllers):

[heat-admin@controller-2 ~]$ sudo iptables-save                                                                                                                                                                                      [22/1298]
# Generated by iptables-save v1.8.4 on Tue Apr 12 12:49:53 2022                                                            
*filter                                                                                                                     
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:PREROUTING - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "000 accept related established rules ipv4" -j ACCEPT
-A INPUT -p icmp -m conntrack --ctstate NEW -m comment --comment "001 accept all icmp ipv4" -j ACCEPT
-A INPUT -i lo -m conntrack --ctstate NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m comment --comment "003 accept ssh from ctlplane subnet 192.168.24.0/24 ipv4" -j ACCEPT
-A INPUT -p udp -m udp --dport 123 -m conntrack --ctstate NEW -m comment --comment "105 ntp ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3300 -m conntrack --ctstate NEW -m comment --comment "110 ceph_mon ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6789 -m conntrack --ctstate NEW -m comment --comment "110 ceph_mon ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 35357 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13000 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5000 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13292 -m conntrack --ctstate NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 9292 -m conntrack --ctstate NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6800:7300 -m conntrack --ctstate NEW -m comment --comment "113 ceph_mgr ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13774 -m conntrack --ctstate NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8774 -m conntrack --ctstate NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13696 -m conntrack --ctstate NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 9696 -m conntrack --ctstate NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT
-A INPUT -p udp -m udp --dport 4789 -m conntrack --ctstate NEW -m comment --comment "118 neutron vxlan networks ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13776 -m conntrack --ctstate NEW -m comment --comment "119 cinder ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8776 -m conntrack --ctstate NEW -m comment --comment "119 cinder ipv4" -j ACCEPT
-A INPUT -p udp -m udp --dport 6081 -m conntrack --ctstate NEW -m comment --comment "119 neutron geneve networks ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3260 -m conntrack --ctstate NEW -m comment --comment "120 iscsi initiator ipv4" -j ACCEPT
-A INPUT -s 172.17.1.0/24 -p tcp -m tcp --dport 11211 -m conntrack --ctstate NEW -m comment --comment "121 memcached 172.17.1.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6642 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6641 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3125 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13808 -m conntrack --ctstate NEW -m comment --comment "122 ceph rgw ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW -m comment --comment "122 ceph rgw ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p udp -m udp --dport 161 -m conntrack --ctstate NEW -m comment --comment "124 snmp 192.168.24.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13004 -m conntrack --ctstate NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8004 -m conntrack --ctstate NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13800 -m conntrack --ctstate NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8000 -m conntrack --ctstate NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -m comment --comment "126 horizon ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -m comment --comment "126 horizon ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 21064 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3121 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2224 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
-A INPUT -p udp -m udp --dport 5405 -m conntrack --ctstate NEW -m comment --comment "131 pacemaker udp ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13080 -m conntrack --ctstate NEW -m comment --comment "137 nova_vnc_proxy ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6080 -m conntrack --ctstate NEW -m comment --comment "137 nova_vnc_proxy ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13778 -m conntrack --ctstate NEW -m comment --comment "138 placement ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8778 -m conntrack --ctstate NEW -m comment --comment "138 placement ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 13775 -m conntrack --ctstate NEW -m comment --comment "139 nova_metadata ipv4" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8775 -m conntrack --ctstate NEW -m comment --comment "139 nova_metadata ipv4" -j ACCEPT
-A INPUT -m conntrack --ctstate NEW -m limit --limit 20/min --limit-burst 15 -m comment --comment "998 log all ipv4" -j LOG
-A INPUT -m conntrack --ctstate NEW -m comment --comment "999 drop all ipv4" -j DROP




Version-Release number of selected component (if applicable):
RHOS-17.0-RHEL-8-20220401.n.1

How reproducible:
100%

Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Luca Miccini 2022-04-12 13:42:02 UTC 
upstream bz: https://bugs.launchpad.net/tripleo/+bug/1961799

wip patches (not sure how up to date they are) https://review.opendev.org/q/topic:bug%252F1961799
Comment 3 Luca Miccini 2022-05-25 06:14:12 UTC 
wallaby cherry-picks:

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839789
https://review.opendev.org/c/openstack/tripleo-ansible/+/839738
Comment 4 Rabi Mishra 2022-05-30 08:39:01 UTC 
*** Bug 2078579 has been marked as a duplicate of this bug. ***
Comment 8 dabarzil 2022-06-21 12:59:49 UTC 
verified in:
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/Phase3/view/OSP%2017.0/view/PidOne/job/DFG-pidone-features-17.0_director-rhel-virthost-3cont_3data_3mess_3net_1comp-ipv4-geneve-tobiko_faults-composable_roles/
Comment 13 errata-xmlrpc 2022-09-21 12:20:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543