Description of problem: Overcloud deployment using composable roles(database,messaging,networker, controllerOpenstack,compute) hangs on `Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3` The step_3 starts *_api_db_sync containers. They are timing out because they can't connect to mysql. It seems there is no iptables rules on controller nodes to allow database traffic for haproxy(running on controllers): [heat-admin@controller-2 ~]$ sudo iptables-save [22/1298] # Generated by iptables-save v1.8.4 on Tue Apr 12 12:49:53 2022 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :PREROUTING - [0:0] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "000 accept related established rules ipv4" -j ACCEPT -A INPUT -p icmp -m conntrack --ctstate NEW -m comment --comment "001 accept all icmp ipv4" -j ACCEPT -A INPUT -i lo -m conntrack --ctstate NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT -A INPUT -s 192.168.24.0/24 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m comment --comment "003 accept ssh from ctlplane subnet 192.168.24.0/24 ipv4" -j ACCEPT -A INPUT -p udp -m udp --dport 123 -m conntrack --ctstate NEW -m comment --comment "105 ntp ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 3300 -m conntrack --ctstate NEW -m comment --comment "110 ceph_mon ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 6789 -m conntrack --ctstate NEW -m comment --comment "110 ceph_mon ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 35357 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13000 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 5000 -m conntrack --ctstate NEW -m comment --comment "111 keystone ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13292 -m conntrack --ctstate NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 9292 -m conntrack --ctstate NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 6800:7300 -m conntrack --ctstate NEW -m comment --comment "113 ceph_mgr ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13774 -m conntrack --ctstate NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8774 -m conntrack --ctstate NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13696 -m conntrack --ctstate NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 9696 -m conntrack --ctstate NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT -A INPUT -p udp -m udp --dport 4789 -m conntrack --ctstate NEW -m comment --comment "118 neutron vxlan networks ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13776 -m conntrack --ctstate NEW -m comment --comment "119 cinder ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8776 -m conntrack --ctstate NEW -m comment --comment "119 cinder ipv4" -j ACCEPT -A INPUT -p udp -m udp --dport 6081 -m conntrack --ctstate NEW -m comment --comment "119 neutron geneve networks ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 3260 -m conntrack --ctstate NEW -m comment --comment "120 iscsi initiator ipv4" -j ACCEPT -A INPUT -s 172.17.1.0/24 -p tcp -m tcp --dport 11211 -m conntrack --ctstate NEW -m comment --comment "121 memcached 172.17.1.0/24 ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 6642 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 6641 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 3125 -m conntrack --ctstate NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13808 -m conntrack --ctstate NEW -m comment --comment "122 ceph rgw ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW -m comment --comment "122 ceph rgw ipv4" -j ACCEPT -A INPUT -s 192.168.24.0/24 -p udp -m udp --dport 161 -m conntrack --ctstate NEW -m comment --comment "124 snmp 192.168.24.0/24 ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13004 -m conntrack --ctstate NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8004 -m conntrack --ctstate NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13800 -m conntrack --ctstate NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8000 -m conntrack --ctstate NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -m comment --comment "126 horizon ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -m comment --comment "126 horizon ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 21064 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 3121 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 2224 -m conntrack --ctstate NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT -A INPUT -p udp -m udp --dport 5405 -m conntrack --ctstate NEW -m comment --comment "131 pacemaker udp ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13080 -m conntrack --ctstate NEW -m comment --comment "137 nova_vnc_proxy ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 6080 -m conntrack --ctstate NEW -m comment --comment "137 nova_vnc_proxy ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13778 -m conntrack --ctstate NEW -m comment --comment "138 placement ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8778 -m conntrack --ctstate NEW -m comment --comment "138 placement ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 13775 -m conntrack --ctstate NEW -m comment --comment "139 nova_metadata ipv4" -j ACCEPT -A INPUT -p tcp -m tcp --dport 8775 -m conntrack --ctstate NEW -m comment --comment "139 nova_metadata ipv4" -j ACCEPT -A INPUT -m conntrack --ctstate NEW -m limit --limit 20/min --limit-burst 15 -m comment --comment "998 log all ipv4" -j LOG -A INPUT -m conntrack --ctstate NEW -m comment --comment "999 drop all ipv4" -j DROP Version-Release number of selected component (if applicable): RHOS-17.0-RHEL-8-20220401.n.1 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
upstream bz: https://bugs.launchpad.net/tripleo/+bug/1961799 wip patches (not sure how up to date they are) https://review.opendev.org/q/topic:bug%252F1961799
wallaby cherry-picks: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839789 https://review.opendev.org/c/openstack/tripleo-ansible/+/839738
*** Bug 2078579 has been marked as a duplicate of this bug. ***
verified in: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/Phase3/view/OSP%2017.0/view/PidOne/job/DFG-pidone-features-17.0_director-rhel-virthost-3cont_3data_3mess_3net_1comp-ipv4-geneve-tobiko_faults-composable_roles/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543