Description of problem: After finishing FFU on the undercloud (moving from 10 to 13), i proceed to check the functionality of removing/adding nodes. This functionality is not working, the new nodes request DHCP on PXE boot, but the undercloud is not offering them, and the nodes just complain about no DHCP offers. When debugging, i found that iptables is causing it. As soon as i disable iptables, i can pxe boot and introspect without problems. iptables version is: iptables-1.4.21-18.3.el7_4.x86_64 Undercloud on 13 has this iptables rules: [root@undercloud-0 stack]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination neutron-openvswi-INPUT all -- anywhere anywhere ironic-inspector udp -- anywhere anywhere udp dpt:bootps ACCEPT all -- anywhere anywhere /* 000 accept related established rules */ state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere /* 000 accept related established rules ipv4 */ state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere /* 001 accept all icmp */ state NEW ACCEPT icmp -- anywhere anywhere /* 001 accept all icmp ipv4 */ state NEW ACCEPT all -- anywhere anywhere /* 002 accept all to lo interface */ state NEW ACCEPT all -- anywhere anywhere /* 002 accept all to lo interface ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ssh /* 003 accept ssh */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ssh /* 003 accept ssh ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27019 /* 101 mongodb_config */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27019 /* 101 mongodb_config ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27018 /* 102 mongodb_sharding */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27018 /* 102 mongodb_sharding ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27017 /* 103 mongod */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27017 /* 103 mongod ipv4 */ state NEW ACCEPT udp -- anywhere anywhere multiport dports ntp /* 105 ntp */ state NEW ACCEPT udp -- anywhere anywhere multiport dports ntp /* 105 ntp ipv4 */ state NEW ACCEPT vrrp -- anywhere anywhere /* 106 vrrp */ state NEW ACCEPT vrrp -- anywhere anywhere /* 106 vrrp ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports snmp-tcp-port /* 107 haproxy stats */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports snmp-tcp-port /* 107 haproxy stats ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6379,26379 /* 108 redis */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6379,26379 /* 108 redis ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports smc-https,6800:6810 /* 110 ceph */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports smc-https,6800:6810 /* 110 ceph ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports commplex-main,13000,openstack-id,13357 /* 111 keystone */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports commplex-main,13000,openstack-id,13357 /* 111 keystone ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports armtechdaemon,sun-as-jpda,13292 /* 112 glance */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports armtechdaemon,sun-as-jpda,13292 /* 112 glance ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6080,13080,8773,13773,8774,13774,8775,13775 /* 113 nova */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6080,13080,8773,13773,8774,13774,8778,13778,8775,13775 /* 113 nova ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 9696,13696 /* 114 neutron server */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 9696,13696 /* 114 neutron server ipv4 */ state NEW ACCEPT udp -- anywhere anywhere multiport dports bootps /* 115 neutron dhcp input */ state NEW ACCEPT udp -- anywhere anywhere multiport dports bootps /* 115 neutron dhcp input ipv4 */ state NEW ACCEPT udp -- anywhere anywhere multiport dports 4789 /* 118 neutron vxlan networks */ state NEW ACCEPT udp -- anywhere anywhere multiport dports 4789 /* 118 neutron vxlan networks ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8776,13776 /* 119 cinder */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8776,13776 /* 119 cinder ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports iscsi-target /* 120 iscsi initiator */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports iscsi-target /* 120 iscsi initiator ipv4 */ state NEW ACCEPT tcp -- localhost anywhere multiport dports memcache /* 121 memcached */ state NEW ACCEPT tcp -- localhost anywhere multiport dports memcache /* 121 memcached ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports webcache,13808 /* 122 swift proxy */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports webcache,13808 /* 122 swift proxy ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rsync,x11,6001,6002 /* 123 swift storage */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rsync,x11,6001,6002 /* 123 swift storage ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8777,13777 /* 124 ceilometer */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8777,13777 /* 124 ceilometer ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports irdmi,13800,mcreport,13003,8004,13004 /* 125 heat */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports irdmi,13800,mcreport,13003,8004,13004 /* 125 heat ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports http,https /* 126 horizon */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports http,https /* 126 horizon ipv4 */ state NEW ACCEPT udp -- anywhere anywhere multiport dports snmp /* 127 snmp */ state NEW ACCEPT udp -- anywhere anywhere multiport dports snmp /* 127 snmp ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports fs-agent,13042 /* 128 aodh */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports fs-agent,13042 /* 128 aodh ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8041,13041 /* 129 gnocchi-api */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8041,13041 /* 129 gnocchi-api ipv4 */ state NEW ACCEPT udp -- anywhere anywhere multiport dports tftp /* 130 tftp */ state NEW ACCEPT udp -- anywhere anywhere multiport dports tftp /* 130 tftp ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rfb:cvsup /* 131 novnc */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rfb:cvsup /* 131 novnc ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports sunwebadmins,13989 /* 132 mistral */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports sunwebadmins,13989 /* 132 mistral ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ddi-tcp-1,13888 /* 133 zaqar */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ddi-tcp-1,13888 /* 133 zaqar ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports cslistener /* 134 zaqar websockets */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports cslistener /* 134 zaqar websockets ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6385,13385 /* 135 ironic */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6385,13385 /* 135 ironic ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8779,13779 /* 136 trove */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8779,13779 /* 136 trove ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports mmcc /* 137 ironic-inspector */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports mmcc /* 137 ironic-inspector ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports msgsrvr /* 138 docker registry */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports msgsrvr,13787 /* 138 docker registry ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports radan-http /* 139 apache vhost */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports radan-http /* 139 apache vhost ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports hbci /* 142 tripleo-ui */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports hbci,https /* 142 tripleo-ui ipv4 */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8977,13977 state NEW /* 143 panko-api ipv4 */ LOG all -- anywhere anywhere /* 998 log all */ state NEW LOG level warning LOG all -- anywhere anywhere /* 998 log all ipv4 */ state NEW LOG level warning DROP all -- anywhere anywhere /* 999 drop all */ state NEW DROP all -- anywhere anywhere /* 999 drop all ipv4 */ state NEW Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-FORWARD all -- anywhere anywhere ACCEPT all -- anywhere 192.0.2.0/24 state NEW /* 140 destination ctlplane-subnet cidr nat ipv4 */ DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere 192.0.2.0/24 /* 140 destination network cidr nat */ state NEW ACCEPT all -- anywhere 192.0.2.0/24 /* 140 destination network cidr nat ipv4 */ state NEW ACCEPT all -- 192.0.2.0/24 anywhere state NEW /* 140 source ctlplane-subnet cidr nat ipv4 */ ACCEPT all -- 192.0.2.0/24 anywhere /* 140 source network cidr nat */ state NEW ACCEPT all -- 192.0.2.0/24 anywhere /* 140 source network cidr nat ipv4 */ state NEW ACCEPT tcp -- anywhere 192.168.122.0/24 /* 141 libvirt network nat */ state NEW ACCEPT tcp -- anywhere 192.168.122.0/24 /* 141 libvirt network nat ipv4 */ state NEW Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-OUTPUT all -- anywhere anywhere ACCEPT udp -- anywhere anywhere multiport dports bootpc /* 116 neutron dhcp output */ state NEW ACCEPT udp -- anywhere anywhere multiport dports bootpc /* 116 neutron dhcp output ipv4 */ state NEW Chain DOCKER (1 references) target prot opt source destination Chain DOCKER-ISOLATION (1 references) target prot opt source destination RETURN all -- anywhere anywhere Chain ironic-inspector (1 references) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain neutron-filter-top (2 references) target prot opt source destination neutron-openvswi-local all -- anywhere anywhere Chain neutron-openvswi-FORWARD (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out tap2ad2286d-71 --physdev-is-bridged /* Accept all packets when port is trusted. */ Chain neutron-openvswi-INPUT (1 references) target prot opt source destination Chain neutron-openvswi-OUTPUT (1 references) target prot opt source destination Chain neutron-openvswi-local (1 references) target prot opt source destination Chain neutron-openvswi-sg-chain (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain neutron-openvswi-sg-fallback (0 references) target prot opt source destination DROP all -- anywhere anywhere /* Default drop rule for unmatched traffic. */
Created attachment 1425876 [details] sosreport on 13
In my initial newton deployment i get: rpm -qa | grep iptables iptables-services-1.4.21-18.3.el7_4.x86_64 iptables-1.4.21-18.3.el7_4.x86_64 Chain INPUT (policy ACCEPT) target prot opt source destination neutron-openvswi-INPUT all -- anywhere anywhere ironic-inspector udp -- anywhere anywhere udp dpt:bootps ACCEPT all -- anywhere anywhere /* 000 accept related established rules */ state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere /* 001 accept all icmp */ state NEW ACCEPT all -- anywhere anywhere /* 002 accept all to lo interface */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ssh /* 003 accept ssh */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27019 /* 101 mongodb_config */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27018 /* 102 mongodb_sharding */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 27017 /* 103 mongod */ state NEW ACCEPT udp -- anywhere anywhere multiport dports ntp /* 105 ntp */ state NEW ACCEPT vrrp -- anywhere anywhere /* 106 vrrp */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports snmp-tcp-port /* 107 haproxy stats */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6379,26379 /* 108 redis */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports smc-https,6800:6810 /* 110 ceph */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports commplex-main,13000,openstack-id,13357 /* 111 keystone */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports armtechdaemon,sun-as-jpda,13292 /* 112 glance */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6080,13080,8773,13773,8774,13774,8775,13775 /* 113 nova */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 9696,13696 /* 114 neutron server */ state NEW ACCEPT udp -- anywhere anywhere multiport dports bootps /* 115 neutron dhcp input */ state NEW ACCEPT udp -- anywhere anywhere multiport dports 4789 /* 118 neutron vxlan networks */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8776,13776 /* 119 cinder */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports iscsi-target /* 120 iscsi initiator */ state NEW ACCEPT tcp -- localhost anywhere multiport dports memcache /* 121 memcached */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports webcache,13808 /* 122 swift proxy */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rsync,x11,6001,6002 /* 123 swift storage */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8777,13777 /* 124 ceilometer */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports irdmi,13800,mcreport,13003,8004,13004 /* 125 heat */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports http,https /* 126 horizon */ state NEW ACCEPT udp -- anywhere anywhere multiport dports snmp /* 127 snmp */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports fs-agent,13042 /* 128 aodh */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8041,13041 /* 129 gnocchi-api */ state NEW ACCEPT udp -- anywhere anywhere multiport dports tftp /* 130 tftp */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports rfb:cvsup /* 131 novnc */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports sunwebadmins,13989 /* 132 mistral */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports ddi-tcp-1,13888 /* 133 zaqar */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports cslistener /* 134 zaqar websockets */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 6385,13385 /* 135 ironic */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports 8779,13779 /* 136 trove */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports mmcc /* 137 ironic-inspector */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports msgsrvr /* 138 docker registry */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports radan-http /* 139 apache vhost */ state NEW ACCEPT tcp -- anywhere anywhere multiport dports hbci /* 142 tripleo-ui */ state NEW LOG all -- anywhere anywhere /* 998 log all */ state NEW LOG level warning DROP all -- anywhere anywhere /* 999 drop all */ state NEW Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-FORWARD all -- anywhere anywhere DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere 192.0.2.0/24 /* 140 destination network cidr nat */ state NEW ACCEPT all -- 192.0.2.0/24 anywhere /* 140 source network cidr nat */ state NEW ACCEPT tcp -- anywhere 192.168.122.0/24 /* 141 libvirt network nat */ state NEW Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- anywhere anywhere neutron-openvswi-OUTPUT all -- anywhere anywhere ACCEPT udp -- anywhere anywhere multiport dports bootpc /* 116 neutron dhcp output */ state NEW Chain DOCKER (1 references) target prot opt source destination Chain DOCKER-ISOLATION (1 references) target prot opt source destination RETURN all -- anywhere anywhere Chain ironic-inspector (1 references) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain neutron-filter-top (2 references) target prot opt source destination neutron-openvswi-local all -- anywhere anywhere Chain neutron-openvswi-FORWARD (1 references) target prot opt source destination Chain neutron-openvswi-INPUT (1 references) target prot opt source destination Chain neutron-openvswi-OUTPUT (1 references) target prot opt source destination Chain neutron-openvswi-local (1 references) target prot opt source destination Chain neutron-openvswi-sg-chain (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain neutron-openvswi-sg-fallback (0 references) target prot opt source destination DROP all -- anywhere anywhere /* Default drop rule for unmatched traffic. */
Created attachment 1425979 [details] sosreport on 10
This is due to the fact that the dhcp filtering used for the inspector dnsmasq changed from iptables-based to dnsmasq-based in OSP-13. This rule still remains on upgrade however and is what is causing the problem: Chain ironic-inspector (1 references) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-port-unreachable This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1570386, I will close as duplicate. *** This bug has been marked as a duplicate of bug 1570386 ***
I applied the suggested changes and i can introspect now