Bug 1570386

Summary:	Undercloud: upgrade to OSP-13 break introspection
Product:	Red Hat OpenStack	Reporter:	Harald Jensås <hjensas>
Component:	puppet-tripleo	Assignee:	Harald Jensås <hjensas>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	high	Docs Contact:
Priority:	medium
Version:	13.0 (Queens)	CC:	augol, bfournie, jjoyce, jschluet, mburns, mcornea, rrasouli, sasha, slinaber, tvignaud, yprokule, yroblamo
Target Milestone:	beta	Keywords:	Triaged
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	instack-undercloud-8.4.1-2.el7ost puppet-tripleo-8.3.2-3.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-27 13:52:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Harald Jensås 2018-04-22 13:38:52 UTC

Description of problem:
Between Queens and Pike we switch the Ironic Inspector PXE filter driver from iptables in Pike to dnsmasq in Queens.

The old iptables driver created a firewall chain, and will in most cases configure a REJECT rule to block any introspection unless the operator start introspection of nodes.

On the upgraded undercloud we have these left-over rules still there:

  357 183K ironic-inspector udp -- br-ctlplane any anywhere anywhere udp dpt:bootps

Chain ironic-inspector (1 references) pkts bytes target prot opt in out source destination
357 183K REJECT all -- any any anywhere anywhere reject-with icmp-port-unreachable


Version-Release number of selected component (if applicable):
instack-undercloud-8.4.0-4.el7ost.noarch
openstack-ironic-inspector-7.2.1-0.20180409163359.2435d97.el7ost.noarch

How reproducible:

Steps to Reproduce:
1. Install Pike Undercloud
2. Upgrade to Queens Undercloud


Actual results:
Intrspection fails.
iptables command reveal that the rule to REJECT any DHCP request is still in place.

We can see that the rules are in the saved IP tables rules:
[stack@uc-upgrade ~]$ sudo grep ironic-inspector /etc/sysconfig/iptables
:ironic-inspector - [0:0]
-A INPUT -i br-ctlplane -p udp -m udp --dport 67 -j ironic-inspector
-A INPUT -p tcp -m multiport --dports 5050 -m state --state NEW -m comment --comment "137 ironic-inspector ipv4" -j ACCEPT
-A ironic-inspector -j REJECT --reject-with icmp-port-unreachable

Expected results:
The ephemeral firewall rules created by Ironic Inspector iptables PXE Filter driver should not be persisted.

Also, Ironic Inspectour should clean up these rules on shutdown.

Additional info:

Comment 1 Bob Fournier 2018-04-23 14:13:34 UTC

*** Bug 1567917 has been marked as a duplicate of this bug. ***

Comment 2 Bob Fournier 2018-04-24 12:33:15 UTC

This was also seen on upgrade from OSP-10 to OSP-13 - https://bugzilla.redhat.com/show_bug.cgi?id=1571182

Comment 3 Bob Fournier 2018-04-24 12:33:44 UTC

*** Bug 1571182 has been marked as a duplicate of this bug. ***

Comment 4 Bob Fournier 2018-04-25 18:18:17 UTC

As the instack-undercloud and puppet-tripleo patches are necessary to fix this issue in OSP-13 and the ironic-inspector patch is really only effective on the version being installed FROM, removing the ironic-inspector patch and moving to POST as the 2 other patches have merged in stable/queens.

Comment 12 Ronnie Rasouli 2018-05-10 07:02:36 UTC

Looks like the issue has been resolved, the ironic hasn't been blocking the dhcp traffic

iptables -L INPUT -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  17M   35G neutron-openvswi-INPUT  all  --  any    any     anywhere             anywhere
  16M   35G ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED /* 000 accept related established rules ipv4 */
    8   408 ACCEPT     icmp --  any    any     anywhere             anywhere             state NEW /* 001 accept all icmp ipv4 */
 772K   46M ACCEPT     all  --  lo     any     anywhere             anywhere             state NEW /* 002 accept all to lo interface ipv4 */
    4   240 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports ssh state NEW /* 003 accept ssh ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports fs-agent state NEW /* 100 aodh_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13042 state NEW /* 100 aodh_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8777 state NEW /* 100 ceilometer_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13777 state NEW /* 100 ceilometer_haproxy_ssl ipv4 */
   18  1080 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports msgsrvr state NEW /* 100 docker-registry_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13787 state NEW /* 100 docker-registry_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports armtechdaemon state NEW /* 100 glance_api_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13292 state NEW /* 100 glance_api_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8041 state NEW /* 100 gnocchi_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13041 state NEW /* 100 gnocchi_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8004 state NEW /* 100 heat_api_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13004 state NEW /* 100 heat_api_haproxy_ssl ipv4 */
    1    60 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports mmcc state NEW /* 100 ironic-inspector_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13050 state NEW /* 100 ironic-inspector_haproxy_ssl ipv4 */
    1    60 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 6385 state NEW /* 100 ironic_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13385 state NEW /* 100 ironic_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports openstack-id state NEW /* 100 keystone_admin_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports commplex-main state NEW /* 100 keystone_public_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13000 state NEW /* 100 keystone_public_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports sunwebadmins state NEW /* 100 mistral_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13989 state NEW /* 100 mistral_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 9696 state NEW /* 100 neutron_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13696 state NEW /* 100 neutron_haproxy_ssl ipv4 */
 2133  128K ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8775 state NEW /* 100 nova_metadata_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8774 state NEW /* 100 nova_osapi_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13774 state NEW /* 100 nova_osapi_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8778 state NEW /* 100 nova_placement_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13778 state NEW /* 100 nova_placement_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8977 state NEW /* 100 panko_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13977 state NEW /* 100 panko_haproxy_ssl ipv4 */
19743 1185K ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports webcache state NEW /* 100 swift_proxy_server_haproxy ipv4 */
    9   540 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13808 state NEW /* 100 swift_proxy_server_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports hbci state NEW /* 100 ui_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports https state NEW /* 100 ui_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports ddi-tcp-1 state NEW /* 100 zaqar_api_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 13888 state NEW /* 100 zaqar_api_haproxy_ssl ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports cslistener state NEW /* 100 zaqar_ws_haproxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports cslistener state NEW /* 100 zaqar_ws_haproxy_ssl ipv4 */
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             multiport dports ntp state NEW /* 105 ntp ipv4 */
    0     0 ACCEPT     vrrp --  any    any     anywhere             anywhere             state NEW /* 106 vrrp ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports snmp-tcp-port state NEW /* 107 haproxy stats ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 6379,26379 state NEW /* 108 redis ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports smc-https,6800:6810 state NEW /* 110 ceph ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports commplex-main,13000,openstack-id,13357 state NEW /* 111 keystone ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports armtechdaemon,sun-as-jpda,13292 state NEW /* 112 glance ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 6080,13080,8773,13773,8774,13774,8778,13778,8775,13775 state NEW /* 113 nova ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 9696,13696 state NEW /* 114 neutron server ipv4 */
   22  7624 ACCEPT     udp  --  any    any     anywhere             anywhere             multiport dports bootps state NEW /* 115 neutron dhcp input ipv4 */
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             multiport dports 4789 state NEW /* 118 neutron vxlan networks ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8776,13776 state NEW /* 119 cinder ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports iscsi-target state NEW /* 120 iscsi initiator ipv4 */
    0     0 ACCEPT     tcp  --  any    any     localhost            anywhere             multiport dports memcache state NEW /* 121 memcached ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports webcache,13808 state NEW /* 122 swift proxy ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports rsync,x11,6001,6002 state NEW /* 123 swift storage ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8777,13777 state NEW /* 124 ceilometer ipv4 */
   64  3840 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports irdmi,13800,mcreport,13003,8004,13004 state NEW /* 125 heat ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports http,https state NEW /* 126 horizon ipv4 */
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             multiport dports snmp state NEW /* 127 snmp ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports fs-agent,13042 state NEW /* 128 aodh ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8041,13041 state NEW /* 129 gnocchi-api ipv4 */
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             multiport dports tftp state NEW /* 130 tftp ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports rfb:cvsup state NEW /* 131 novnc ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports sunwebadmins,13989 state NEW /* 132 mistral ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports ddi-tcp-1,13888 state NEW /* 133 zaqar ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports cslistener state NEW /* 134 zaqar websockets ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 6385,13385 state NEW /* 135 ironic ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8779,13779 state NEW /* 136 trove ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports mmcc state NEW /* 137 ironic-inspector ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports msgsrvr,13787 state NEW /* 138 docker registry ipv4 */
    2   128 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports radan-http state NEW /* 139 apache vhost ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports hbci,https state NEW /* 142 tripleo-ui ipv4 */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             multiport dports 8977,13977 state NEW /* 143 panko-api ipv4 */
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere
   58  2848 ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             state NEW tcp dpt:ssh
    0     0 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-host-prohibited
    0     0 LOG        all  --  any    any     anywhere             anywhere             state NEW /* 998 log all ipv4 */ LOG level warning
    0     0 DROP       all  --  any    any     anywhere             anywhere             state NEW /* 999 drop all ipv4 */

Comment 13 Ronnie Rasouli 2018-05-10 14:04:45 UTC

versions:
instack-undercloud-8.4.1-4.el7ost.noarch
puppet-tripleo-8.3.2-5.el7ost.noarch

Comment 15 Alexander Chuzhoy 2018-05-11 13:40:31 UTC

Verified based on comment #12 and comment #13.
Ronnie did the check after upgrade.

Comment 16 Yolanda Robla 2018-05-22 09:06:24 UTC

I continue hitting that in the context of FFU. I upgraded undercloud from 10 to 13, and left overcloud on 10. At this point, i remove a node and i add a new one, and introspection is not working.
I had to stop iptables service, and then i could introspect. That's the content of iptables -L:

(undercloud) [stack@undercloud-0 ~]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-openvswi-INPUT  all  --  anywhere             anywhere            
ironic-inspector  udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             /* 000 accept related established rules ipv4 */ state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere             /* 001 accept all icmp ipv4 */ state NEW
ACCEPT     all  --  anywhere             anywhere             /* 002 accept all to lo interface ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports ssh /* 003 accept ssh ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 27019 /* 101 mongodb_config ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 27018 /* 102 mongodb_sharding ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 27017 /* 103 mongod ipv4 */ state NEW
ACCEPT     udp  --  anywhere             anywhere             multiport dports ntp /* 105 ntp ipv4 */ state NEW
ACCEPT     vrrp --  anywhere             anywhere             /* 106 vrrp ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports snmp-tcp-port /* 107 haproxy stats ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 6379,26379 /* 108 redis ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports smc-https,6800:6810 /* 110 ceph ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports commplex-main,13000,openstack-id,13357 /* 111 keystone ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports armtechdaemon,sun-as-jpda,13292 /* 112 glance ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 6080,13080,8773,13773,8774,13774,8778,13778,8775,13775 /* 113 nova ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 9696,13696 /* 114 neutron server ipv4 */ state NEW
ACCEPT     udp  --  anywhere             anywhere             multiport dports bootps /* 115 neutron dhcp input ipv4 */ state NEW
ACCEPT     udp  --  anywhere             anywhere             multiport dports 4789 /* 118 neutron vxlan networks ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 8776,13776 /* 119 cinder ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports iscsi-target /* 120 iscsi initiator ipv4 */ state NEW
ACCEPT     tcp  --  localhost            anywhere             multiport dports memcache /* 121 memcached ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports webcache,13808 /* 122 swift proxy ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports rsync,x11,6001,6002 /* 123 swift storage ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 8777,13777 /* 124 ceilometer ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports irdmi,13800,mcreport,13003,8004,13004 /* 125 heat ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports http,https /* 126 horizon ipv4 */ state NEW
ACCEPT     udp  --  anywhere             anywhere             multiport dports snmp /* 127 snmp ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports fs-agent,13042 /* 128 aodh ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 8041,13041 /* 129 gnocchi-api ipv4 */ state NEW
ACCEPT     udp  --  anywhere             anywhere             multiport dports tftp /* 130 tftp ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports rfb:cvsup /* 131 novnc ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports sunwebadmins,13989 /* 132 mistral ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports ddi-tcp-1,13888 /* 133 zaqar ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports cslistener /* 134 zaqar websockets ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 6385,13385 /* 135 ironic ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 8779,13779 /* 136 trove ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports mmcc /* 137 ironic-inspector ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports msgsrvr,13787 /* 138 docker registry ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports radan-http /* 139 apache vhost ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports hbci,https /* 142 tripleo-ui ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 8977,13977 state NEW /* 143 panko-api ipv4 */
LOG        all  --  anywhere             anywhere             /* 998 log all ipv4 */ state NEW LOG level warning
DROP       all  --  anywhere             anywhere             /* 999 drop all ipv4 */ state NEW

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
neutron-filter-top  all  --  anywhere             anywhere            
neutron-openvswi-FORWARD  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             192.0.2.0/24         state NEW /* 140 destination ctlplane-subnet cidr nat ipv4 */
DOCKER-ISOLATION  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             192.0.2.0/24         /* 140 destination network cidr nat ipv4 */ state NEW
ACCEPT     all  --  192.0.2.0/24         anywhere             state NEW /* 140 source ctlplane-subnet cidr nat ipv4 */
ACCEPT     all  --  192.0.2.0/24         anywhere             /* 140 source network cidr nat ipv4 */ state NEW
ACCEPT     tcp  --  anywhere             192.168.122.0/24     /* 141 libvirt network nat ipv4 */ state NEW

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-filter-top  all  --  anywhere             anywhere            
neutron-openvswi-OUTPUT  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             multiport dports bootpc /* 116 neutron dhcp output ipv4 */ state NEW

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain ironic-inspector (1 references)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain neutron-filter-top (2 references)
target     prot opt source               destination         
neutron-openvswi-local  all  --  anywhere             anywhere            

Chain neutron-openvswi-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap6439efa9-cd --physdev-is-bridged /* Accept all packets when port is trusted. */

Chain neutron-openvswi-INPUT (1 references)
target     prot opt source               destination         

Chain neutron-openvswi-OUTPUT (1 references)
target     prot opt source               destination         

Chain neutron-openvswi-local (1 references)
target     prot opt source               destination         

Chain neutron-openvswi-sg-chain (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain neutron-openvswi-sg-fallback (0 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* Default drop rule for unmatched traffic. */

Comment 17 Yolanda Robla 2018-05-22 09:35:01 UTC

An undercloud reboot fixed the issue...

Comment 18 Harald Jensås 2018-05-22 11:26:48 UTC

Yes, a reboot or a restarting iptables + neutron agent services should fix this issue.

The problem here is that the ephemeral rules managed by ironic-inspector is not torn down when the openstack-ironic-inspector service is stopped. This is due to a bug in ironic-inspector that causees the service to immediately exit when reciving TERM signal, instead of executuing the proper shutdown method that is supposed to do the clean up.

Backport this change would fix the issue: https://review.openstack.org/563335

But for FFU, maby it is better to just document the reboot/service restart requirement?

Comment 19 Amit Ugol 2018-06-10 07:42:40 UTC

verified so removing needinfo flag.

Comment 21 errata-xmlrpc 2018-06-27 13:52:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086