Bug 1387390

Summary:	Port TCP 16509 is not allowed in compute firewall which breaks instance live migration
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	James Slagle <jslagle>
Status:	CLOSED ERRATA	QA Contact:	Marius Cornea <mcornea>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	dbecker, dnavale, jschluet, jslagle, mburns, morazi, rhel-osp-director-maint, sclewis
Target Milestone:	rc	Keywords:	Regression, Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-0.8.0rc3.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, the TCP port '16509' was blocked in 'iptables'. As a result, the 'nova' Compute 'libvirt' instances could not be live migrated between Compute nodes. With this update, TCP port '16509' is configured to be opened in the 'iptables'. As a result, the 'nova' Compute 'libvirt' instances can now be live migrated between Compute nodes.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 16:24:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-10-20 18:38:05 UTC

Description of problem:
Live migration of instance fails with:

2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host


Version-Release number of selected component (if applicable):
openstack-heat-templates-0.0.1-0.20161004223740.f123aa1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with 2 compute nodes
2. Live migrate instance from one host to another

Actual results:
Live migration fails with the following error in /var/log/nova/nova-compute.log:
2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host

Expected results:


Additional info:
From source to destination:
[heat-admin@comp-r00-00 ~]$ nc comp-r00-01.redhat.local 16509
Ncat: No route to host.

On the destination host:
[heat-admin@comp-r00-01 ~]$ nc comp-r00-01.redhat.local 16509
Ncat: Broken pipe.

iptables rules:
[heat-admin@comp-r00-01 ~]$ sudo iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-openvswi-INPUT  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* 000 accept related established rules */ state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            /* 001 accept all icmp */ state NEW
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface */ state NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 003 accept ssh */ state NEW
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 123 /* 105 ntp */ state NEW
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 4789 /* 118 neutron vxlan networks */ state NEW
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 161 /* 127 snmp */ state NEW
ACCEPT     47   --  0.0.0.0/0            0.0.0.0/0            /* 136 neutron gre networks */ state NEW
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
LOG        all  --  0.0.0.0/0            0.0.0.0/0            /* 998 log all */ LOG flags 0 level 4
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* 999 drop all */ state NEW

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
neutron-filter-top  all  --  0.0.0.0/0            0.0.0.0/0           
neutron-openvswi-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-filter-top  all  --  0.0.0.0/0            0.0.0.0/0           
neutron-openvswi-OUTPUT  all  --  0.0.0.0/0            0.0.0.0/0           

Chain neutron-filter-top (2 references)
target     prot opt source               destination         
neutron-openvswi-local  all  --  0.0.0.0/0            0.0.0.0/0           

Chain neutron-openvswi-FORWARD (1 references)
target     prot opt source               destination         
neutron-openvswi-sg-chain  all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
neutron-openvswi-sg-chain  all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */

Chain neutron-openvswi-INPUT (1 references)
target     prot opt source               destination         
neutron-openvswi-o2b63863d-d  all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */

Chain neutron-openvswi-OUTPUT (1 references)
target     prot opt source               destination         

Chain neutron-openvswi-i2b63863d-d (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
RETURN     udp  --  172.16.19.11         0.0.0.0/0            udp spt:67 udp dpt:68
RETURN     udp  --  172.16.19.10         0.0.0.0/0            udp spt:67 udp dpt:68
RETURN     udp  --  172.16.19.12         0.0.0.0/0            udp spt:67 udp dpt:68
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
RETURN     icmp --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
neutron-openvswi-sg-fallback  all  --  0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-openvswi-local (1 references)
target     prot opt source               destination         

Chain neutron-openvswi-o2b63863d-d (2 references)
target     prot opt source               destination         
RETURN     udp  --  0.0.0.0              255.255.255.255      udp spt:68 dpt:67 /* Allow DHCP client traffic. */
neutron-openvswi-s2b63863d-d  all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:68 dpt:67 /* Allow DHCP client traffic. */
DROP       udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:67 udp dpt:68 /* Prevent DHCP Spoofing by VM. */
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
neutron-openvswi-sg-fallback  all  --  0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-openvswi-s2b63863d-d (1 references)
target     prot opt source               destination         
RETURN     all  --  172.16.19.19         0.0.0.0/0            MAC FA:16:3E:E1:CC:59 /* Allow traffic from defined IP/MAC pairs. */
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* Drop traffic without an IP/MAC allow rule. */

Chain neutron-openvswi-sg-chain (2 references)
target     prot opt source               destination         
neutron-openvswi-i2b63863d-d  all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */
neutron-openvswi-o2b63863d-d  all  --  0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

Chain neutron-openvswi-sg-fallback (2 references)
target     prot opt source               destination         
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* Default drop rule for unmatched traffic. */

Comment 4 Marius Cornea 2016-10-31 07:37:05 UTC

[root@comp-r00-01 heat-admin]# iptables -nL | grep 16509
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 16509 /* 200 nova_libvirt */ state NEW

Comment 6 errata-xmlrpc 2016-12-14 16:24:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html