Description of problem: Live migration of instance fails with: 2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host Version-Release number of selected component (if applicable): openstack-heat-templates-0.0.1-0.20161004223740.f123aa1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with 2 compute nodes 2. Live migrate instance from one host to another Actual results: Live migration fails with the following error in /var/log/nova/nova-compute.log: 2016-10-20 18:29:41.470 9410 ERROR nova.virt.libvirt.driver [req-abe8c10c-a496-49dd-b414-fa0fe0a66a6f 2cc73868cdb84c85a142794fe852c7ed b424e6e8863240f7a5edb629a5db834d - - -] [instance: 029ad115-38f8-49e7-89f1-7161d76b0ed3] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://comp-r00-01.redhat.local/system: unable to connect to server at 'comp-r00-01.redhat.local:16509': No route to host Expected results: Additional info: From source to destination: [heat-admin@comp-r00-00 ~]$ nc comp-r00-01.redhat.local 16509 Ncat: No route to host. On the destination host: [heat-admin@comp-r00-01 ~]$ nc comp-r00-01.redhat.local 16509 Ncat: Broken pipe. iptables rules: [heat-admin@comp-r00-01 ~]$ sudo iptables -nL Chain INPUT (policy ACCEPT) target prot opt source destination neutron-openvswi-INPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 000 accept related established rules */ state RELATED,ESTABLISHED ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 /* 001 accept all icmp */ state NEW ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 002 accept all to lo interface */ state NEW ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 22 /* 003 accept ssh */ state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 123 /* 105 ntp */ state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 4789 /* 118 neutron vxlan networks */ state NEW ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 161 /* 127 snmp */ state NEW ACCEPT 47 -- 0.0.0.0/0 0.0.0.0/0 /* 136 neutron gre networks */ state NEW ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited LOG all -- 0.0.0.0/0 0.0.0.0/0 /* 998 log all */ LOG flags 0 level 4 DROP all -- 0.0.0.0/0 0.0.0.0/0 /* 999 drop all */ state NEW Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-filter-top (2 references) target prot opt source destination neutron-openvswi-local all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-FORWARD (1 references) target prot opt source destination neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */ Chain neutron-openvswi-INPUT (1 references) target prot opt source destination neutron-openvswi-o2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */ Chain neutron-openvswi-OUTPUT (1 references) target prot opt source destination Chain neutron-openvswi-i2b63863d-d (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN udp -- 172.16.19.11 0.0.0.0/0 udp spt:67 udp dpt:68 RETURN udp -- 172.16.19.10 0.0.0.0/0 udp spt:67 udp dpt:68 RETURN udp -- 172.16.19.12 0.0.0.0/0 udp spt:67 udp dpt:68 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-openvswi-local (1 references) target prot opt source destination Chain neutron-openvswi-o2b63863d-d (2 references) target prot opt source destination RETURN udp -- 0.0.0.0 255.255.255.255 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ neutron-openvswi-s2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 /* Allow DHCP client traffic. */ DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 udp dpt:68 /* Prevent DHCP Spoofing by VM. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ Chain neutron-openvswi-s2b63863d-d (1 references) target prot opt source destination RETURN all -- 172.16.19.19 0.0.0.0/0 MAC FA:16:3E:E1:CC:59 /* Allow traffic from defined IP/MAC pairs. */ DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Drop traffic without an IP/MAC allow rule. */ Chain neutron-openvswi-sg-chain (2 references) target prot opt source destination neutron-openvswi-i2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */ neutron-openvswi-o2b63863d-d all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2b63863d-d4 --physdev-is-bridged /* Jump to the VM specific chain. */ ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-sg-fallback (2 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 /* Default drop rule for unmatched traffic. */
[root@comp-r00-01 heat-admin]# iptables -nL | grep 16509 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 16509 /* 200 nova_libvirt */ state NEW
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html