Bug 1528229

Created attachment 1370803 [details]
XML of VM

Description of problem:
This is doing NFV live migration with openvswitch and vhost-user single queue. Migration always fails, we hit below two kinds of issues:

(1) After migrating from src to des, the guest network can not recover at all.

(2) Sometimes the first migration from src to des fail with qemu core dump.

Version-Release number of selected component (if applicable):
3.10.0-823.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7.x86_64
libvirt-3.9.0-6.el7.x86_64
tuned-2.9.0-1.el7.noarch
openvswitch-2.9.0-0.1.20171212git6625e43.el7fdb.x86_64
dpdk-17.11-4.el7.x86_64

How reproducible:
2/5

Steps to Reproduce:
1. Boot ovs on src and des host, see[1]. 

2. Set multiple pmds for ovs, set totally 6 cpus as there are 6 ports.

ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x1554
ovs-vsctl set Interface dpdk0 options:n_rxq=1
ovs-vsctl set Interface dpdk1 options:n_rxq=1
ovs-vsctl set Interface dpdk2 options:n_rxq=1

3. Boot VM, see attachment.

4. In VM, start testpmd.

5. In another host, generate packets to VM.
./build/MoonGen examples/l2-load-latency.lua 0 1 5000


6. Do migration from src to des
# /bin/virsh migrate --verbose --persistent --live rhel7.5_nonrt qemu+ssh://192.168.1.2/system

7. Migration fails with 2 issue mentioned above.


Actual results:
(1) After migrating from src to des, the guest network can not recover at all. Related log look like below:

====cat /var/log/libvirt/qemu/rhel7.5_nonrt.log====
...
2017-12-21 10:17:34.912+0000: initiating migration
2017-12-21T10:17:35.268775Z qemu-kvm: Failed to read msg header. Read 0 instead of 12. Original request 6.
2017-12-21T10:17:35.268820Z qemu-kvm: vhost_set_log_base failed: Success (0)
2017-12-21T10:17:35.268827Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.268833Z qemu-kvm: vhost_set_features failed: Success (0)
2017-12-21T10:17:35.268838Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.268843Z qemu-kvm: vhost_set_vring_addr failed: Success (0)
2017-12-21T10:17:35.268849Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.268855Z qemu-kvm: vhost_set_vring_addr failed: Success (0)
2017-12-21T10:17:35.312092Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.312117Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Success (0)
2017-12-21T10:17:35.312133Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.312138Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Success (0)
2017-12-21T10:17:35.312554Z qemu-kvm: Failed to read from slave.
2017-12-21T10:17:35.312571Z qemu-kvm: Failed to read from slave.
2017-12-21T10:17:35.312618Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.312626Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2017-12-21T10:17:35.312633Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.312639Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2017-12-21T10:17:35.312984Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.312994Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2017-12-21T10:17:35.313001Z qemu-kvm: Failed to set msg fds.
2017-12-21T10:17:35.313006Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2017-12-21 10:17:44.416+0000: shutting down, reason=migrated
2017-12-21T10:17:44.417363Z qemu-kvm: terminating on signal 15 from pid 1338 (/usr/sbin/libvirtd)

====check host # dmesg====
...
[110599.158973] pmd90[31429]: segfault at 2 ip 000056488562d8e1 sp 00007f90cdff8780 error 4
[110599.159011] pmd91[31428]: segfault at 2 ip 000056488562f76a sp 00007f90ce7fb580 error 4
[110599.159014]  in ovs-vswitchd[564885465000+478000]

[110599.180449]  in ovs-vswitchd[564885465000+478000]


(2) Sometimes the first migration from src to des fail with qemu core dump.

2017-12-20 05:54:30.196+0000: initiating migration
2017-12-20T05:54:30.552693Z qemu-kvm: Failed to read msg header. Read -1 instead of 12. Original request 6.
2017-12-20T05:54:30.552746Z qemu-kvm: vhost_set_log_base failed: Input/output error (5)
2017-12-20T05:54:30.552780Z qemu-kvm: Failed to set msg fds.
2017-12-20T05:54:30.552787Z qemu-kvm: vhost_set_vring_addr failed: Invalid argument (22)
2017-12-20T05:54:30.552793Z qemu-kvm: Failed to set msg fds.
2017-12-20T05:54:30.552799Z qemu-kvm: vhost_set_vring_addr failed: Invalid argument (22)
2017-12-20T05:54:30.552804Z qemu-kvm: Failed to set msg fds.
2017-12-20T05:54:30.552810Z qemu-kvm: vhost_set_features failed: Invalid argument (22)
2017-12-20 05:54:30.769+0000: shutting down, reason=crashed


Expected results:
Migration should work well.


Additional info:
1. Without multiple pmds, doesn't hit this issue: Do 10 migration runs without step2, all migration runs work well.

2. This is a regression bug.
openvswitch-2.8.0-4.el7fdb.x86_64.rpm    work well 

Reference:
[1]
# ovs-vsctl show
b9357f5b-bb2c-429e-8e6d-b171d7242b7c
    Bridge "ovsbr1"
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
        Port "dpdk2"
            Interface "dpdk2"
                type: dpdk
                options: {dpdk-devargs="0000:06:00.0", n_rxq="1", n_txq="1"}
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
    Bridge "ovsbr0"
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {dpdk-devargs="0000:04:00.1", n_rxq="1", n_txq="1"}
        Port "vhost-user0"
            Interface "vhost-user0"
                type: dpdkvhostuser
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:04:00.0", n_rxq="1", n_txq="1"}