RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1444785 - The second migration fails over openvswitch+dpdk
Summary: The second migration fails over openvswitch+dpdk
Keywords:
Status: CLOSED DUPLICATE of bug 1441165
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: yafu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-24 09:53 UTC by Pei Zhang
Modified: 2017-04-26 20:04 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-26 20:04:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
/var/log/libvirt/libvirtd.log from src host (3.64 MB, text/plain)
2017-04-25 03:01 UTC, Pei Zhang
no flags Details
/var/log/libvirt/libvirtd.log from des host (2.79 MB, text/plain)
2017-04-25 03:02 UTC, Pei Zhang
no flags Details
/var/log/libvirt/qemu/rhel7.4_nonrt.log from src host (30.37 KB, text/plain)
2017-04-25 03:04 UTC, Pei Zhang
no flags Details
/var/log/libvirt/qemu/rhel7.4_nonrt.log from des host (15.02 KB, text/plain)
2017-04-25 03:05 UTC, Pei Zhang
no flags Details

Description Pei Zhang 2017-04-24 09:53:22 UTC
Description of problem:
This issue is found when testing ping-pong migration over openvswitch and dpdk. First, do migrate from src to des, works. Then migrate from des to src, migration will fail. This is an regression issue.


Version-Release number of selected component (if applicable):
libvirt-3.2.0-3.el7.x86_64
3.10.0-653.el7.x86_64
qemu-kvm-rhev-2.9.0-1.el7.x86_64
dpdk-16.11-4.el7fdp.x86_64
openvswitch-2.6.1-15.git20161206.el7fdp.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Install dpdk/openvswitch

2. Start openvswitch with 3 vhostuser sock, please refer to[1]

3. Boot VM, please refer to[2]

4. Do migration from src to des host

5. Do migration from des to src host, fail.
# /bin/virsh migrate --verbose --persistent --live rhel7.4_nonrt qemu+ssh://10.73.72.152/system 
error: operation failed: migration job: is not active


Actual results:
Migration fails.


Expected results:
Migration should work well.


Additional info:
1. This is an regression issue.
libvirt-3.2.0-1.el7.x86_64  work
libvirt-3.2.0-3.el7.x86_64  fail

Reference
[1]
# ovs-vsctl show
c1b3795a-0aca-4771-b588-672e7c0875cf
    Bridge "ovsbr1"
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {n_rxq="1"}
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser1.sock"}
    Bridge "ovsbr0"
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {n_rxq="1"}
        Port "vhost-user0"
            Interface "vhost-user0"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser0.sock"}
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    Bridge "ovsbr2"
        Port "dpdk2"
            Interface "dpdk2"
                type: dpdk
                options: {n_rxq="1"}
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser2.sock"}
        Port "ovsbr2"
            Interface "ovsbr2"
                type: internal


[2]<domain type='kvm'>
  <name>rhel7.4_nonrt</name>
  <uuid>2a9c4b5c-28cb-11e7-acd8-14187748a2bb</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='19'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='14'/>
    <emulatorpin cpuset='5,7,9,11,13,15'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
  </features>
<cpu mode='host-passthrough'>
  <feature policy='require' name='tsc-deadline'/>
  <numa>
    <cell id='0' cpus='0-3' memory='8388608' unit='KiB' memAccess='shared'/>
  </numa>
</cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='threads'/>
      <source file='/mnt/nfv/rhel7.4_nonrt.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'/>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='18:66:da:5f:dd:01'/>
      <source bridge='switch'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='18:66:da:5f:dd:02'/>
      <source type='unix' path='/tmp/vhostuser0.sock' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='18:66:da:5f:dd:03'/>
      <source type='unix' path='/tmp/vhostuser1.sock' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='18:66:da:5f:dd:04'/>
      <source type='unix' path='/tmp/vhostuser2.sock' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Comment 3 Jiri Denemark 2017-04-24 10:33:40 UTC
Could you upload debug logs from both sides of migration?

Comment 4 Xuesong Zhang 2017-04-25 02:45:15 UTC
It seems this bug is duplicated with regression BZ1441165, which libvirt found in the qe consumption testing of libvirt-3.2.0-2.el7.x86_64.

Comment 5 Pei Zhang 2017-04-25 03:01:12 UTC
Created attachment 1273793 [details]
/var/log/libvirt/libvirtd.log from src host

Comment 6 Pei Zhang 2017-04-25 03:02:02 UTC
Created attachment 1273794 [details]
/var/log/libvirt/libvirtd.log from des host

Comment 7 Pei Zhang 2017-04-25 03:04:46 UTC
Created attachment 1273796 [details]
/var/log/libvirt/qemu/rhel7.4_nonrt.log from src host

Comment 8 Pei Zhang 2017-04-25 03:05:19 UTC
Created attachment 1273797 [details]
/var/log/libvirt/qemu/rhel7.4_nonrt.log from des host

Comment 9 Jiri Denemark 2017-04-26 20:03:55 UTC
Yes, this is a duplicate of bug 1441165.

Comment 10 Jiri Denemark 2017-04-26 20:04:17 UTC

*** This bug has been marked as a duplicate of bug 1441165 ***


Note You need to log in before you can comment on or make changes to this bug.