RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1172643 - migration/pci-bridge: 7.1rhev->7.0 fails
Summary: migration/pci-bridge: 7.1rhev->7.0 fails
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Dr. David Alan Gilbert
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1184319 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-10 13:48 UTC by vivian zhang
Modified: 2016-03-28 07:14 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-16 18:05:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description vivian zhang 2014-12-10 13:48:36 UTC 
Description of problem:
Attempting to do cross migration from RHEL 7.0->7.1->7.0 with controller pci model='pci-bridge', migration will success from RHEL 7.0 to 7.1, but failed with migrate back the same guest from RHEL 7.1 -> 7.0
 

Version-Release number of selected component (if applicable):
RHEL7.1: libvirt-1.2.8-10.el7.x86_64 qemu-img-rhev-2.1.2-15.el7.x86_64
RHEL7.0: libvirt-1.1.1-29.el7.x86_64 qemu-img-rhev-1.5.3-60.el7_0.11.x86_64

How reproducible:
100%

Steps to Reproduce:

1. Prepare a migration env between two hosts with nfs img disk, one is RHEL7.0 and another one is RHEL 7.1.
 
2. Define a guest on RHEL7.0 host with a pci controller model='pci-bridge' as below.
start guest in running, and work well
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 52    rhel7                          running

# virsh dumpxml rhel7
<domain type='kvm' id='55'>
  <name>rhel7</name>
  <uuid>49a3438a-70a3-4ba8-92ce-3a05e0934608</uuid>
  <title>rhel7-title</title>
  <description>rhel7-desc</description>
  <metadata>
    <app1:foo xmlns:app1="http://app1.org/app1/">..</app1:foo>
    <app2:bar xmlns:app2="http://app1.org/app2/">..</app2:bar>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>1048576</hard_limit>
    <soft_limit unit='KiB'>131072</soft_limit>
    <swap_hard_limit unit='KiB'>2097152</swap_hard_limit>
  </memtune>
  <vcpu placement='static' cpuset='1' current='1'>2</vcpu>
  <cputune>
    <shares>2048</shares>
    <period>1000000</period>
    <quota>-1</quota>
    <emulator_period>1000000</emulator_period>
    <emulator_quota>-1</emulator_quota>
    <emulatorpin cpuset='1'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
    <loader>/usr/share/seabios/bios.bin</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source startupPolicy='optional'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <serial></serial>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='threads' discard='unmap'/>
      <source file='/mnt/wzhang/rhel7.0.qcow2'>
        <seclabel model='selinux' relabel='no'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <iotune>
        <total_bytes_sec>10000000</total_bytes_sec>
        <read_iops_sec>400000</read_iops_sec>
        <write_iops_sec>100000</write_iops_sec>
      </iotune>
      <shareable/>
      <serial>f65effa5-90a6-47f2-8487-a9f64c95d4f5</serial>
      <encryption format='default'>
        <secret type='passphrase' uuid='e0f8d7d9-9e90-4a71-b7ef-e6cb05457dba'/>
      </encryption>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb0'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb0'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb0'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='fdc' index='0'>
      <alias name='fdc0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='pci' index='1' model='pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1a' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:b0:8d:2b'/>
      <source network='default'/>
      <target dev='vnet1'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:c6:3b:95'/>
      <source network='default'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <driver name='vhost' queues='5'/>
      <link state='down'/>
      <bandwidth>
        <inbound average='1000' peak='5000' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='16384' heads='1'>
        <acceleration accel3d='yes' accel2d='yes'/>
      </model>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
    <panic>
      <address type='isa' iobase='0x505'/>
    </panic>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c134,c661</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c134,c661</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>107:107</label>
    <imagelabel>107:107</imagelabel>
  </seclabel>
</domain>



2. do migration from RHEL 7.0 to RHEL7.1, it will success, after migration, guest still work well, pci-bridge still exist
# virsh migrate rhel7 --live qemu+ssh://10.66.7.206/system --verbose
root.7.206's password:
Migration: [100 %]

# virsh dumpxml rhel7 |grep pci-bridge
    <controller type='pci' index='1' model='pci-bridge'>
    <controller type='pci' index='2' model='pci-bridge'>
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 87    rhel7                          running

3. do migration back from RHEL7.1 to RHEL7.0 for the same guest, it will failed

# virsh migrate rhel7 --live qemu+ssh://10.66.6.12/system --verbose
root.6.12's password:
Migration: [100 %]error: Unable to read from monitor: Connection reset by peer


4. check the qemu log from RHEL7.0 host
# tail -f /var/log/libvirt/qemu/rhel7.log
...
Unknown savevm section or instance 'PCIBUS' 1
load of migration failed
2014-12-10 08:31:59.373+0000: shutting down

5. migration between RHEL7.0 -> 7.0 and RHEL7.1 -> 7.1 for the same guest xml success



Actual results:
cross migration from RHEL 7.1->7.0 failed with <controller type='pci' index='1' model='pci-bridge'>

Expected results:
cross migration from RHEL 7.1->7.0 with <controller type='pci' index='1' model='pci-bridge'> should success without error


Additional info:
Comment 2 Dr. David Alan Gilbert 2014-12-12 16:52:04 UTC 
This looks like it's a qemu-kvm-rhev problem:

src (qemu-kvm-rhev 7.1):

/usr/libexec/qemu-kvm -nographic -nodefaults -M rhel6.5.0 -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x3 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.0,addr=0x5 -monitor stdio

dest (qemu-kvm-rhev 7.0):
/usr/libexec/qemu-kvm -nographic -nodefaults -M rhel6.5.0 -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x3 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.0,addr=0x5 -monitor stdio -incoming tcp::4444

migrate -d tcp:pair:4444

QEMU 1.5.3 monitor - type 'help' for more information
(qemu) Unknown savevm section or instance 'PCIBUS' 1
load of migration failed

it only happens with the 2nd PCI bridge.
Comment 3 Jiri Denemark 2014-12-12 16:56:57 UTC 
I tried to reproduce with qemu-kvm (the RHEL variant) from 7.0.z and 7.1 and everything works. So the difference is in qemu-kvm-rhev.
Comment 4 Dr. David Alan Gilbert 2014-12-15 18:19:12 UTC 
The difference here seems to be that qemu-kvm-1.5.3-78.el7.x86_64 on 7.0 doesn't seem to create new buses with that command line; from info pci:

  Bus  0, device   3, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 0.
      BUS 0.
      secondary bus 0.
      subordinate bus 0.
      IO range [0x0000, 0x0fff]
      memory range [0x00000000, 0x000fffff]
      prefetchable memory range [0x00000000, 0x000fffff]
      BAR0: 64 bit memory at 0xffffffffffffffff [0x000000fe].
      id "pci.1"
  Bus  0, device   5, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 0.
      BUS 0.
      secondary bus 0.
      subordinate bus 0.
      IO range [0x0000, 0x0fff]
      memory range [0x00000000, 0x000fffff]
      prefetchable memory range [0x00000000, 0x000fffff]
      BAR0: 64 bit memory at 0xffffffffffffffff [0x000000fe].
      id "pci.2"

Note the secondary and subordinate's showing bus 0;

now on rhev 7.1/2.1.x:
  Bus  0, device   3, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 11.
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xd000, 0xdfff]
      memory range [0xfe400000, 0xfe5fffff]
      prefetchable memory range [0xfea00000, 0xfebfffff]
      BAR0: 64 bit memory at 0xfe600000 [0xfe6000ff].
      id "pci.1"
  Bus  0, device   5, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 0.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xc000, 0xcfff]
      memory range [0xfe200000, 0xfe3fffff]
      prefetchable memory range [0xfe800000, 0xfe9fffff]
      BAR0: 64 bit memory at 0xfe601000 [0xfe6010ff].
      id "pci.2"

each of the bridges sensibly has gained it's own secondary/subordinate bus.
Comment 5 Dr. David Alan Gilbert 2014-12-15 19:39:28 UTC 
but the qtree/qdm is the same;  what seems to be different is that
vmstate_register is getting called on PCIBUS once in rhel7, and 3 times (once per bus?) in 7.1, so that the migrated item is for PCIBUS instance 1, when the destination only has PCIBUS instance 0.

(Possibly due to

 d2f69df7 pci: Move VMState registration/unregistration to QOM realize/unrealize 
)
Comment 9 Dr. David Alan Gilbert 2014-12-16 18:05:59 UTC 
WONTFIX:
  The bug here is in 7.0 where the feature is in tech-preview.

7.0 is only migrating one of the PCI Busses, and thus gets upset when 7.1 tries to send it the busses that are attached to the new bridges.
Comment 10 Dr. David Alan Gilbert 2015-01-21 12:28:38 UTC 
*** Bug 1184319 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.