Bug 1323273

Summary: migration: i82801b11 bridge broken (~q35)
Product: Red Hat Enterprise Linux 7 Reporter: Dr. David Alan Gilbert <dgilbert>
Component: qemu-kvm-rhevAssignee: Dr. David Alan Gilbert <dgilbert>
Status: CLOSED ERRATA QA Contact: jingzhao <jinzhao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: chayang, dgilbert, huding, jinzhao, juzhang, knoel, mrezanin, mzhan, qizhu, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 2.6.0-rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 21:03:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dr. David Alan Gilbert 2016-04-01 16:37:54 UTC
Description of problem:
Migration of an f23 guest fails in q35 on a VM created through virt-manager, I've managed to boil it down to a fairly minimal qemu where if I remove the i82801b11 it works.

Failing rune:
/usr/libexec/qemu-kvm -nographic -machine pc,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
 -device i82801b11-bridge,id=pci.1,bus=pci.0,addr=0x1e \
 -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
 -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
 $*

working rune:
/usr/libexec/qemu-kvm -nographic -machine pc,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
 -device pci-bridge,id=pci.1,chassis_nr=2,bus=pci.0,addr=0x1e \
 -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \
 -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
 $*

Note originally this started off as a libvirt created q35 and I've chipped away at it.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.5.0-3.el7.x86_64  also upstream 2.4.x and 2.5.x and head (2.6.0rc0)

How reproducible:
100%

Steps to Reproduce:
1. Take an f23 install with serial console enabled
2. Start it up with the qemu command line above
3. let it boot and log in
4. vmstat 5 & dmesg -w &
5. migrate (i.e. migrate_set_speed 100G   migrate -d tcp:otherhost:port)
6. On the destination issue the command
   w&
   and see it doesn't complete.
   (You can look at the stack of the w process and find it's blocking on io)

  In a full desktop setup the symptoms were a hang where the running dmesg -w  would show timeouts/spurious interrupts etc


Actual results:
A hung guest

Expected results:
A happy running guest

Additional info:
Our 7.2 RHEV might use that but only on tech-preview q35 VMs, however a user could manually add one of those bridges into his i440fx VM if they felt enthusiastic and I don't think there's anything to say that's not supported?

Comment 5 jingzhao 2016-05-19 10:03:20 UTC
Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps

1. Boot vm with following cmd:
[root@localhost home]# cat bug.sh 
gdb --args /usr/libexec/qemu-kvm \
-nographic \
-machine q35,accel=kvm,usb=off,vmport=off \
-cpu SandyBridge \
-m 4096 \
-realtime mlock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \
-drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \
-vnc :0 \
-vga qxl \
-qmp tcp:0:6666,server,nowait \
-serial unix:/tmp/ttyS0,server,nowait \
-monitor stdio \

2. login guest and run "vmstat 5 & dmesg -w &"

3. Boot another vm with "-incoming tcp:0:5800" on another host

4. In src, "migrate -d tcp:10.66.5.235:5800" 

5. After migration, vm run successfully on the destination host

Could you help me check steps ?

Thanks
Jing

Comment 6 Dr. David Alan Gilbert 2016-05-19 10:10:53 UTC
(In reply to jingzhao from comment #5)
> Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and
> qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps
> 
> 1. Boot vm with following cmd:
> [root@localhost home]# cat bug.sh 
> gdb --args /usr/libexec/qemu-kvm \
> -nographic \
> -machine q35,accel=kvm,usb=off,vmport=off \
> -cpu SandyBridge \
> -m 4096 \
> -realtime mlock=off \
> -smp 4,sockets=4,cores=1,threads=1 \
> -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \
> -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \
> -device
> virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0,
> bootindex=1 \
> -vnc :0 \
> -vga qxl \
> -qmp tcp:0:6666,server,nowait \
> -serial unix:/tmp/ttyS0,server,nowait \
> -monitor stdio \
> 
> 2. login guest and run "vmstat 5 & dmesg -w &"
> 
> 3. Boot another vm with "-incoming tcp:0:5800" on another host
> 
> 4. In src, "migrate -d tcp:10.66.5.235:5800" 
> 
> 5. After migration, vm run successfully on the destination host
> 
> Could you help me check steps ?

Interesting;  I was using pc rather than q35 in my test and it would reliably fail for me.

Dave
> 
> Thanks
> Jing

Comment 7 jingzhao 2016-05-23 01:29:04 UTC
(In reply to Dr. David Alan Gilbert from comment #6)
> (In reply to jingzhao from comment #5)
> > Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and
> > qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps
> > 
> > 1. Boot vm with following cmd:
> > [root@localhost home]# cat bug.sh 
> > gdb --args /usr/libexec/qemu-kvm \
> > -nographic \
> > -machine q35,accel=kvm,usb=off,vmport=off \
> > -cpu SandyBridge \
> > -m 4096 \
> > -realtime mlock=off \
> > -smp 4,sockets=4,cores=1,threads=1 \
> > -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \
> > -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \
> > -device
> > virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0,
> > bootindex=1 \
> > -vnc :0 \
> > -vga qxl \
> > -qmp tcp:0:6666,server,nowait \
> > -serial unix:/tmp/ttyS0,server,nowait \
> > -monitor stdio \
> > 
> > 2. login guest and run "vmstat 5 & dmesg -w &"
> > 
> > 3. Boot another vm with "-incoming tcp:0:5800" on another host
> > 
> > 4. In src, "migrate -d tcp:10.66.5.235:5800" 
> > 
> > 5. After migration, vm run successfully on the destination host
> > 
> > Could you help me check steps ?
> 
> Interesting;  I was using pc rather than q35 in my test and it would
> reliably fail for me.
--Changed to the pc machine type and also didn't reproduce,kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64, the right reproduce steps? 

Thanks
Jing
> 
> Dave
> > 
> > Thanks
> > Jing

Comment 8 jingzhao 2016-05-24 08:04:34 UTC
(In reply to jingzhao from comment #7)
> (In reply to Dr. David Alan Gilbert from comment #6)
> > (In reply to jingzhao from comment #5)
> > > Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and
> > > qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps
> > > 
> > > 1. Boot vm with following cmd:
> > > [root@localhost home]# cat bug.sh 
> > > gdb --args /usr/libexec/qemu-kvm \
> > > -nographic \
> > > -machine q35,accel=kvm,usb=off,vmport=off \
> > > -cpu SandyBridge \
> > > -m 4096 \
> > > -realtime mlock=off \
> > > -smp 4,sockets=4,cores=1,threads=1 \
> > > -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \
> > > -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \
> > > -device
> > > virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0,
> > > bootindex=1 \
> > > -vnc :0 \
> > > -vga qxl \
> > > -qmp tcp:0:6666,server,nowait \
> > > -serial unix:/tmp/ttyS0,server,nowait \
> > > -monitor stdio \
> > > 
> > > 2. login guest and run "vmstat 5 & dmesg -w &"
> > > 
> > > 3. Boot another vm with "-incoming tcp:0:5800" on another host
> > > 
> > > 4. In src, "migrate -d tcp:10.66.5.235:5800" 
> > > 
> > > 5. After migration, vm run successfully on the destination host
> > > 
> > > Could you help me check steps ?
> > 
> > Interesting;  I was using pc rather than q35 in my test and it would
> > reliably fail for me.
> --Changed to the pc machine type and also didn't
> reproduce,kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64,
> the right reproduce steps? 
  Right reproduce version and right reproduced steps?
> 
> Thanks
> Jing
> > 
> > Dave
> > > 
> > > Thanks
> > > Jing

Comment 9 jingzhao 2016-06-01 03:25:33 UTC
Any comments for this issue because I didn't reproduce the issue.

Comment 10 Dr. David Alan Gilbert 2016-07-08 16:41:09 UTC
Well, I'm not sure since it reproduced reliably in my testing; however the really important thing is just to check that a q35 with an 82801b11 bridge (with devices on the PCI bus that the bridge creates) works on the current qemu.  If that works across a migration then OK the bz.

Comment 11 jingzhao 2016-08-22 05:53:58 UTC
As comment 10, test it with an 82801b11 bridge and it works

host kernel:3.10.0-492.el7.x86_64
qemu-kvm-rhev-2.6.0-20.el7.x86_64

Test steps:
1. Boot vm with 82801b11 bridge with device on the PCI bus that the bridge creates

2.(qemu) info block
drive-virtio-disk0 (#block152): /mnt/floppy.qcow2 (qcow2)
    Cache mode:       writeback, direct

pci-block1 (#block362): /mnt/pci-block1.qcow2 (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk1 (#block527): /mnt/bug/block.qcow2 (qcow2)
    Cache mode:       writeback, direct

3.(qemu) info qtree
bus: main-system-bus
  type System
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 0 (0 B)
    short_root_bus = 0 (0x0)
    bus: pcie.0
      type PCIE
      dev: i82801b11-bridge, id "bridge1"
        addr = 06.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        x-pcie-lnksta-dllla = true
        class PCI bridge, addr 00:06.0, pci id 8086:244e (sub 0000:0000)
        bus: bridge1
          type PCI
          dev: pci-bridge, id "bridge2"
            chassis_nr = 1 (0x1)
            msi = true
            shpc = true
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            x-pcie-lnksta-dllla = true
            class PCI bridge, addr 08:00.0, pci id 1b36:0001 (sub 0000:0000)
            bar 0: mem at 0xf8800000 [0xf88000ff]
            bus: bridge2
              type PCI
              dev: virtio-blk-pci, id "virtio-disk1"
                class = 0 (0x0)
                ioeventfd = true
                vectors = 2 (0x2)
                virtio-pci-bus-master-bug-migration = false
                disable-legacy = "off"
                disable-modern = false
                migrate-extra = true
                modern-pio-notify = false
                x-disable-pcie = false
                addr = 06.0
                romfile = ""
                rombar = 1 (0x1)
                multifunction = false
                command_serr_enable = true
                x-pcie-lnksta-dllla = true
                class SCSI controller, addr 09:06.0, pci id 1af4:1001 (sub 1af4:0002)
                bar 0: i/o at 0xc000 [0xc03f]
                bar 1: mem at 0xf8600000 [0xf8600fff]
                bar 4: mem at 0xfd800000 [0xfdffffff]
                bus: virtio-bus
                  type virtio-pci-bus
                  dev: virtio-blk-device, id ""
                    drive = "drive-virtio-disk1"
                    logical_block_size = 512 (0x200)
                    physical_block_size = 512 (0x200)
                    min_io_size = 0 (0x0)
                    opt_io_size = 0 (0x0)
                    discard_granularity = 4294967295 (0xffffffff)
                    cyls = 1015 (0x3f7)
                    heads = 16 (0x10)
                    secs = 63 (0x3f)
                    serial = ""
                    config-wce = true
                    scsi = false
                    request-merging = true
                    indirect_desc = true
                    event_idx = true
                    notify_on_empty = true
                    any_layout = true

4.(qemu) migrate -d tcp:10.66.144.41:5800
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off 
Migration status: active
total time: 2865 milliseconds
expected downtime: 300 milliseconds
setup: 14 milliseconds
transferred ram: 33971 kbytes
throughput: 94.81 mbps
remaining ram: 3512776 kbytes
total ram: 4326096 kbytes
duplicate: 195281 pages
skipped: 0 pages
normal: 8048 pages
normal bytes: 32192 kbytes
dirty sync count: 1

5. Check the block info on the destination guest

[root@openshift-159 ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:03.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:05.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
00:06.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
02:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream) (rev 02)
03:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01)
03:01.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01)
03:02.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01)
06:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01)
08:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
09:06.0 SCSI storage controller: Red Hat, Inc Virtio block device
[root@openshift-159 ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]--
           |                               +-01.0-[05]--
           |                               \-02.0-[06]----00.0  Red Hat, Inc Virtio network device
           +-04.0  Red Hat, Inc Virtio block device
           +-05.0-[07]--
           +-06.0-[08-09]----00.0-[09]----06.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@openshift-159 ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0   10G  0 disk 
vdb           252:16   0   30G  0 disk 
├─vdb1        252:17   0    1G  0 part /boot
└─vdb2        252:18   0   29G  0 part 
  ├─rhel-root 253:0    0   26G  0 lvm  /
  └─rhel-swap 253:1    0    3G  0 lvm  [SWAP]
vdc           252:32   0  500M  0 disk 
[root@openshift-159 ~]# cat /sys/bus/pci/devices/0000\:09\:06.0/virtio3/features 
0010101001110000000000000000110010000000000000000000000000000000

6.guest can reboot successfully

Add info :
/usr/libexec/qemu-kvm \
-M q35 \
-cpu Nehalem \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.3 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device ioh3420,id=root.0,slot=1 \
-drive file=/mnt/floppy.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-drive file=/mnt/pci-block1.qcow2,if=none,id=pci-block1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=pci-block1,id=pciblock1,bus=pcie.0 \
-device ioh3420,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=downstream3 \
-device i82801b11-bridge,id=bridge1,bus=pcie.0 \
-device pci-bridge,id=bridge2,bus=bridge1,chassis_nr=1 \
-drive file=/mnt/bug/block.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=bridge2,drive=drive-virtio-disk1,id=virtio-disk1,addr=0x6.0 \
-monitor stdio \

Above test is right and is it enough?

Thanks
Jing Zhao

Comment 12 Dr. David Alan Gilbert 2016-08-22 08:50:20 UTC
Yes, I think if that test works I think we should be OK.
Thanks!

Comment 15 errata-xmlrpc 2016-11-07 21:03:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html