Hide Forgot
Description of problem: Migration of an f23 guest fails in q35 on a VM created through virt-manager, I've managed to boil it down to a fairly minimal qemu where if I remove the i82801b11 it works. Failing rune: /usr/libexec/qemu-kvm -nographic -machine pc,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \ -device i82801b11-bridge,id=pci.1,bus=pci.0,addr=0x1e \ -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \ $* working rune: /usr/libexec/qemu-kvm -nographic -machine pc,accel=kvm,usb=off,vmport=off -cpu SandyBridge -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \ -device pci-bridge,id=pci.1,chassis_nr=2,bus=pci.0,addr=0x1e \ -drive id=image,file=/home/vms/f23-serial.qcow2,if=none,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \ $* Note originally this started off as a libvirt created q35 and I've chipped away at it. Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.5.0-3.el7.x86_64 also upstream 2.4.x and 2.5.x and head (2.6.0rc0) How reproducible: 100% Steps to Reproduce: 1. Take an f23 install with serial console enabled 2. Start it up with the qemu command line above 3. let it boot and log in 4. vmstat 5 & dmesg -w & 5. migrate (i.e. migrate_set_speed 100G migrate -d tcp:otherhost:port) 6. On the destination issue the command w& and see it doesn't complete. (You can look at the stack of the w process and find it's blocking on io) In a full desktop setup the symptoms were a hang where the running dmesg -w would show timeouts/spurious interrupts etc Actual results: A hung guest Expected results: A happy running guest Additional info: Our 7.2 RHEV might use that but only on tech-preview q35 VMs, however a user could manually add one of those bridges into his i440fx VM if they felt enthusiastic and I don't think there's anything to say that's not supported?
Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps 1. Boot vm with following cmd: [root@localhost home]# cat bug.sh gdb --args /usr/libexec/qemu-kvm \ -nographic \ -machine q35,accel=kvm,usb=off,vmport=off \ -cpu SandyBridge \ -m 4096 \ -realtime mlock=off \ -smp 4,sockets=4,cores=1,threads=1 \ -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \ -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \ -device virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0,bootindex=1 \ -vnc :0 \ -vga qxl \ -qmp tcp:0:6666,server,nowait \ -serial unix:/tmp/ttyS0,server,nowait \ -monitor stdio \ 2. login guest and run "vmstat 5 & dmesg -w &" 3. Boot another vm with "-incoming tcp:0:5800" on another host 4. In src, "migrate -d tcp:10.66.5.235:5800" 5. After migration, vm run successfully on the destination host Could you help me check steps ? Thanks Jing
(In reply to jingzhao from comment #5) > Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and > qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps > > 1. Boot vm with following cmd: > [root@localhost home]# cat bug.sh > gdb --args /usr/libexec/qemu-kvm \ > -nographic \ > -machine q35,accel=kvm,usb=off,vmport=off \ > -cpu SandyBridge \ > -m 4096 \ > -realtime mlock=off \ > -smp 4,sockets=4,cores=1,threads=1 \ > -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \ > -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \ > -device > virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0, > bootindex=1 \ > -vnc :0 \ > -vga qxl \ > -qmp tcp:0:6666,server,nowait \ > -serial unix:/tmp/ttyS0,server,nowait \ > -monitor stdio \ > > 2. login guest and run "vmstat 5 & dmesg -w &" > > 3. Boot another vm with "-incoming tcp:0:5800" on another host > > 4. In src, "migrate -d tcp:10.66.5.235:5800" > > 5. After migration, vm run successfully on the destination host > > Could you help me check steps ? Interesting; I was using pc rather than q35 in my test and it would reliably fail for me. Dave > > Thanks > Jing
(In reply to Dr. David Alan Gilbert from comment #6) > (In reply to jingzhao from comment #5) > > Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and > > qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps > > > > 1. Boot vm with following cmd: > > [root@localhost home]# cat bug.sh > > gdb --args /usr/libexec/qemu-kvm \ > > -nographic \ > > -machine q35,accel=kvm,usb=off,vmport=off \ > > -cpu SandyBridge \ > > -m 4096 \ > > -realtime mlock=off \ > > -smp 4,sockets=4,cores=1,threads=1 \ > > -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \ > > -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \ > > -device > > virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0, > > bootindex=1 \ > > -vnc :0 \ > > -vga qxl \ > > -qmp tcp:0:6666,server,nowait \ > > -serial unix:/tmp/ttyS0,server,nowait \ > > -monitor stdio \ > > > > 2. login guest and run "vmstat 5 & dmesg -w &" > > > > 3. Boot another vm with "-incoming tcp:0:5800" on another host > > > > 4. In src, "migrate -d tcp:10.66.5.235:5800" > > > > 5. After migration, vm run successfully on the destination host > > > > Could you help me check steps ? > > Interesting; I was using pc rather than q35 in my test and it would > reliably fail for me. --Changed to the pc machine type and also didn't reproduce,kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64, the right reproduce steps? Thanks Jing > > Dave > > > > Thanks > > Jing
(In reply to jingzhao from comment #7) > (In reply to Dr. David Alan Gilbert from comment #6) > > (In reply to jingzhao from comment #5) > > > Didn't reproduced with kernel-3.10.0-402.el7.x86_64 and > > > qemu-kvm-rhev-2.5.0-3.el7.x86_64, following is my reproduce steps > > > > > > 1. Boot vm with following cmd: > > > [root@localhost home]# cat bug.sh > > > gdb --args /usr/libexec/qemu-kvm \ > > > -nographic \ > > > -machine q35,accel=kvm,usb=off,vmport=off \ > > > -cpu SandyBridge \ > > > -m 4096 \ > > > -realtime mlock=off \ > > > -smp 4,sockets=4,cores=1,threads=1 \ > > > -device i82801b11-bridge,id=pcie.1,bus=pcie.0,addr=0x1e \ > > > -drive id=image,file=/mnt/bug/fedora.qcow2,if=none,cache=none,format=qcow2 \ > > > -device > > > virtio-blk-pci,scsi=off,bus=pcie.1,addr=0x5,drive=image,id=virtio-disk0, > > > bootindex=1 \ > > > -vnc :0 \ > > > -vga qxl \ > > > -qmp tcp:0:6666,server,nowait \ > > > -serial unix:/tmp/ttyS0,server,nowait \ > > > -monitor stdio \ > > > > > > 2. login guest and run "vmstat 5 & dmesg -w &" > > > > > > 3. Boot another vm with "-incoming tcp:0:5800" on another host > > > > > > 4. In src, "migrate -d tcp:10.66.5.235:5800" > > > > > > 5. After migration, vm run successfully on the destination host > > > > > > Could you help me check steps ? > > > > Interesting; I was using pc rather than q35 in my test and it would > > reliably fail for me. > --Changed to the pc machine type and also didn't > reproduce,kernel-3.10.0-402.el7.x86_64 and qemu-kvm-rhev-2.5.0-3.el7.x86_64, > the right reproduce steps? Right reproduce version and right reproduced steps? > > Thanks > Jing > > > > Dave > > > > > > Thanks > > > Jing
Any comments for this issue because I didn't reproduce the issue.
Well, I'm not sure since it reproduced reliably in my testing; however the really important thing is just to check that a q35 with an 82801b11 bridge (with devices on the PCI bus that the bridge creates) works on the current qemu. If that works across a migration then OK the bz.
As comment 10, test it with an 82801b11 bridge and it works host kernel:3.10.0-492.el7.x86_64 qemu-kvm-rhev-2.6.0-20.el7.x86_64 Test steps: 1. Boot vm with 82801b11 bridge with device on the PCI bus that the bridge creates 2.(qemu) info block drive-virtio-disk0 (#block152): /mnt/floppy.qcow2 (qcow2) Cache mode: writeback, direct pci-block1 (#block362): /mnt/pci-block1.qcow2 (qcow2) Cache mode: writeback, direct drive-virtio-disk1 (#block527): /mnt/bug/block.qcow2 (qcow2) Cache mode: writeback, direct 3.(qemu) info qtree bus: main-system-bus type System dev: kvm-ioapic, id "" gpio-in "" 24 gsi_base = 0 (0x0) mmio 00000000fec00000/0000000000001000 dev: q35-pcihost, id "" MCFG = 2952790016 (0xb0000000) pci-hole64-size = 0 (0 B) short_root_bus = 0 (0x0) bus: pcie.0 type PCIE dev: i82801b11-bridge, id "bridge1" addr = 06.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true x-pcie-lnksta-dllla = true class PCI bridge, addr 00:06.0, pci id 8086:244e (sub 0000:0000) bus: bridge1 type PCI dev: pci-bridge, id "bridge2" chassis_nr = 1 (0x1) msi = true shpc = true addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true x-pcie-lnksta-dllla = true class PCI bridge, addr 08:00.0, pci id 1b36:0001 (sub 0000:0000) bar 0: mem at 0xf8800000 [0xf88000ff] bus: bridge2 type PCI dev: virtio-blk-pci, id "virtio-disk1" class = 0 (0x0) ioeventfd = true vectors = 2 (0x2) virtio-pci-bus-master-bug-migration = false disable-legacy = "off" disable-modern = false migrate-extra = true modern-pio-notify = false x-disable-pcie = false addr = 06.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true x-pcie-lnksta-dllla = true class SCSI controller, addr 09:06.0, pci id 1af4:1001 (sub 1af4:0002) bar 0: i/o at 0xc000 [0xc03f] bar 1: mem at 0xf8600000 [0xf8600fff] bar 4: mem at 0xfd800000 [0xfdffffff] bus: virtio-bus type virtio-pci-bus dev: virtio-blk-device, id "" drive = "drive-virtio-disk1" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 4294967295 (0xffffffff) cyls = 1015 (0x3f7) heads = 16 (0x10) secs = 63 (0x3f) serial = "" config-wce = true scsi = false request-merging = true indirect_desc = true event_idx = true notify_on_empty = true any_layout = true 4.(qemu) migrate -d tcp:10.66.144.41:5800 (qemu) info migrate capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off Migration status: active total time: 2865 milliseconds expected downtime: 300 milliseconds setup: 14 milliseconds transferred ram: 33971 kbytes throughput: 94.81 mbps remaining ram: 3512776 kbytes total ram: 4326096 kbytes duplicate: 195281 pages skipped: 0 pages normal: 8048 pages normal bytes: 32192 kbytes dirty sync count: 1 5. Check the block info on the destination guest [root@openshift-159 ~]# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 00:03.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device 00:05.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 00:06.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01) 02:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream) (rev 02) 03:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01) 03:01.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01) 03:02.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01) 06:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01) 08:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 09:06.0 SCSI storage controller: Red Hat, Inc Virtio block device [root@openshift-159 ~]# lspci -vvv -t -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Red Hat, Inc. QXL paravirtual graphic card +-02.0-[01]----00.0 Red Hat, Inc Virtio block device +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]-- | +-01.0-[05]-- | \-02.0-[06]----00.0 Red Hat, Inc Virtio network device +-04.0 Red Hat, Inc Virtio block device +-05.0-[07]-- +-06.0-[08-09]----00.0-[09]----06.0 Red Hat, Inc Virtio block device +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller [root@openshift-159 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 10G 0 disk vdb 252:16 0 30G 0 disk ├─vdb1 252:17 0 1G 0 part /boot └─vdb2 252:18 0 29G 0 part ├─rhel-root 253:0 0 26G 0 lvm / └─rhel-swap 253:1 0 3G 0 lvm [SWAP] vdc 252:32 0 500M 0 disk [root@openshift-159 ~]# cat /sys/bus/pci/devices/0000\:09\:06.0/virtio3/features 0010101001110000000000000000110010000000000000000000000000000000 6.guest can reboot successfully Add info : /usr/libexec/qemu-kvm \ -M q35 \ -cpu Nehalem \ -nodefaults -rtc base=utc \ -m 4G \ -smp 2,sockets=2,cores=1,threads=1 \ -enable-kvm \ -name rhel7.3 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -serial unix:/tmp/console,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -chardev file,path=/home/seabios.log,id=seabios \ -device isa-debugcon,chardev=seabios,iobase=0x402 \ -qmp tcp::8887,server,nowait \ -vga qxl \ -spice port=5932,disable-ticketing \ -device ioh3420,id=root.0,slot=1 \ -drive file=/mnt/floppy.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -device ioh3420,id=root.1,slot=2 \ -device x3130-upstream,bus=root.1,id=upstream1 \ -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \ -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \ -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \ -drive file=/mnt/pci-block1.qcow2,if=none,id=pci-block1,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=pci-block1,id=pciblock1,bus=pcie.0 \ -device ioh3420,id=root.2,slot=3 \ -netdev tap,id=hostnet1 \ -device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=downstream3 \ -device i82801b11-bridge,id=bridge1,bus=pcie.0 \ -device pci-bridge,id=bridge2,bus=bridge1,chassis_nr=1 \ -drive file=/mnt/bug/block.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,bus=bridge2,drive=drive-virtio-disk1,id=virtio-disk1,addr=0x6.0 \ -monitor stdio \ Above test is right and is it enough? Thanks Jing Zhao
Yes, I think if that test works I think we should be OK. Thanks!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2673.html