Bug 1144490

Summary: QEMU crashes on source host during migration
Product: [Fedora] Fedora Reporter: Daniel BerrangĂ© <berrange>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: amit.shah, berrange, cfergeau, crobinso, dwmw2, gkurz, itamar, mst, pbonzini, rjones, scottt.tw, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-2.1.2-4.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-11 07:02:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Berrangé 2014-09-19 14:12:14 UTC
Description of problem:
Testing qemu 2.1.1 migration with openstack I see the source QEMU crashing during migration. This is a regression from 2.1.0 version

2014-09-19 12:46:05.760+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-00000006 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu Conroe -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ef67b53c-92ea-4bd3-a22a-b27052fc6f2a -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2014.2.0,serial=f09d05e7-68d8-4cab-affb-69a368983b1b,uuid=ef67b53c-92ea-4bd3-a22a-b27052fc6f2a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:4e:01:19,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/home/berrange/src/cloud/data/nova/instances/ef67b53c-92ea-4bd3-a22a-b27052fc6f2a/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/26 (label charserial1)
qemu-system-x86_64: /home/berrange/src/fedora/qemu/qemu-2.1.1/hw/net/virtio-net.c:1348: virtio_net_save: Assertion `!n->vhost_started' failed.
2014-09-19 12:51:20.423+0000: shutting down


Version-Release number of selected component (if applicable):
qemu-2.1.1-1.fc21.x86_64

How reproducible:
Always (with openstack VM config at least)

Steps to Reproduce:
1. Boot guest with a virtio NIC
2. Trigger migration
3.

Actual results:
Source QEMU crashes with an assert

Expected results:
Migration works !

Additional info:

Comment 1 Cole Robinson 2014-09-19 18:14:54 UTC
I know there are some patches that are already accumulating for qemu 2.1.2 that I think are migration related. mst, does that assertion `!n->vhost_started' sound familiar?

Comment 2 Michael S. Tsirkin 2014-09-21 08:44:12 UTC
yes, a known bug, will be fixed in 2.1.2

Comment 3 Fedora Update System 2014-09-21 19:46:45 UTC
qemu-2.1.1-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.1-2.fc21

Comment 4 Fedora Update System 2014-09-24 15:45:37 UTC
Package qemu-2.1.1-2.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qemu-2.1.1-2.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-11180/qemu-2.1.1-2.fc21
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2014-09-26 19:07:24 UTC
qemu-2.1.2-2.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.2-2.fc21

Comment 6 Fedora Update System 2014-09-29 04:03:07 UTC
qemu-2.1.2-2.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Greg Kurz 2014-10-06 15:11:16 UTC
Because of that BZ, the latest QEMU packages for fedora21 are broken with pseries booting from a virtio disk.

When looking at the source RPM, I have spotted the following patch file:

0006-virtio-pci-enable-bus-master-for-old-guests.patch

From a9862ff2c205aa289b62abcb7ddd03c7630f5c7a Mon Sep 17 00:00:00 2001
From: "Michael S. Tsirkin" <mst>
Date: Thu, 11 Sep 2014 18:45:33 +0200
Subject: [PATCH] virtio-pci: enable bus master for old guests

commit cc943c36faa192cd4b32af8fe5edb31894017d35
    pci: Use bus master address space for delivering MSI/MSI-X messages
breaks virtio-net for rhel6.[56] x86 guests because they don't
enable bus mastering for virtio PCI devices. For the same reason,
rhel6.[56] ppc64 guests cannot boot on a virtio-blk disk anymore.

Old guests forgot to enable bus mastering, enable it automatically on
DRIVER (guests use some devices before DRIVER_OK).

Reported-by: Greg Kurz <gkurz.ibm.com>
Reviewed-by: Greg Kurz <gkurz.ibm.com>
Tested-by: Greg Kurz <gkurz.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst>
(cherry picked from commit e43c0b2ea5574efb0bedebf6a7d05916eefeba52)

This commit has been reviewed as toxic and reverted from the QEMU master branch upstream:

http://git.qemu-project.org/?p=qemu.git;a=commit;h=45363e46aeebfc99753389649eac7c7fc22bfe52

If fedora21 is supposed to act as an hypervisor for pseries guest, you should consider reverting
this commit as well.

Comment 8 Greg Kurz 2014-10-06 15:25:34 UTC
(In reply to Greg Kurz from comment #7)
> Because of that BZ, the latest QEMU packages for fedora21 are broken with
> pseries booting from a virtio disk.
> 
> When looking at the source RPM, I have spotted the following patch file:
> 
> 0006-virtio-pci-enable-bus-master-for-old-guests.patch
> 

Oops my bad, wrong patch file... I meant:

0007-virtio-pci-fix-migration-for-pci-bus-master.patch

From 2c8d6826ebaf1bf846edd213f8fe8f95c0c300cb Mon Sep 17 00:00:00 2001
From: "Michael S. Tsirkin" <mst>
Date: Thu, 11 Sep 2014 18:34:29 +0300
Subject: [PATCH] virtio-pci: fix migration for pci bus master

...

Cc: Jason Wang <jasowang>
Cc: Greg Kurz <gkurz.ibm.com>
Signed-off-by: Michael S. Tsirkin <mst>
(cherry picked from commit 4d43d3f3c8147ade184df9a1e9e82826edd39e19)

Comment 9 Fedora Update System 2014-10-06 18:44:12 UTC
qemu-2.1.2-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/qemu-2.1.2-4.fc21

Comment 10 Fedora Update System 2014-10-08 19:10:17 UTC
Package qemu-2.1.2-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qemu-2.1.2-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-12404/qemu-2.1.2-4.fc21
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-10-11 07:02:41 UTC
qemu-2.1.2-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.