Bug 1118543

Summary: Can not continue live commiting when using glusterfs(native) backend after relaunch qemu
Product: Red Hat Enterprise Linux 6 Reporter: Qian Guo <qiguo>
Component: qemu-kvmAssignee: Jeff Cody <jcody>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.6CC: bsarathy, juzhang, knoel, mazhang, michen, mkenneth, qzhang, rbalakri, shu, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.434.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 07:02:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Qian Guo 2014-07-11 03:21:46 UTC
Description of problem:
Kill qemu-kvm when do live commiting, and then try to boot guest with the last snapshot, and do live commit again via qmp, it prompts and failed.
{"error": {"class": "NotSupported", "desc": "Not supported", "data": {}}}

Version-Release number of selected component (if applicable):
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64
# uname -r
2.6.32-489.el6.x86_64
Gluseterfs client build:
glusterfs-3.6.0.24-1.el6.x86_64
How reproducible:
100%

Steps to Reproduce:
1.Boot guest with glusterfs:native backend:
# /usr/libexec/qemu-kvm -cpu Penryn -m 4G -smp 4 -M pc -enable-kvm -name rhel6u4 -nodefaults -nodefconfig -vga std -monitor stdio -drive file=gluster://10.66.9.152/gv0/rhel5.11cp2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,werror=stop,rerror=stop,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 -vnc :20 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-netpci0,mac=54:52:1b:36:1a:02 -qmp unix:/tmp/q1,server,nowait -monitor unix:/tmp/m1,server,nowait


2.Do live snapshots via qmp:
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive-virtio-disk0", "snapshot-file": "gluster://10.66.9.152/gv0/sn1", "format": "qcow2" } }


{"return": {}}
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive-virtio-disk0", "snapshot-file": "gluster://10.66.9.152/gv0/sn2", "format": "qcow2" } }

{"return": {}}
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive-virtio-disk0", "snapshot-file": "gluster://10.66.9.152/gv0/sn3", "format": "qcow2" } }

{"return": {}}

Then check info block:
(qemu) info block
drive-virtio-disk0: removable=0 io-status=ok file=gluster://10.66.9.152/gv0/sn3 backing_file=gluster://10.66.9.152/gv0/sn2 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0


3.dd a 1G file inside guest

4.Do live snapshot again
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive-virtio-disk0", "snapshot-file": "gluster://10.66.9.152/gv0/sn4", "format": "qcow2" } }

{"return": {}}

check info block:
(qemu) info block
drive-virtio-disk0: removable=0 io-status=ok file=gluster://10.66.9.152/gv0/sn4 backing_file=gluster://10.66.9.152/gv0/sn3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0

5.Do live commit sn3>sn1:
{ "execute": "__com.redhat_block-commit", "arguments": { "device": "drive-virtio-disk0", "base": "gluster://10.66.9.152/gv0/sn1", "top": "gluster://10.66.9.152/gv0/sn3", "speed": 100} }
{"return": {}}

6.During step5, kill qemu process

7.Boot guest with sn4:
# /usr/libexec/qemu-kvm -cpu Penryn -m 4G -smp 4 -M pc -enable-kvm -name rhel6u4 -nodefaults -nodefconfig -vga std -monitor stdio -drive file=gluster://10.66.9.152/gv0/sn4,if=none,id=drive-virtio-disk0,format=qcow2,werror=stop,rerror=stop,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 -vnc :20 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-netpci0,mac=54:52:1b:36:1a:02 -qmp unix:/tmp/q1,server,nowait -monitor unix:/tmp/m1,server,nowait

8.check info block:

(qemu) info block
drive-virtio-disk0: removable=0 io-status=ok file=gluster://10.66.9.152/gv0/sn4 backing_file=gluster://10.66.9.152/gv0/sn3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0


9.Do live commit again sn3>sn1:
{ "execute": "__com.redhat_block-commit", "arguments": { "device": "drive-virtio-disk0", "base": "gluster://10.66.9.152/gv0/sn1", "top": "gluster://10.66.9.152/gv0/sn3", "speed": 100} }

Actual results:
After step9
prompts and won't do live commit
{"error": {"class": "NotSupported", "desc": "Not supported", "data": {}}}


Expected results:

Can finish live commit successfully.
Additional info:
Test with local file snapshot, won't hit such issue

Comment 2 Qian Guo 2014-07-11 07:52:16 UTC
Test this with qemu-kvm-rhev-0.12.1.2-2.426.el6.x86_64, hit such issue, so this bug is not regression

Comment 3 Jeff Cody 2014-07-29 18:18:46 UTC
This doesn't have anything to do with actually killing the process, it has to do with performing block commit on a snapshot chain that is already existing.  RHEL6 qemu gluster driver does not have reopen support, which means we cannot change the backing file target to r/w, in order to write into it.

It works when the snapshot chain is created in the same session, because since we lack the ability to perform a reopen, we don't set the backing files to r/o after the snapshot.  So they are writable, which allows the block-commit to happen.

Upstream gluster has reopen support, so these patches need to be backported for RHEL6:

1b37b34 block: gluster - code movements, state storage changes
adccfbc block: gluster - add reopen support.

Comment 5 Jeff Nelson 2014-07-31 00:16:39 UTC
Fix included in qemu-kvm-0.12.1.2-2.434.el6

Comment 7 mazhang 2014-08-04 05:52:32 UTC
Reproduce this bug on qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64.

Host:
kernel-2.6.32-493.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.430.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64
qemu-img-rhev-0.12.1.2-2.430.el6.x86_64
gpxe-roms-qemu-0.9.7-6.11.el6.noarch
qemu-kvm-rhev-debuginfo-0.12.1.2-2.430.el6.x86_64

Guest:
kernel-2.6.32-488.el6.x86_64

Cli:
/usr/libexec/qemu-kvm \
-M rhel6.6.0 \
-cpu SandyBridge \
-m 4G \
-smp 4,sockets=1,cores=2,threads=2,maxcpus=160 \
-enable-kvm \
-name rhel6.6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:6666,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev socket,id=seabios,path=/tmp/seabios0,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-monitor unix:/tmp/socket0,server,nowait -device sga \
-vga qxl \
-spice port=5900,disable-ticketing \
-usb \
-device usb-tablet,id=tablet0 \
-drive file=gluster://10.66.8.129/gv0/sn4,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \

Steps refer to comment#0.

Result:
Live commit failed.

{ "execute": "__com.redhat_block-commit", "arguments": { "device": "drive-virtio-disk0", "base": "gluster://10.66.8.129/gv0/sn1", "top": "gluster://10.66.8.129/gv0/sn3", "speed": 100}}
{"error": {"class": "NotSupported", "desc": "Not supported", "data": {}}}



Verified this bug on qemu-kvm-rhev-0.12.1.2-2.434.el6.

Host:
qemu-kvm-rhev-debuginfo-0.12.1.2-2.434.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.434.el6.x86_64
qemu-img-rhev-0.12.1.2-2.434.el6.x86_64
gpxe-roms-qemu-0.9.7-6.11.el6.noarch
qemu-kvm-rhev-tools-0.12.1.2-2.434.el6.x86_64
kernel-2.6.32-493.el6.x86_64

Guest:
kernel-2.6.32-488.el6.x86_64

Result:
Can't hit the problem, live commit completed.
{"QMP": {"version": {"qemu": {"micro": 1, "minor": 12, "major": 0}, "package": "(qemu-kvm-rhev-0.12.1.2-2.434.el6)"}, "capabilities": []}}
{"execute":"qmp_capabilities"}
{"return": {}}
{ "execute": "__com.redhat_block-commit", "arguments": { "device": "drive-virtio-disk0", "base": "gluster://10.66.8.129/gv0/sn1", "top": "gluster://10.66.8.129/gv0/sn3", "speed": 100}}
{"return": {}}
{"timestamp": {"seconds": 1407131065, "microseconds": 834606}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-disk0", "len": 107374182400, "offset": 107374182400, "speed": 100, "type": "commit"}}


So this bug has been fixed.

Comment 8 errata-xmlrpc 2014-10-14 07:02:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html