Bug 1130483

Summary: qemu failed to reopen target image after drive-mirror
Product: Red Hat Enterprise Linux 6 Reporter: Xiaoqing Wei <xwei>
Component: qemu-kvmAssignee: Jeff Cody <jcody>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.6CC: ailan, chayang, juzhang, mkenneth, mzhan, qzhang, rbalakri, rpacheco, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-06 10:21:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1359965    

Description Xiaoqing Wei 2014-08-15 10:58:50 UTC
Description of problem:
qemu failed to reopen target image after drive-mirror

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-tools-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.438.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.438.el6.x86_64
qemu-img-rhev-0.12.1.2-2.438.el6.x86_64
also tested 6.5.z(_5.14)
and 6.5 GA

How reproducible:
100%

Steps to Reproduce:
1. create LVMs, 10GB * 1, 32GB * 1
# lvscan 
  ACTIVE            '/dev/VolGroup/LogVol_swap' [12.00 GiB] inherit
  ACTIVE            '/dev/VolGroup/LogVol_root' [409.25 GiB] inherit
  ACTIVE            '/dev/VolGroup/lv10g' [10.00 GiB] inherit
  ACTIVE            '/dev/VolGroup/raw30g' [32.00 GiB] inherit

2. having a guest img, virtual size 20G, actual size < 10G
# qemu-img info /mnt/linux/RHEL-Server-6.5-64-virtio.qcow2 
image: /mnt/linux/RHEL-Server-6.5-64-virtio.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.6G
cluster_size: 65536

3. convert img from file based to lvm based.
#qemu-img convert -f qcow2 -O qcow2 /mnt/linux/RHEL-Server-6.5-64-virtio.qcow2 /dev/VolGroup/lv10g -p

4. boot w/ lvm based
/usr/bin/qemu-kvm -monitor stdio \
    -S  \
    -name 'virt-tests-vm1' \
    -nodefaults  \
    -vga std \
    -M rhel6.6.0  \
    -chardev socket,id=qmp_id_qmp1,path=/tmp/monitor-qmp1-20140812-133800-bf4OXeCV,server,nowait \
    -mon chardev=qmp_id_qmp1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140812-133800-bf4OXeCV,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20140812-133800-bf4OXeCV,path=/tmp/seabios-20140812-133800-bf4OXeCV,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20140812-133800-bf4OXeCV,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
\
\
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,werror=stop,rerror=stop,format=qcow2,file='/dev/VolGroup/lv10g' \
\
\
    -device ide-drive,id=image1,drive=drive_image1,bus=ide.0,unit=0 \
    -device virtio-net-pci,mac=9a:9b:9c:9d:9e:9f,id=idokVqY6,vectors=4,netdev=idcsqU1f,bus=pci.0,addr=05  \
    -netdev tap,id=idcsqU1f,vhost=on \
    -m 4096  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'SandyBridge' \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off \
    -enable-kvm

5. consume more space, make LVM out of space, and qemu pauses VM(io-error)
# dd if=/dev/urandom of=/root/4GB.img bs=1M count=4096

6. mirror LVM to the one has enough space.
# nc -U /tmp/monitor-qmp1-20140812-133800-bf4OXeCV 
{"QMP": {"version": {"qemu": {"micro": 1, "minor": 12, "major": 0}, "package": "(qemu-kvm-rhev-0.12.1.2-2.438.el6)"}, "capabilities": []}}
{'execute': 'qmp_capabilities', 'id': 'XNISVhf3'}
{"return": {}, "id": "XNISVhf3"}
{"execute": "query-block-jobs", "id": "K4RBuwiZ"}
{"return": [], "id": "K4RBuwiZ"}
{"execute": "__com.redhat_drive-mirror", "arguments": {"device": "drive_image1", "format": "qcow2", "full": true, "target": "/dev/VolGroup/raw30g", "mode": "absolute-paths"}, "id": "BH2iL2Kl"}
{"id": "BH2iL2Kl", "error": {"class": "GenericError", "desc": "/dev/VolGroup/raw30g: error while creating qcow2: Invalid argument", "data": {"message": "/dev/VolGroup/raw30g: error while creating qcow2: Invalid argument"}}}
{"execute": "__com.redhat_drive-mirror", "arguments": {"device": "drive_image1", "format": "raw", "full": true, "target": "/dev/VolGroup/raw30g", "mode": "absolute-paths"}, "id": "BH2iL2Kl"}
{"return": {}, "id": "BH2iL2Kl"}

7. wait till mirror done

(qemu) block I/O error in device 'drive_image1': No space left on device (28)

(qemu) 
(qemu) Formatting '/dev/VolGroup/raw30g', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 

(qemu) 
(qemu) info block-jobs 
No active jobs
(qemu) Formatting '/dev/VolGroup/raw30g', fmt=raw size=21474836480 
info block-jobs 
Type mirror, device drive_image1: Completed 10972233728 of 21474836480 bytes, speed limit 0 bytes/s
(qemu) info status 
VM status: paused (io-error)
(qemu) info block-jobs 
Type mirror, device drive_image1: Completed 21474836480 of 21474836480 bytes, speed limit 0 bytes/s
(qemu) info block-jobs 
Type mirror, device drive_image1: Completed 21474836480 of 21474836480 bytes, speed limit 0 bytes/s

8. switch VM storage to target LVM(or called reopen)
{"execute": "__com.redhat_drive-reopen", "arguments": {"device": "drive_image1", "new-image-file": "/dev/VolGroup/raw30g", "format": "raw"}}
{"timestamp": {"seconds": 1408098876, "microseconds": 761571}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "mirror"}}
{"error": {"class": "IOError", "desc": "An IO error has occurred", "data": {}}}

9. recheck VM storage after 8. raise error
(qemu) info block-jobs 
No active jobs
(qemu) info block
drive_image1: removable=0 io-status=nospace file=/dev/VolGroup/lv10g ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
drive_cd1: removable=1 locked=1 tray-open=0 io-status=ok file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso ro=1 drv=raw encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
(qemu) info status 
VM status: paused (io-error)


Actual results:
as 8. outputs, qemu still io-error
and 9. the underlying storage didn't switched.

Expected results:
9. should shown underlying storage switched, and
8. should not error happen.

Additional info:
identical test performed and error on
6.5.Z
6.5 GA
6.4 GA


identical test performed and GOOD on
RHEL 7.1 W/ RHEV 7.1 pkg
libcacard-tools-rhev-2.1.0-1.el7.x86_64
qemu-img-rhev-2.1.0-1.el7.x86_64
qemu-kvm-common-rhev-2.1.0-1.el7.x86_64
libcacard-devel-rhev-2.1.0-1.el7.x86_64
libcacard-rhev-2.1.0-1.el7.x86_64
qemu-kvm-rhev-2.1.0-1.el7.x86_64
qemu-kvm-tools-rhev-2.1.0-1.el7.x86_64
qemu-kvm-rhev-debuginfo-2.1.0-1.el7.x86_64


(qemu) info status 
VM status: running
(qemu) block I/O error in device 'drive_image1': No space left on device (28)
Formatting '/dev/rhel/raw30g', fmt=qcow2 size=32212254720 encryption=off cluster_size=65536 lazy_refcounts=off 
info block
drive_image1: /dev/rhel/lv10g (qcow2)
    I/O status:       nospace

drive_cd1: /root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso (raw, read-only)
    Removable device: not locked, tray closed
(qemu) info block-jobs 
Type mirror, device drive_image1: Completed 21831286784 of 32212254720 bytes, speed limit 0 bytes/s



# nc -U /tmp/monitor-qmpmonitor1-20140815-110559-kPOHZHpD 
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 2}, "package": " (qemu-kvm-2.1.0-1.el7)"}, "capabilities": []}}
{'execute': 'qmp_capabilities', 'id': 'XNISVhf3'}
{"return": {}, "id": "XNISVhf3"}
{"execute": "drive-mirror", "arguments": {"device": "drive_image1", "mode": "absolute-paths", "format": "qcow2", "target": "/dev/rhel/raw30g", "sync": "full"}, "id": "rUUcP4XX"}
{"return": {}, "id": "rUUcP4XX"}
{"timestamp": {"seconds": 1408092522, "microseconds": 227620}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_image1", "len": 32212254720, "offset": 32212254720, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1408092535, "microseconds": 154378}, "event": "RESUME"}
{"timestamp": {"seconds": 1408092535, "microseconds": 166620}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive_image1", "__com.redhat_debug_info": {"message": "No space left on device", "errno": 28}, "__com.redhat_reason": "enospc", "operation": "write", "action": "stop"}}
{"timestamp": {"seconds": 1408092535, "microseconds": 166835}, "event": "STOP"}
{"execute": "block-job-complete", "arguments": { "device": "drive_image1"} }
{"return": {}}
{"timestamp": {"seconds": 1408093092, "microseconds": 102092}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len": 32212254720, "offset": 32212254720, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1408093105, "microseconds": 762376}, "event": "RESUME"}


(qemu) info block
drive_image1: /dev/rhel/raw30g (qcow2)

drive_cd1: /root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso (raw, read-only)
    Removable device: not locked, tray closed
(qemu) info status 
VM status: running
(qemu)

Comment 7 Jan Kurik 2017-12-06 10:21:39 UTC
Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.

The official life cycle policy can be reviewed here:

http://redhat.com/rhel/lifecycle

This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:

https://access.redhat.com/