Bug 1745393

Summary: Leaked clusters were noticed during image check.No data integrity problem was found though.
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Lei Yang <leiyang>
Component: qemu-kvmAssignee: Ademar Reis <areis>
Status: CLOSED CURRENTRELEASE QA Contact: Tingting Mao <timao>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 8.1CC: chayang, coli, jinzhao, juzhang, pezhang, timao, virt-maint, xfu, xiagao, yuhuang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-20 06:48:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1764721    
Bug Blocks:    
Attachments:
Description Flags
debug log none

Description Lei Yang 2019-08-26 03:16:31 UTC
Created attachment 1607982 [details]
debug log

Description of problem:
Autotest with virtual network test loop and check the test result.Will be found the error message in debug log.The error message as follows:
13:18:05 ERROR| RuntimeError: Failures occurred while postprocess:
13:18:05 ERROR| Postprocess: Leaked clusters were noticed during image check. No data integrity problem was found though. (/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2) 

Version-Release number of selected component (if applicable):
qemu-img-4.1.0-4.module+el8.1.0+4020+16089f93.x86_64
kernel-4.18.0-135.el8.x86_64

How reproducible:
Use different parameter test:
drive_cache = none  -->(reproduced)
drive_cache = directsync  -->(reproduced)
cache-clean-interval = 1 -->(reproduced)

Steps to Reproduce:
1.use parameter (drive_cache = none) test.
# avocado run --vt-config /var/lib/avocado/data/avocado-vt/backends/qemu/cfg/tests-example.cfg
QEMU CLI:
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_i2xf9x63/monitor-qmpmonitor1-20190820-130640-y4exqeNK,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_i2xf9x63/monitor-catch_monitor-20190820-130640-y4exqeNK,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idFJ61uv \
    -chardev socket,nowait,server,path=/var/tmp/avocado_i2xf9x63/serial-serial0-20190820-130640-y4exqeNK,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20190820-130640-y4exqeNK,path=/var/tmp/avocado_i2xf9x63/seabios-20190820-130640-y4exqeNK,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190820-130640-y4exqeNK,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:93:0a:43:8a:22,status=on,id=id4cSQuG,netdev=idGTeUiz,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idGTeUiz,vhost=on,vhostfd=22,fd=21 \
    -m 7168  \
    -smp 6,maxcpus=6,cores=3,threads=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0

2.use parameter (drive_cache = directsync) test.
# avocado run --vt-config /var/lib/avocado/data/avocado-vt/backends/qemu/cfg/tests-example1.cfg
QEMU CLI:
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_uyaazerp/monitor-qmpmonitor1-20190822-153634-tkkk0Ane,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_uyaazerp/monitor-catch_monitor-20190822-153634-tkkk0Ane,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idcnL8WS \
    -chardev socket,nowait,id=chardev_serial0,path=/var/tmp/avocado_uyaazerp/serial-serial0-20190822-153634-tkkk0Ane,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20190822-153634-tkkk0Ane,path=/var/tmp/avocado_uyaazerp/seabios-20190822-153634-tkkk0Ane,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190822-153634-tkkk0Ane,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,cache=directsync,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:19:76:38:1a:61,id=idS5BJdl,netdev=idjyW4LA,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idjyW4LA,vhost=on,vhostfd=23,fd=22 \
    -m 7168  \
    -smp 6,maxcpus=6,cores=3,threads=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0


Actual results:
The debug log appered error message.

Expected results:
No rror mesage appered debug log.

Additional info:

1.Will use the following parameters to continu testing.
drive_cache = writeback
drive_cache = writethrough
drive_cache = unsafe

2. This issue must be autotest to reproduced.

Comment 3 Ademar Reis 2019-08-28 14:56:14 UTC
We need a way to reproduce this outside of the avocado-vt environment, which is challenging to debug. There could be a problem with the image from a previous test, or from different steps performed during the testing.

Comment 4 Lei Yang 2019-08-29 06:28:35 UTC
(In reply to Ademar Reis from comment #3)
> We need a way to reproduce this outside of the avocado-vt environment, which
> is challenging to debug. There could be a problem with the image from a
> previous test, or from different steps performed during the testing.

Hi,Ademar

I have tried to manually reproduce this problem, but it's difficult to reproduce.

Best regards.
LeiYang

Comment 7 xiagao 2019-09-03 02:33:45 UTC
Hit this issue with automation test.

drive_cache = none

test case:
020-Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.Win2012.x86_64.r2.io-github-autotest-qemu.virtio_driver_sign_check.with_viostor.virtio_win_iso_media.q35

qemu version:
qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64

Comment 8 Lei Yang 2019-09-18 03:26:44 UTC
Hi,Tingting Mao

I verified this bz in the latest qemu package 'qemu-img-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64'. There is not reproduce this issue.
So could you please test again with latest qemu package?From my point of view, there should be not a bug now.
Test Version:
rhel810-64-virtio-scsi.qcow
Job link:
http://10.66.4.244/kvm_autotest_job_log/?jobid=3793056

Thanks,
LeiYang

Comment 9 Tingting Mao 2019-09-20 06:48:35 UTC
From my test, I ran qcow2 loop with 218 cases with win2012 and win2012r automatically. Just hit 1 case failed for cluster leak, and not image corruption. 

Plus confirmed with xfu, there is no the issue in acceptance test with latest qemu package. 

So I will close this bug now. Reopen it if anyone who can reproduce it in the future, please.

Comment 10 Lei Yang 2019-10-24 08:58:52 UTC
Reproduce this issue with automation test:

drive_cache=none
test case:
059-Host_RHEL.m8.u2.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.set_link.reboot_system_reset.status_on.q35/

Test version:
KVM version: 4.18.0-147.6.el8.x86_64
QEMU version: qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64

Job link:
http://10.66.4.244/autotest_static_job_log/3840588/