Bug 2013976

Hi Vadim,

Does this only apply to viroscsi, or viostor also need it? Thanks.

Regards,
Qianqian

(In reply to Qianqian Zhu from comment #1)
> Hi Vadim,
> 
> Does this only apply to viroscsi, or viostor also need it? Thanks.
> 
> Regards,
> Qianqian

Yep. 
viostor will require the same type of changes. but I would prefer to postpone this task until 
we fully implement and verify bus reset feature in vioscsi driver.

Best,
Vadim.

Hi Vadim,

Could you help to provide the detail reproduce steps and you used versions?

I tried to reproduce this issue with vioscsi, but I didn't reproduce it. My steps as:

1. Boot a vm up with virtio-scsi-pci.

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 1024 \
    -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -device pvpanic,ioport=0x505,id=idEUO235 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:8d:38:31:be:c9,id=idUmi8py,netdev=idBRC1Z2,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idBRC1Z2,vhost=on  \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -blockdev node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-212.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_virtio \
    -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \

2. Run iozone test on system disk, open 3 processes.
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'

3. update windows inside guest when do iozone, windows update can be successful.

4. Run iozone test again, and open 3 processes.
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'

5. Check cpu usage up to 65%, and restart the vm.

6. Check the Event log in vm, I did not find Event ID 129 in the system event log file.

Used versions:
kernel-4.18.0-345.1.el8.x86_64
qemu-kvm-6.1.0-1.module+el8.6.0+12721+8d053ff2.x86_64
virtio-win-prewhql-212
seabios-bin-1.14.0-1.module+el8.6.0+12721+8d053ff2.noarch

And I also tried test on a RHEL8.3.0 host, tested with pc, other steps are similar with upper, also cannot reproduce it:
kernel-4.18.0-240.el8.x86_64
qemu-kvm-common-5.2.0-16.module+el8.4.0+12191+622ad3bb.5.x86_64
virtio-win-prewhql-212
seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64

qemu commands:
-name guest=testing,debug-threads=on \
-device VGA,bus=pci.0 \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu Skylake-Server-noTSX-IBRS,ssbd=on,md-clear=on,mpx=off,hypervisor=on,pku=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-stimer,hv-reset,hv-frequencies,hv-reenlightenment,hv-tlbflush \
-m 4G \
-smp 8,maxcpus=128,sockets=16,dies=1,cores=8,threads=1 \
-object iothread,id=iothread1 \
-uuid 3cf071ed-52de-457b-be73-798e13c00d64 \
-smbios 'type=1,manufacturer=Red Hat,product=RHEL,version=8.3-1.el8ev,serial=8ac3453b-60f1-ac23-e811-68f024d26d80,uuid=3cf071ed-52de-457b-be73-798e13c00d64,family=RHV' \
-no-user-config \
-nodefaults \
-boot strict=on \
-device virtio-serial-pci,id=ua-4ee13c1f-4be3-4969-a158-4f1eabfe64e7,max_ports=16,bus=pci.0,addr=0x3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x7 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/win2019-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device virtio-net-pci,mac=9a:36:83:b6:3d:05,id=idJVpmsF,netdev=id23ZUK6,bus=pci.0  \
-netdev tap,id=id23ZUK6,vhost=on,script=/etc/qemu-ifup \
-vnc :10 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa \
-device qemu-xhci,id=usb1,bus=pci.0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-qmp tcp:0:1231,server,nowait \
-monitor stdio \

Thanks a lot~
Peixiu

(In reply to Peixiu Hou from comment #3)
> Hi Vadim,
> 
> Could you help to provide the detail reproduce steps and you used versions?
> 
> I tried to reproduce this issue with vioscsi, but I didn't reproduce it. My
> steps as:
> 
> 1. Boot a vm up with virtio-scsi-pci.
> 
> /usr/libexec/qemu-kvm \
>     -name 'avocado-vt-vm1'  \
>     -sandbox on  \
>     -machine q35 \
>     -device
> pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> chassis=1 \
>     -device
> pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
>     -nodefaults \
>     -device VGA,bus=pcie.0,addr=0x2 \
>     -m 1024 \
>     -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
>     -cpu
> 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,
> hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,
> hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
>     -device pvpanic,ioport=0x505,id=idEUO235 \
>     -device
> pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> chassis=2 \
>     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
>     -device
> pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> chassis=3 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
>     -blockdev
> node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.
> qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> flush=off,file=file_image1 \
>     -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
>     -device
> pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,
> chassis=4 \
>     -device
> virtio-net-pci,mac=9a:8d:38:31:be:c9,id=idUmi8py,netdev=idBRC1Z2,bus=pcie-
> root-port-3,addr=0x0  \
>     -netdev tap,id=idBRC1Z2,vhost=on  \
>     -blockdev
> node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,
> filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,
> cache.no-flush=off \
>     -blockdev
> node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-
> flush=off,file=file_cd1 \
>     -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
>     -blockdev
> node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-
> 0.1-212.iso,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-
> flush=off,file=file_virtio \
>     -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on  \
>     -vnc :0  \
>     -rtc base=localtime,clock=host,driftfix=slew  \
>     -boot menu=off,order=cdn,once=c,strict=off \
>     -enable-kvm \
>     -monitor stdio \
> 
> 2. Run iozone test on system disk, open 3 processes.
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> 
> 3. update windows inside guest when do iozone, windows update can be
> successful.
> 
> 4. Run iozone test again, and open 3 processes.
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> 
> 5. Check cpu usage up to 65%, and restart the vm.
> 
> 6. Check the Event log in vm, I did not find Event ID 129 in the system
> event log file.
> 
> Used versions:
> kernel-4.18.0-345.1.el8.x86_64
> qemu-kvm-6.1.0-1.module+el8.6.0+12721+8d053ff2.x86_64
> virtio-win-prewhql-212
> seabios-bin-1.14.0-1.module+el8.6.0+12721+8d053ff2.noarch
> 
> And I also tried test on a RHEL8.3.0 host, tested with pc, other steps are
> similar with upper, also cannot reproduce it:
> kernel-4.18.0-240.el8.x86_64
> qemu-kvm-common-5.2.0-16.module+el8.4.0+12191+622ad3bb.5.x86_64
> virtio-win-prewhql-212
> seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64
> 
> qemu commands:
> -name guest=testing,debug-threads=on \
> -device VGA,bus=pci.0 \
> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
> -cpu
> Skylake-Server-noTSX-IBRS,ssbd=on,md-clear=on,mpx=off,hypervisor=on,pku=on,
> hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-
> synic,hv-stimer,hv-reset,hv-frequencies,hv-reenlightenment,hv-tlbflush \
> -m 4G \
> -smp 8,maxcpus=128,sockets=16,dies=1,cores=8,threads=1 \
> -object iothread,id=iothread1 \
> -uuid 3cf071ed-52de-457b-be73-798e13c00d64 \
> -smbios 'type=1,manufacturer=Red
> Hat,product=RHEL,version=8.3-1.el8ev,serial=8ac3453b-60f1-ac23-e811-
> 68f024d26d80,uuid=3cf071ed-52de-457b-be73-798e13c00d64,family=RHV' \
> -no-user-config \
> -nodefaults \
> -boot strict=on \
> -device
> virtio-serial-pci,id=ua-4ee13c1f-4be3-4969-a158-4f1eabfe64e7,max_ports=16,
> bus=pci.0,addr=0x3 \
> -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x7 \
> -blockdev
> node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/win2019-64-virtio-scsi.qcow2,cache.direct=on,
> cache.no-flush=off \
> -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> flush=off,file=file_image1 \
> -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> -device
> virtio-net-pci,mac=9a:36:83:b6:3d:05,id=idJVpmsF,netdev=id23ZUK6,bus=pci.0  \
> -netdev tap,id=id23ZUK6,vhost=on,script=/etc/qemu-ifup \
> -vnc :10 \
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 \
> -object rng-random,id=objrng0,filename=/dev/urandom \
> -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa \
> -device qemu-xhci,id=usb1,bus=pci.0 \
> -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> -qmp tcp:0:1231,server,nowait \
> -monitor stdio \
> 
> Thanks a lot~
> Peixiu

Hi Peixiu,

It's a tough question at the moment. We supposes the the problem can be reproduced
on a extremely overcommitted network storage setup. I can reproduce this scenario 
on a modified qemu but it is only for own testing and debugging purpose.
CNV QE team is trying to reporuce this problem by decreasing IoTimeoutValue and running
fio inside of the guest. So let's wait for an update from this team.

Meanwhile, you can try decreasing the IoTimeoutValue for vioscsi by running the following
command line from elevated command prompt

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters /v IoTimeoutValue /t REG_DWORD /d 5 /f

"/d 5" in this case is the timout in seconds. I would suggest start with this value and then try to reduce it gradually.
Please be advices that the system reboot (or vioscsi driver disable/enable cycle) is required after every timeout parameter
changing event.


Best regards,
Vadim.

Hi Vadim,

Do we have any specific reason to remove the keyword 'RFE'? I though we agree with it on previous meeting.

Thanks,
Qianqian

(In reply to Qianqian Zhu from comment #11)
> Hi Vadim,
> 
> Do we have any specific reason to remove the keyword 'RFE'? I though we
> agree with it on previous meeting.
> 
> Thanks,
> Qianqian

Hoi Qianqian,
Did I do it ? 
Putting it back.
Sorry for any inconveniences.
Vadim>

merged upstream
https://github.com/virtio-win/kvm-guest-drivers-windows/pull/684