2013976 – Add bus reset handler to vioscsi driver.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2013976 - Add bus reset handler to vioscsi driver.

Summary: Add bus reset handler to vioscsi driver.

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	virtio-win
Sub Component:
Version:	9.0
Hardware:	Unspecified
OS:	Windows
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Vadim Rozenfeld
QA Contact:	Peixiu Hou
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	2010485 2028000
TreeView+	depends on / blocked

Reported:	2021-10-14 08:25 UTC by Vadim Rozenfeld
Modified:	2023-03-14 06:19 UTC (History)
CC List:	21 users (show)
Fixed In Version:	virtio-win-1.9.24-1.el8_4
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2028000 (view as bug list)
Environment:
Last Closed:	2022-01-27 02:08:02 UTC
Type:	Feature Request
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-99796	0	None	None	None	2021-10-14 09:08:27 UTC

Description Vadim Rozenfeld 2021-10-14 08:25:35 UTC

Description of problem:

Current version of viroscsi driver is not capable to handle 
LUN reset (SRB_FUNCTION_RESET_LOGICAL_UNIT) requests properly

Version-Release number of selected component (if applicable):


How reproducible:
Create a load conition that leads the storage stack to trigger timeout event
(Event ID 129 in the system event log file). Since the driver is not capable to
handle the reset request properly and complete all outstenting requests as
SRB_STATUS_BUSY, Windows keeps sending the reset request every 30 seconds making
the storage subsystem unresponsive.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Qianqian Zhu 2021-10-15 01:24:02 UTC

Hi Vadim,

Does this only apply to viroscsi, or viostor also need it? Thanks.

Regards,
Qianqian

Comment 2 Vadim Rozenfeld 2021-10-15 02:35:50 UTC

(In reply to Qianqian Zhu from comment #1)
> Hi Vadim,
> 
> Does this only apply to viroscsi, or viostor also need it? Thanks.
> 
> Regards,
> Qianqian

Yep. 
viostor will require the same type of changes. but I would prefer to postpone this task until 
we fully implement and verify bus reset feature in vioscsi driver.

Best,
Vadim.

Comment 3 Peixiu Hou 2021-10-15 07:58:08 UTC

Hi Vadim,

Could you help to provide the detail reproduce steps and you used versions?

I tried to reproduce this issue with vioscsi, but I didn't reproduce it. My steps as:

1. Boot a vm up with virtio-scsi-pci.

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 1024 \
    -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -device pvpanic,ioport=0x505,id=idEUO235 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:8d:38:31:be:c9,id=idUmi8py,netdev=idBRC1Z2,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idBRC1Z2,vhost=on  \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -blockdev node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-212.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_virtio \
    -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \

2. Run iozone test on system disk, open 3 processes.
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'

3. update windows inside guest when do iozone, windows update can be successful.

4. Run iozone test again, and open 3 processes.
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
# iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'

5. Check cpu usage up to 65%, and restart the vm.

6. Check the Event log in vm, I did not find Event ID 129 in the system event log file.

Used versions:
kernel-4.18.0-345.1.el8.x86_64
qemu-kvm-6.1.0-1.module+el8.6.0+12721+8d053ff2.x86_64
virtio-win-prewhql-212
seabios-bin-1.14.0-1.module+el8.6.0+12721+8d053ff2.noarch

And I also tried test on a RHEL8.3.0 host, tested with pc, other steps are similar with upper, also cannot reproduce it:
kernel-4.18.0-240.el8.x86_64
qemu-kvm-common-5.2.0-16.module+el8.4.0+12191+622ad3bb.5.x86_64
virtio-win-prewhql-212
seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64

qemu commands:
-name guest=testing,debug-threads=on \
-device VGA,bus=pci.0 \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu Skylake-Server-noTSX-IBRS,ssbd=on,md-clear=on,mpx=off,hypervisor=on,pku=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-stimer,hv-reset,hv-frequencies,hv-reenlightenment,hv-tlbflush \
-m 4G \
-smp 8,maxcpus=128,sockets=16,dies=1,cores=8,threads=1 \
-object iothread,id=iothread1 \
-uuid 3cf071ed-52de-457b-be73-798e13c00d64 \
-smbios 'type=1,manufacturer=Red Hat,product=RHEL,version=8.3-1.el8ev,serial=8ac3453b-60f1-ac23-e811-68f024d26d80,uuid=3cf071ed-52de-457b-be73-798e13c00d64,family=RHV' \
-no-user-config \
-nodefaults \
-boot strict=on \
-device virtio-serial-pci,id=ua-4ee13c1f-4be3-4969-a158-4f1eabfe64e7,max_ports=16,bus=pci.0,addr=0x3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x7 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/win2019-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device virtio-net-pci,mac=9a:36:83:b6:3d:05,id=idJVpmsF,netdev=id23ZUK6,bus=pci.0  \
-netdev tap,id=id23ZUK6,vhost=on,script=/etc/qemu-ifup \
-vnc :10 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa \
-device qemu-xhci,id=usb1,bus=pci.0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-qmp tcp:0:1231,server,nowait \
-monitor stdio \

Thanks a lot~
Peixiu

Comment 4 Vadim Rozenfeld 2021-10-15 09:09:45 UTC

(In reply to Peixiu Hou from comment #3)
> Hi Vadim,
> 
> Could you help to provide the detail reproduce steps and you used versions?
> 
> I tried to reproduce this issue with vioscsi, but I didn't reproduce it. My
> steps as:
> 
> 1. Boot a vm up with virtio-scsi-pci.
> 
> /usr/libexec/qemu-kvm \
>     -name 'avocado-vt-vm1'  \
>     -sandbox on  \
>     -machine q35 \
>     -device
> pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> chassis=1 \
>     -device
> pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
>     -nodefaults \
>     -device VGA,bus=pcie.0,addr=0x2 \
>     -m 1024 \
>     -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
>     -cpu
> 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,
> hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,
> hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
>     -device pvpanic,ioport=0x505,id=idEUO235 \
>     -device
> pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> chassis=2 \
>     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
>     -device
> pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> chassis=3 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
>     -blockdev
> node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.
> qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> flush=off,file=file_image1 \
>     -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
>     -device
> pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,
> chassis=4 \
>     -device
> virtio-net-pci,mac=9a:8d:38:31:be:c9,id=idUmi8py,netdev=idBRC1Z2,bus=pcie-
> root-port-3,addr=0x0  \
>     -netdev tap,id=idBRC1Z2,vhost=on  \
>     -blockdev
> node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,
> filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,
> cache.no-flush=off \
>     -blockdev
> node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-
> flush=off,file=file_cd1 \
>     -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
>     -blockdev
> node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-
> 0.1-212.iso,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-
> flush=off,file=file_virtio \
>     -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on  \
>     -vnc :0  \
>     -rtc base=localtime,clock=host,driftfix=slew  \
>     -boot menu=off,order=cdn,once=c,strict=off \
>     -enable-kvm \
>     -monitor stdio \
> 
> 2. Run iozone test on system disk, open 3 processes.
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> 
> 3. update windows inside guest when do iozone, windows update can be
> successful.
> 
> 4. Run iozone test again, and open 3 processes.
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> # iozone.exe -azR -r 64k -n 1G -g 10G -M -i 0 -i 1 -I -f C:\\testfile'
> 
> 5. Check cpu usage up to 65%, and restart the vm.
> 
> 6. Check the Event log in vm, I did not find Event ID 129 in the system
> event log file.
> 
> Used versions:
> kernel-4.18.0-345.1.el8.x86_64
> qemu-kvm-6.1.0-1.module+el8.6.0+12721+8d053ff2.x86_64
> virtio-win-prewhql-212
> seabios-bin-1.14.0-1.module+el8.6.0+12721+8d053ff2.noarch
> 
> And I also tried test on a RHEL8.3.0 host, tested with pc, other steps are
> similar with upper, also cannot reproduce it:
> kernel-4.18.0-240.el8.x86_64
> qemu-kvm-common-5.2.0-16.module+el8.4.0+12191+622ad3bb.5.x86_64
> virtio-win-prewhql-212
> seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64
> 
> qemu commands:
> -name guest=testing,debug-threads=on \
> -device VGA,bus=pci.0 \
> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
> -cpu
> Skylake-Server-noTSX-IBRS,ssbd=on,md-clear=on,mpx=off,hypervisor=on,pku=on,
> hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-
> synic,hv-stimer,hv-reset,hv-frequencies,hv-reenlightenment,hv-tlbflush \
> -m 4G \
> -smp 8,maxcpus=128,sockets=16,dies=1,cores=8,threads=1 \
> -object iothread,id=iothread1 \
> -uuid 3cf071ed-52de-457b-be73-798e13c00d64 \
> -smbios 'type=1,manufacturer=Red
> Hat,product=RHEL,version=8.3-1.el8ev,serial=8ac3453b-60f1-ac23-e811-
> 68f024d26d80,uuid=3cf071ed-52de-457b-be73-798e13c00d64,family=RHV' \
> -no-user-config \
> -nodefaults \
> -boot strict=on \
> -device
> virtio-serial-pci,id=ua-4ee13c1f-4be3-4969-a158-4f1eabfe64e7,max_ports=16,
> bus=pci.0,addr=0x3 \
> -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x7 \
> -blockdev
> node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/win2019-64-virtio-scsi.qcow2,cache.direct=on,
> cache.no-flush=off \
> -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> flush=off,file=file_image1 \
> -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> -device
> virtio-net-pci,mac=9a:36:83:b6:3d:05,id=idJVpmsF,netdev=id23ZUK6,bus=pci.0  \
> -netdev tap,id=id23ZUK6,vhost=on,script=/etc/qemu-ifup \
> -vnc :10 \
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 \
> -object rng-random,id=objrng0,filename=/dev/urandom \
> -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0xa \
> -device qemu-xhci,id=usb1,bus=pci.0 \
> -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> -qmp tcp:0:1231,server,nowait \
> -monitor stdio \
> 
> Thanks a lot~
> Peixiu

Hi Peixiu,

It's a tough question at the moment. We supposes the the problem can be reproduced
on a extremely overcommitted network storage setup. I can reproduce this scenario 
on a modified qemu but it is only for own testing and debugging purpose.
CNV QE team is trying to reporuce this problem by decreasing IoTimeoutValue and running
fio inside of the guest. So let's wait for an update from this team.

Meanwhile, you can try decreasing the IoTimeoutValue for vioscsi by running the following
command line from elevated command prompt

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters /v IoTimeoutValue /t REG_DWORD /d 5 /f

"/d 5" in this case is the timout in seconds. I would suggest start with this value and then try to reduce it gradually.
Please be advices that the system reboot (or vioscsi driver disable/enable cycle) is required after every timeout parameter
changing event.


Best regards,
Vadim.

Comment 11 Qianqian Zhu 2021-10-20 01:37:22 UTC

Hi Vadim,

Do we have any specific reason to remove the keyword 'RFE'? I though we agree with it on previous meeting.

Thanks,
Qianqian

Comment 12 Vadim Rozenfeld 2021-10-20 02:10:43 UTC

(In reply to Qianqian Zhu from comment #11)
> Hi Vadim,
> 
> Do we have any specific reason to remove the keyword 'RFE'? I though we
> agree with it on previous meeting.
> 
> Thanks,
> Qianqian

Hoi Qianqian,
Did I do it ? 
Putting it back.
Sorry for any inconveniences.
Vadim>

Comment 50 Vadim Rozenfeld 2021-12-01 09:09:17 UTC

merged upstream
https://github.com/virtio-win/kvm-guest-drivers-windows/pull/684

Note You need to log in before you can comment on or make changes to this bug.

ailan
coli
danken
fdeutsch
hpopal
ipinto
jinzhao
juzhang
mdean
menli
mkedzier
mprivozn
pelauter
phou
qinwang
qizhu
rcain
rmohr
xiagao
xzhou
zhguo