RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1787022 - Windows virtio-scsi driver performs poorly when reading/writing big data blocks
Summary: Windows virtio-scsi driver performs poorly when reading/writing big data blocks
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: virtio-win
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Vadim Rozenfeld
QA Contact: Yanhui Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-30 12:28 UTC by Vadim Rozenfeld
Modified: 2020-11-04 04:17 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 04:17:35 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)

Description Vadim Rozenfeld 2019-12-30 12:28:44 UTC 
Description of problem:
Windows virtio-scsi driver performs poorly when reading/writing data blocks bigger than 128K

Version-Release number of selected component (if applicable):


How reproducible:

Always

Steps to Reproduce:
1.
Extend iometer tests with read/write cases for transfer size 256K/512K/1M/2M/4M

2.
Run the test and check performance results.  

3.
Windows virtio-scsi driver doesn't perform optimal on big data transfers 

Actual results:


Expected results:


Additional info:
Comment 3 Vadim Rozenfeld 2020-01-22 06:22:52 UTC 
(In reply to Quan Wenli from comment #2)
> Hello Vadim
> 
> Could you help review the results which comparison between build 175 and 177
> with 128k/256k/1m/2m fio tests
> 
> 
> - in Win2019 result, some degradation in read/write/randread tests.
> 
>  http://10.73.60.69/results/request/Bug1787022/raw.virtio_scsi.*.Win2019.
> x86_64.html 
> 
> - in Win2012r2 result, slightly degradation in read/write tests.
> 
>  http://10.73.60.69/results/request/Bug1787022/raw.virtio_scsi.*.Win2012.
> x86_64.r2.html

Thanks a lot for the testing.
Can you please run this test on NTFS-formatted volume?

Best,
Vadim.
Comment 4 Quan Wenli 2020-01-22 08:27:51 UTC 
(In reply to Vadim Rozenfeld from comment #3)
> (In reply to Quan Wenli from comment #2)
> > Hello Vadim
> > 
> > Could you help review the results which comparison between build 175 and 177
> > with 128k/256k/1m/2m fio tests
> > 
> > 
> > - in Win2019 result, some degradation in read/write/randread tests.
> > 
> >  http://10.73.60.69/results/request/Bug1787022/raw.virtio_scsi.*.Win2019.
> > x86_64.html 
> > 
> > - in Win2012r2 result, slightly degradation in read/write tests.
> > 
> >  http://10.73.60.69/results/request/Bug1787022/raw.virtio_scsi.*.Win2012.
> > x86_64.r2.html
> 
> Thanks a lot for the testing.
> Can you please run this test on NTFS-formatted volume?

OK, I will update the results after china holiday. 

> 
> Best,
> Vadim.
Comment 11 Vadim Rozenfeld 2020-03-30 11:05:46 UTC 
After some considerations I desided to make the maximum transfer size adjustable, rather 
then keep it hard coded. By default the maximum transfer size will be 256K (this is 
value was hard coded since the first release of viostor driver). While 256K works
quite good for file and nvme back-ended scnarious, for some cases, like scsi_dbg RAM
disk its performance is sub-optimal. 
With the lates changes 
https://github.com/virtio-win/kvm-guest-drivers-windows/commit/a18fc89a8139884dc51add0f641c466f2d9826b2
the maximum transfer lenght can be adjusted by creating and editing the following Registry key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters\Device]
"PhysicalBreaks"=dword:000000ff
Comment 16 Vadim Rozenfeld 2020-06-14 08:45:39 UTC 
New Registry key "PhysicalBreaks" has been introduced in build 181 and available now.

 Please use the following .reg file to add "PhysicalBreaks" parameter to the Registry
    
    Windows Registry Editor Version 5.00
    
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters\Device]
    "PhysicalBreaks"=dword:000000ff

By modifying "PhysicalBreaks" we can redefine the maximum transfer lengte that miniport 
drivers reports to the rest of the OS. "PhysicalBreaks" can be adjusted in the range
16..255 which is 64K..1024K maximum transfer length. If "PhysicalBreaks" is not explicitly
specified in the Registry, then miniport driver will report the default value - 64 breaks/
256K maximum transfer length.
While we didn't go through the proper performance test cycle yet, some preliminary results 
show up to 10..15% performance improvident on particular back-ends (scsi_dbg, for example)
for 512K/1024K sequential read/write loads.
Comment 17 Yanhui Ma 2020-06-22 05:50:10 UTC 
(In reply to Vadim Rozenfeld from comment #16)
> New Registry key "PhysicalBreaks" has been introduced in build 181 and
> available now.
> 
>  Please use the following .reg file to add "PhysicalBreaks" parameter to the
> Registry
>     
>     Windows Registry Editor Version 5.00
>     
>    
> [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters\Devi
> ce]
>     "PhysicalBreaks"=dword:000000ff
> 
> By modifying "PhysicalBreaks" we can redefine the maximum transfer lengte
> that miniport 
> drivers reports to the rest of the OS. "PhysicalBreaks" can be adjusted in
> the range
> 16..255 which is 64K..1024K maximum transfer length. If "PhysicalBreaks" is
> not explicitly
> specified in the Registry, then miniport driver will report the default
> value - 64 breaks/
> 256K maximum transfer length.
> While we didn't go through the proper performance test cycle yet, some
> preliminary results 
> show up to 10..15% performance improvident on particular back-ends
> (scsi_dbg, for example)
> for 512K/1024K sequential read/write loads.

Hi Vadim,

I want to confirm some questions with you about how to verify the bug. Do you think following tests can verify the bug? If no, could you tell us how to better verify it? 

Scenario 1:
Don't specify the "PhysicalBreaks", use the default 256k, then compare fio performance between virtio-win-prewhql-180 and virtio-win-prewhql-181 with following steps.

1. on host:
#mkfs.xfs /dev/sdb & mount /dev/sdb /mnt/test
/dev/sdb is a 60GB ssd.
#qemu-img create -f qcow2 /mnt/test/storage2.qcow2 40G

2.boot a guest with above image:
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \
    -drive id=drive_disk1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/mnt/test/storage2.qcow2 \

3. running fio on the data disk created by step1 on guest:
echo select disk 1 > imDiskpart.script && echo create partition primary >> imDiskpart.script && echo assign letter=I >> imDiskpart.script&& echo exit >> imDiskpart.script && diskpart /s imDiskpart.script && format I: /FS:NTFS /V:local /Q /y

fio.exe --rw=%s --bs=%s --iodepth=%s --runtime=1m --direct=1 --filename=I\:\%s_%s_%s --name=job1 --ioengine=windowsaio --thread --group_reporting --numjobs=16 --size=512MB --time_based --output="C:\\fio_result"

- rw: read write randread randwrite randrw
- bs: 256k 512k 1M 2M
- iodepth: 1 8 64

4. expected results: no performance regression

Scenario 2:

Adjusting the "PhysicalBreaks" to run above test again, what value do you recommend?

Thanks Yanhui.
Comment 18 Vadim Rozenfeld 2020-07-07 06:53:20 UTC 
(In reply to Yanhui Ma from comment #17)
> (In reply to Vadim Rozenfeld from comment #16)
> > New Registry key "PhysicalBreaks" has been introduced in build 181 and
> > available now.
> > 
> >  Please use the following .reg file to add "PhysicalBreaks" parameter to the
> > Registry
> >     
> >     Windows Registry Editor Version 5.00
> >     
> >    
> > [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vioscsi\Parameters\Devi
> > ce]
> >     "PhysicalBreaks"=dword:000000ff
> > 
> > By modifying "PhysicalBreaks" we can redefine the maximum transfer lengte
> > that miniport 
> > drivers reports to the rest of the OS. "PhysicalBreaks" can be adjusted in
> > the range
> > 16..255 which is 64K..1024K maximum transfer length. If "PhysicalBreaks" is
> > not explicitly
> > specified in the Registry, then miniport driver will report the default
> > value - 64 breaks/
> > 256K maximum transfer length.
> > While we didn't go through the proper performance test cycle yet, some
> > preliminary results 
> > show up to 10..15% performance improvident on particular back-ends
> > (scsi_dbg, for example)
> > for 512K/1024K sequential read/write loads.
> 
> Hi Vadim,
> 
> I want to confirm some questions with you about how to verify the bug. Do
> you think following tests can verify the bug? If no, could you tell us how
> to better verify it? 
> 
> Scenario 1:
> Don't specify the "PhysicalBreaks", use the default 256k, then compare fio
> performance between virtio-win-prewhql-180 and virtio-win-prewhql-181 with
> following steps.
> 
> 1. on host:
> #mkfs.xfs /dev/sdb & mount /dev/sdb /mnt/test
> /dev/sdb is a 60GB ssd.
> #qemu-img create -f qcow2 /mnt/test/storage2.qcow2 40G
> 
> 2.boot a guest with above image:
>     -device
> virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \
>     -drive
> id=drive_disk1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/mnt/test/storage2.qcow2 \
> 
> 3. running fio on the data disk created by step1 on guest:
> echo select disk 1 > imDiskpart.script && echo create partition primary >>
> imDiskpart.script && echo assign letter=I >> imDiskpart.script&& echo exit
> >> imDiskpart.script && diskpart /s imDiskpart.script && format I: /FS:NTFS
> /V:local /Q /y
> 
> fio.exe --rw=%s --bs=%s --iodepth=%s --runtime=1m --direct=1
> --filename=I\:\%s_%s_%s --name=job1 --ioengine=windowsaio --thread
> --group_reporting --numjobs=16 --size=512MB --time_based
> --output="C:\\fio_result"
> 
> - rw: read write randread randwrite randrw
> - bs: 256k 512k 1M 2M
> - iodepth: 1 8 64
> 
> 4. expected results: no performance regression
> 
> Scenario 2:
> 
> Adjusting the "PhysicalBreaks" to run above test again, what value do you
> recommend?
> 

Currently it limited by 255 as the upper limit and 16 as the lower. 
So any value greater than 0x100 will be trimmed down. In the future
we will probably adjust this value. But for now it should be absolutely
and reasonable safe to check two scenarios - default (no PhysicalBreaks
key created or PhysicalBreaks = 64 (0x40h)  vs PhysicalBreaks = 256 (0x100h)

Sorry again for delayed response.
Vadim.

 
> Thanks Yanhui.
Comment 19 Yanhui Ma 2020-07-13 04:03:33 UTC 
(In reply to Vadim Rozenfeld from comment #18)

> > 
> > Scenario 2:
> > 
> > Adjusting the "PhysicalBreaks" to run above test again, what value do you
> > recommend?
> > 
> 
> Currently it limited by 255 as the upper limit and 16 as the lower. 
> So any value greater than 0x100 will be trimmed down. In the future
> we will probably adjust this value. But for now it should be absolutely
> and reasonable safe to check two scenarios - default (no PhysicalBreaks
> key created or PhysicalBreaks = 64 (0x40h)  vs PhysicalBreaks = 256 (0x100h)
> 
> Sorry again for delayed response.
> Vadim.
>

Here are results between default(no PhysicalBreaks key created) and PhysicalBreaks = 256 (0x100h) with virtio-win-prewhql-0.1-185:

http://kvm-perf.englab.nay.redhat.com/results/regression/bug1787022/qcow2.virtio_scsi.*.Win2019.x86_64.html

No regression, there are improvements for some data. But the standard deviation for some data is big. I will run it again.
 
>  
> > Thanks Yanhui.
Comment 20 Yanhui Ma 2020-07-14 05:06:44 UTC 
(In reply to Yanhui Ma from comment #19)
> (In reply to Vadim Rozenfeld from comment #18)
> 
> > > 
> > > Scenario 2:
> > > 
> > > Adjusting the "PhysicalBreaks" to run above test again, what value do you
> > > recommend?
> > > 
> > 
> > Currently it limited by 255 as the upper limit and 16 as the lower. 
> > So any value greater than 0x100 will be trimmed down. In the future
> > we will probably adjust this value. But for now it should be absolutely
> > and reasonable safe to check two scenarios - default (no PhysicalBreaks
> > key created or PhysicalBreaks = 64 (0x40h)  vs PhysicalBreaks = 256 (0x100h)
> > 
> > Sorry again for delayed response.
> > Vadim.
> >
> 
> Here are results between default(no PhysicalBreaks key created) and
> PhysicalBreaks = 256 (0x100h) with virtio-win-prewhql-0.1-185:
> 
> http://kvm-perf.englab.nay.redhat.com/results/regression/bug1787022/qcow2.
> virtio_scsi.*.Win2019.x86_64.html
> 
> No regression, there are improvements for some data. But the standard
> deviation for some data is big. I will run it again.
> 

Here are the second test results:
http://kvm-perf.englab.nay.redhat.com/results/regression/bug1787022/repeat/qcow2.virtio_scsi.*.Win2019.x86_64.html 

Based on the results, set the bug verified.

> >  
> > > Thanks Yanhui.
Comment 21 Vadim Rozenfeld 2020-07-14 05:58:38 UTC 
(In reply to Yanhui Ma from comment #20)
> (In reply to Yanhui Ma from comment #19)
> > (In reply to Vadim Rozenfeld from comment #18)
> > 
> > > > 
> > > > Scenario 2:
> > > > 
> > > > Adjusting the "PhysicalBreaks" to run above test again, what value do you
> > > > recommend?
> > > > 
> > > 
> > > Currently it limited by 255 as the upper limit and 16 as the lower. 
> > > So any value greater than 0x100 will be trimmed down. In the future
> > > we will probably adjust this value. But for now it should be absolutely
> > > and reasonable safe to check two scenarios - default (no PhysicalBreaks
> > > key created or PhysicalBreaks = 64 (0x40h)  vs PhysicalBreaks = 256 (0x100h)
> > > 
> > > Sorry again for delayed response.
> > > Vadim.
> > >
> > 
> > Here are results between default(no PhysicalBreaks key created) and
> > PhysicalBreaks = 256 (0x100h) with virtio-win-prewhql-0.1-185:
> > 
> > http://kvm-perf.englab.nay.redhat.com/results/regression/bug1787022/qcow2.
> > virtio_scsi.*.Win2019.x86_64.html
> > 
> > No regression, there are improvements for some data. But the standard
> > deviation for some data is big. I will run it again.
> > 
> 
> Here are the second test results:
> http://kvm-perf.englab.nay.redhat.com/results/regression/bug1787022/repeat/
> qcow2.virtio_scsi.*.Win2019.x86_64.html 
> 
> Based on the results, set the bug verified.

Thank you a lot.
Vadim.
> 
> > >  
> > > > Thanks Yanhui.
Comment 25 errata-xmlrpc 2020-11-04 04:17:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virtio-win bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4840
Note You need to log in before you can comment on or make changes to this bug.