RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1067225 - Windows guest performing out-of-bounds accesses on virtio device
Summary: Windows guest performing out-of-bounds accesses on virtio device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: virtio-win
Version: 6.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Vadim Rozenfeld
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1388553 (view as bug list)
Depends On:
Blocks: 1195487
TreeView+ depends on / blocked
 
Reported: 2014-02-19 23:59 UTC by David Gibson
Modified: 2019-08-15 03:46 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, Windows guests in some cases issued out-of-bounds read and write requests to a virtio device, which caused the guest to become unresponsive. Now, the Windows virtio-blk device drive performs logical block addressing (LBA) sanity checks before submitting requests to QEMU, and does not submit requests that are out-of-bounds. As a result, the described problem no longer occurs.
Clone Of:
: 1195487 (view as bug list)
Environment:
Last Closed: 2016-05-10 16:35:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Patch showing diagnostic alterations to qemu (7.98 KB, patch)
2014-02-20 00:21 UTC, David Gibson 		no flags Details | Diff
couldnotshutdown (251.48 KB, image/png)
2014-10-21 07:07 UTC, Min Deng 		no flags Details
rhevandspice-install-service-up (87.74 KB, image/png)
2014-10-21 07:08 UTC, Min Deng 		no flags Details
fromdevicemanager (83.46 KB, image/png)
2014-10-21 07:09 UTC, Min Deng 		no flags Details
nclogfromhost (10.21 KB, text/plain)
2014-10-21 07:11 UTC, Min Deng 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 740963 0 None None None Never
Red Hat Product Errata RHBA-2016:1011 0 normal SHIPPED_LIVE virtio-win bug fix and enhancement update 2016-05-10 20:35:16 UTC

Description David Gibson 2014-02-19 23:59:02 UTC 
Description of problem:

NOTE: virtio-win component is just a guess.  The problem is definitely somewhere inside the Windows guest, and virtio-win is the only plausibly responsible component that is RH provided, so we need to either find the bug there, or rule it out.

Several Windows Server 2008R2 guests under RHEV were periodically pausing due to an IO error.  Adding debugging to qemu showed that these EIO errors were because the guest itself was attempting virtio accesses beyond the logical end of device.

Version-Release number of selected component (if applicable):

virtio-win-1.4.0.iso (I think)

How reproducible:

Customer has several systems which trigger this problem daily.

Steps to Reproduce:

Happens daily on customer system.  Usually about the same time each day, so it's probably triggered by a scheduled job, but the exact sequence of guest-side events which trigger this problem is unknown so far.

Actual results:

Guest issues out-of-bounds accesses to the virtio virtual hardware, causing qemu to pause the guest.

Expected results:

virtio-win drivers reject out of bounds access requests and report them on the guest side.

Additional info:

The accesses have a strange pattern.  They appear to come in a burst, the first is the sector immediately beyond the logical end of device, then there are several more each 2-4x the distance beyond the end-of-device as the last one.

qemu diagnostics with more details coming.
Comment 3 David Gibson 2014-02-20 00:21:07 UTC 
Created attachment 865302 [details]
Patch showing diagnostic alterations to qemu

This is the debug patch I used to generate the diagnostics for out of bounds accesses.  The code can also be found in the 'private-dgibson-sfdc01018528' branch of the qemu-kvm-rhev rhpkg tree.
Comment 4 Mike Cao 2014-02-20 01:27:50 UTC 
Can you try to reproduce it with virito-win-1.6.8-4 ?
Do you know which app is running when this issue occurs ?
Comment 9 Mike Cao 2014-04-09 03:29:42 UTC 
This bug may dup of https://bugzilla.redhat.com/show_bug.cgi?id=1080996
Comment 10 David Gibson 2014-04-10 00:45:06 UTC 
@Mike,

This is closely related to bug 1080996, but it's not a dupe per se.

Bug 1080996 (itself a dupe of 1064643) is covering the fact that qemu and the stack above don't deal well with errors like this which are unambiguously the guest's fault.  The proposed fix is to differently classify guest parameter caused errors, and (usually) always report them rather than pausing the VM.

This bug is addressing the fact that the guest is initiating the bad accesses in the first place.  The proposal is to either fix the virtio-win drivers so it doesn't make these accesses, or to determine that the problem lies within something else on the guest side.
Comment 11 Vadim Rozenfeld 2014-09-17 05:59:16 UTC 
Hi Mike,

Can we try reproducing this problem on a fresh system with the following apps
installed:

Sophos Anti-Virus
Sophos AutoUpdate
Sophos Remote Management System

Thanks,
Vadim.
Comment 12 Mike Cao 2014-09-18 02:59:44 UTC 
mdeng ,pls handle the needinfo per comment #11.
Comment 14 Min Deng 2014-10-21 07:07:33 UTC 
Created attachment 948817 [details]
couldnotshutdown
Comment 15 Min Deng 2014-10-21 07:08:19 UTC 
Created attachment 948818 [details]
rhevandspice-install-service-up
Comment 16 Min Deng 2014-10-21 07:09:00 UTC 
Created attachment 948819 [details]
fromdevicemanager
Comment 17 Min Deng 2014-10-21 07:11:16 UTC 
Created attachment 948820 [details]
nclogfromhost
Comment 19 Vadim Rozenfeld 2015-01-16 07:46:27 UTC 
(In reply to David Gibson from comment #10)

> This bug is addressing the fact that the guest is initiating the bad
> accesses in the first place.  The proposal is to either fix the virtio-win
> drivers so it doesn't make these accesses, or to determine that the problem
> lies within something else on the guest side.

It looks like some application(s) - presumably Sophos Anti-Virus performs raw reads/writes, bypassing file system driver.
I will add an extra sanity check to validate IO boundaries.
Comment 25 Vadim Rozenfeld 2015-03-03 02:52:19 UTC 
Should be fixed in build 101, available at http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/101/win/virtio-win-prewhql-0.1.zip
Comment 26 lijin 2015-04-09 06:52:46 UTC 
cannot reproduce this issue from QE side,guest works fine.

package info:
kernel-2.6.32-540.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64
virtio-win-1.6.4-1.el6_4.noarch/virtio-win-prewhql-102
seabios-0.6.1.2-29.el6.x86_64

steps:
1.boot win2k8R2 guest with virtio-blk device:
2.install software “Sophos Endpoint Security and Control 10.3” which include "Sophos Anti-Virus" and "Sophos AutoUpdate" on guest;
3.configure schedule of scan computers and autoupdate of sophos
4.make the guest running two days

guest works fine after Sophos Anti-Virus scan the computer;
And I got following error message after Sophos AutoUpdate try to update,I guess it's due to I have no product license:
Message: ERROR:   Download of Sophos Endpoint Security and Control failed from server sophos
Message: ERROR:   Could not find a source for updated packages
Comment 27 Vadim Rozenfeld 2015-05-24 06:13:22 UTC 
Please re-check again with the latest build http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/104/win/virtio-win-prewhql-0.1.zip

Thanks,
Vadim.
Comment 28 lijin 2015-05-28 02:44:29 UTC 
Mike,please verify the bug with build104
Comment 29 Mike Cao 2015-06-02 05:19:28 UTC 
(In reply to Vadim Rozenfeld from comment #27)
> Please re-check again with the latest build
> http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> 104/win/virtio-win-prewhql-0.1.zip
> 
> Thanks,
> Vadim.

Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the internet 
Can you suggest some other similiar tools instead?

Mike
Comment 30 Vadim Rozenfeld 2015-06-02 06:03:06 UTC 
(In reply to Mike Cao from comment #29)
> (In reply to Vadim Rozenfeld from comment #27)
> > Please re-check again with the latest build
> > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> > 104/win/virtio-win-prewhql-0.1.zip
> > 
> > Thanks,
> > Vadim.
> 
> Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the
> internet 
> Can you suggest some other similiar tools instead?

Hi Mike,
No idea. But if needed, I can create a simple app which will do out-of-bound IOs.

Best regards,
Vadim.

> 
> Mike
Comment 31 Mike Cao 2015-06-02 06:16:23 UTC 
(In reply to Vadim Rozenfeld from comment #30)
> (In reply to Mike Cao from comment #29)
> > (In reply to Vadim Rozenfeld from comment #27)
> > > Please re-check again with the latest build
> > > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> > > 104/win/virtio-win-prewhql-0.1.zip
> > > 
> > > Thanks,
> > > Vadim.
> > 
> > Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the
> > internet 
> > Can you suggest some other similiar tools instead?
> 
> Hi Mike,
> No idea. But if needed, I can create a simple app which will do out-of-bound
> IOs.
> 
> Best regards,
> Vadim.

Hi, Vadim

Pls help do it.

Thanks,
Mike
> 
> > 
> > Mike
Comment 34 lijin 2015-12-23 07:52:55 UTC 
As rhel6.8 will ship the same viostor version with rhel7.2 and the same bug1195487 on rhel7.2 has been verified,change status to verified.
Comment 36 errata-xmlrpc 2016-05-10 16:35:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1011.html
Comment 37 Ladi Prosek 2016-10-26 16:22:15 UTC 
*** Bug 1388553 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.