Description of problem:
NOTE: virtio-win component is just a guess. The problem is definitely somewhere inside the Windows guest, and virtio-win is the only plausibly responsible component that is RH provided, so we need to either find the bug there, or rule it out.
Several Windows Server 2008R2 guests under RHEV were periodically pausing due to an IO error. Adding debugging to qemu showed that these EIO errors were because the guest itself was attempting virtio accesses beyond the logical end of device.
Version-Release number of selected component (if applicable):
virtio-win-1.4.0.iso (I think)
Customer has several systems which trigger this problem daily.
Steps to Reproduce:
Happens daily on customer system. Usually about the same time each day, so it's probably triggered by a scheduled job, but the exact sequence of guest-side events which trigger this problem is unknown so far.
Guest issues out-of-bounds accesses to the virtio virtual hardware, causing qemu to pause the guest.
virtio-win drivers reject out of bounds access requests and report them on the guest side.
The accesses have a strange pattern. They appear to come in a burst, the first is the sector immediately beyond the logical end of device, then there are several more each 2-4x the distance beyond the end-of-device as the last one.
qemu diagnostics with more details coming.
Created attachment 865302 [details]
Patch showing diagnostic alterations to qemu
This is the debug patch I used to generate the diagnostics for out of bounds accesses. The code can also be found in the 'private-dgibson-sfdc01018528' branch of the qemu-kvm-rhev rhpkg tree.
Can you try to reproduce it with virito-win-1.6.8-4 ?
Do you know which app is running when this issue occurs ?
This bug may dup of https://bugzilla.redhat.com/show_bug.cgi?id=1080996
This is closely related to bug 1080996, but it's not a dupe per se.
Bug 1080996 (itself a dupe of 1064643) is covering the fact that qemu and the stack above don't deal well with errors like this which are unambiguously the guest's fault. The proposed fix is to differently classify guest parameter caused errors, and (usually) always report them rather than pausing the VM.
This bug is addressing the fact that the guest is initiating the bad accesses in the first place. The proposal is to either fix the virtio-win drivers so it doesn't make these accesses, or to determine that the problem lies within something else on the guest side.
Can we try reproducing this problem on a fresh system with the following apps
Sophos Remote Management System
mdeng ,pls handle the needinfo per comment #11.
Created attachment 948817 [details]
Created attachment 948818 [details]
Created attachment 948819 [details]
Created attachment 948820 [details]
(In reply to David Gibson from comment #10)
> This bug is addressing the fact that the guest is initiating the bad
> accesses in the first place. The proposal is to either fix the virtio-win
> drivers so it doesn't make these accesses, or to determine that the problem
> lies within something else on the guest side.
It looks like some application(s) - presumably Sophos Anti-Virus performs raw reads/writes, bypassing file system driver.
I will add an extra sanity check to validate IO boundaries.
Should be fixed in build 101, available at http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/101/win/virtio-win-prewhql-0.1.zip
cannot reproduce this issue from QE side,guest works fine.
1.boot win2k8R2 guest with virtio-blk device:
2.install software “Sophos Endpoint Security and Control 10.3” which include "Sophos Anti-Virus" and "Sophos AutoUpdate" on guest;
3.configure schedule of scan computers and autoupdate of sophos
4.make the guest running two days
guest works fine after Sophos Anti-Virus scan the computer;
And I got following error message after Sophos AutoUpdate try to update,I guess it's due to I have no product license:
Message: ERROR: Download of Sophos Endpoint Security and Control failed from server sophos
Message: ERROR: Could not find a source for updated packages
Please re-check again with the latest build http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/104/win/virtio-win-prewhql-0.1.zip
Mike,please verify the bug with build104
(In reply to Vadim Rozenfeld from comment #27)
> Please re-check again with the latest build
Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the internet
Can you suggest some other similiar tools instead?
(In reply to Mike Cao from comment #29)
> (In reply to Vadim Rozenfeld from comment #27)
> > Please re-check again with the latest build
> > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> > 104/win/virtio-win-prewhql-0.1.zip
> > Thanks,
> > Vadim.
> Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the
> Can you suggest some other similiar tools instead?
No idea. But if needed, I can create a simple app which will do out-of-bound IOs.
(In reply to Vadim Rozenfeld from comment #30)
> (In reply to Mike Cao from comment #29)
> > (In reply to Vadim Rozenfeld from comment #27)
> > > Please re-check again with the latest build
> > > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> > > 104/win/virtio-win-prewhql-0.1.zip
> > >
> > > Thanks,
> > > Vadim.
> > Vadim ,I can not find Sophos Endpoint Security and Control 10.3 on the
> > internet
> > Can you suggest some other similiar tools instead?
> Hi Mike,
> No idea. But if needed, I can create a simple app which will do out-of-bound
> Best regards,
Pls help do it.
> > Mike
As rhel6.8 will ship the same viostor version with rhel7.2 and the same bug1195487 on rhel7.2 has been verified,change status to verified.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
*** Bug 1388553 has been marked as a duplicate of this bug. ***