Description of problem: By default, vdsm starts VMs with libvirt's error_policy='stop', which translates to werror=stop,rerror=stop options to qemu, causing all block IO errors to cause the VM to pause. However, this can be changed by starting the vm with propagateErrors=On, or by a hook which alters the libvirt XML or qemu command line. With propagateErrors=On, libvirt's error_policy is set to enospace, which translates to "werror=enospc" to qemu. In this mode only out-of-space errors cause a VM pause, other IO errors are reported immediately to the guest via the virtual hardware. However, in this mode, *any* IO error, still results in vdsm marking the VM as stopped (self._guestCpuRunning = False in the Vm object, even though the guest is still running. Version-Release number of selected component (if applicable): vdsm-4.10.2-24.1.el6ev.x86_64 (But the behaviour appears to be the same in current upstream vdsm git) How reproducible: 100% Steps to Reproduce: 1. Start a VM with propagateErrors=On (under RHEV this can be done by using psql to set the propagate_errors field to "On" in the base_disks table) 2. Trigger an IO error, e.g. by having a guest-side driver access beyond the end of a virtio-blk device. Actual results: Guest keeps running, but is reported as Paused to RHEV. Expected results: Guest keeps running, and displays as such in RHEV. Additional info: The problem appears to be at line 550-552 in vdsm/clientIF.py, where we handle IO error events from libvirt: elif eventid == libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON: srcPath, devAlias, action, reason = args[:-1] v._onAbnormalStop(devAlias, reason) This doesn't check the action code before calling _onAbnormalStop() which marks the guest as stopped. The guest is stopped only if the action code is "stop", but not if it is "report".
David, can you please point out (or better yet, attach) the vdsm and engine relevant logs? There's a huge amount of data in the tracker.
This isn't about deciding what in the case of I/O errors - that job can only be done inside qemu. This is about keeping vdsm's notion of whether the guest is running in sync with reality. You can determine whether qemu *did* pause the guest by checking the "action" parameter in the IO error message, no deeper understanding of which IO errors will trigger that is required. I'm not clear on how the vdsm logs, let alone the engine logs are useful here.
Created attachment 862997 [details] Draft patch to correctly determine if a reported IO error stopped the guest Attached draft patch. I have a customer trying this in a test package at the moment.
Actually, arguably vdsm should never mark the guest as paused directly on IO errors. In cases where the guest does pause, qemu/libvirt will send an explicit lifecycle message noting that the guest is paused immediately afterwards, which vdsm will already process. It's possible it's done this way to more easily link the IO error reason to the pause though.
David, We need to confirm our understanding of the bug with the above logs. Not having them available might postpone bug resolution. Please attach or direct us to them. We do not consider this an easyfix at this time, removing again. Note that you can push draft patches to - gerrit.us:vdsm HEAD:refs/drafs/master .
Please take a look at this RFE: https://bugzilla.redhat.com/show_bug.cgi?id=1024428#c3 VDSM does not support the report option at this moment.
This is *not* a duplicate of 1024428. That one is about being able to set the error_policy conveniently. This one is about vdsm correctly handling errors reported from qemu. Note that regardless of the rhev/vdsm state, qemu could be put into an error mode which doesn't always pause on errors by using a vdsm hook which adjusts the qemu command line.
it is handling the supported error_policy which is enospace and stop. If you're setting something else via hook. So you either get it as part of bug 1024428 or closed notabug
But currently it *doesn't* handle enospace - that's what I'm trying to use. With error_policy=enospace qemu-kvm will correctly continue operating on a non-ENOSPC disk error, but vdsm will still report the guest as paused when it sees *any* IO error.
Francesco, Mostly looking good. That else clause looks as though it could give a really misleading error if qemu/libvirt ever added an action type other than the current stop/report/ignore. Better to just print "unexpected error action: <value>" or something, maybe?
Hi David, Thanks for your comments; we are reworking the logging for unexpected action types. Feel free to post comments on gerrit, they are very welcome.
Verified in vdsm-4.14.7-0.2.rc.el6ev.x86_64 (av8). Verification steps: I used two local storages on the host. One "regular" under `/mnt/localstorage/`. And a second "flakey" one under `/mnt/errstorage/`, simulating I/O errors using `dmseup` utility. (For details how to create a flakey device with dmsetup see bug 1090079 comment 3.) In RHEVM GUI, add both local storages (/mnt/localstorage as master SD). Create new VM with two disks - one "healthy" disk on the `localstorage` domain and a second "flakey" disk (1G) on the `errorstorage` domain. In RHEVM DB, update both disks to propagate errors to guest: psql: UPDATE base_disks SET propagate_errors = 'On'; Restart ovirt-engine service. In RHEVM GUI, install the guest OS *on the healthy disk* (I used Fedora 19). In the guest, mount the second flakey disk to `/mnt/errdisk/` and run some I/O operation on it. I used `dd`: # dd if=/dev/zero of=/mnt/errdisk/test bs=1000 mount=1M and after few seconds I got a splash of I/O errors "Buffer I/O error on device vdb, logical block ...". Results: The qemu process runs with correct parameter 'werror=enospc'. After the I/O errors, the guest is still running. Both, QEMU/VDSM and RHEVM, are also reporting the guest as running.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0504.html