Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1064643

Summary: qemu makes no distinction between external and guest triggered block I/O errors
Product: Red Hat Enterprise Linux 6 Reporter: David Gibson <dgibson>
Component: qemu-kvmAssignee: Markus Armbruster <armbru>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: adevolder, agkesos, armbru, bsarathy, chayang, juli, juzhang, mkalinin, mkenneth, pmatouse, qzhang, rbalakri, sauchter, sluo, uobergfe, virt-bugs, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.432.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 06:55:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1085232    

Description David Gibson 2014-02-13 02:42:21 UTC
Description of problem:

When in werror=stop,rerror=stop mode (as set by libvirt's error_mode=stop) qemu will pause the VM on any IO error.

This makes a lot of sense for out of space errors from the qcow2 layer, since it allows libvirt or other management software to expand as necessary.

This can often make sense for "external" disk errors such as an I/O error propagated from the host side OS or hardware.  In these cases it can be useful to examine and correct the external error before resuming the guest to avoid data loss.

This behaviour almost never makes sense, however, for I/O errors triggered entirely by the guest, such as the guest attempting to access beyond the (logical) end of a block device.  In this case there's really nothing the host can do except to provide an error notification to the guest.

But at present, qemu provides no way to distinguish between externally triggered errors, and errors cause by bad guest actions.  There are no available error modes which treat these two types of errors differently.

Version-Release number of selected component (if applicable):

See in a customer case with 
qemu-kvm-rhev-0.12.1.2-2.355.el6_5.7.x86_64, but AFAICT the behaviour is unchanged in current upstream qemu git.

Comment 3 Markus Armbruster 2014-04-11 11:30:27 UTC
*** Bug 1080996 has been marked as a duplicate of this bug. ***

Comment 4 Markus Armbruster 2014-04-11 11:32:12 UTC
Note: bug 1080996 has some very detailed debugging results.

Comment 6 Markus Armbruster 2014-06-06 09:07:21 UTC
We need to test at least ide-hd, scsi-hd and virtio-blk.

The RHEL-7 buddy bug 1085232 has a nice reproducer for ide-hd in
comment#2.  I just asked for it to be extended to cover scsi-hd and
virtio-blk.

Can you reproduce the bug on RHEL-6 with it, with all three kinds of
disks?

Comment 7 Markus Armbruster 2014-06-06 09:08:27 UTC
Correction: ide-drive with media=disk, because RHEL-6 doesn't have ide-hd.

Comment 8 juzhang 2014-06-06 09:24:28 UTC
(In reply to Markus Armbruster from comment #6)
> We need to test at least ide-hd, scsi-hd and virtio-blk.
> 
> The RHEL-7 buddy bug 1085232 has a nice reproducer for ide-hd in
> comment#2.  I just asked for it to be extended to cover scsi-hd and
> virtio-blk.
> 
> Can you reproduce the bug on RHEL-6 with it, with all three kinds of
> disks?

Hi Qzhang,

Could you give a test and update the testing result in the bz?

Best Regards,
Junyi

Comment 9 Qunfang Zhang 2014-06-06 09:27:31 UTC
(In reply to Markus Armbruster from comment #6)
> We need to test at least ide-hd, scsi-hd and virtio-blk.
> 
> The RHEL-7 buddy bug 1085232 has a nice reproducer for ide-hd in
> comment#2.  I just asked for it to be extended to cover scsi-hd and
> virtio-blk.
> 
> Can you reproduce the bug on RHEL-6 with it, with all three kinds of
> disks?

We will test the bug and update the result here.

Comment 10 Jun Li 2014-06-09 10:22:21 UTC
IDE:
---
Steps as followings:
1, boot RHEL-Server-5.8-64 guest with cli:
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/home/juli/rhel-6.6-20140603.n.0.raw,if=none,id=img,rerror=stop,werror=stop -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/data-disk.qcow2,if=none,id=ide-img,rerror=stop,werror=stop,media=disk -device ide-drive,drive=ide-img,bus=ide.0,unit=1,id=sys-ide-img -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11
-------
2, compile vm_stop.c and run it inside guest.
# cat vm_stop.c 
/*
 *  * program 'vm_stop.c' - compile with 'make vm_stop'
 *   */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <linux/hdreg.h>

char args[7];

main()
{
    int fd;

    if ((fd = open("/dev/sda", O_RDONLY)) == -1) {
        perror("open");
        _exit(1);
    }

    /*
 *      * lba is 0xfffffff (268435455)
 *           * To reproduce the problem, the size of the virtual disk image
 *                * must be less than 268435455 * 512 = 137438953472 bytes (128G).
 *                     */
    args[6] = 0x4f;     // select : lba bits 24 .. 27
    args[5] = 0xff;     // hcyl   : lba bits 16 .. 23
    args[4] = 0xff;     // lcyl   : lba bits  8 .. 15
    args[3] = 0xff;     // sector : lba bits  0 ..  7
    args[2] = 0x01;     // number of sectors
    args[1] = 0x00;
    args[0] = WIN_READ;

    if (ioctl(fd, HDIO_DRIVE_TASK, args) == -1) {
        perror("ioctl");
        _exit(1);
    }

    if (close(fd) == -1) {
        perror("close");
        _exit(1);
    }
}
---
# make vm_stop
# ./vm_stop
-----------
Results:
After step2, guest will hang.
(qemu) info status 
VM status: paused (io-error)
============================================
scsi-hd:
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/home/juli/rhel-6.6-20140603.n.0.raw,if=none,id=img,rerror=stop,werror=stop -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/data-disk.qcow2,if=none,id=ide-img,rerror=stop,werror=stop,media=disk -device virtio-scsi-pci,id=scsi0 -device scsi-hd,drive=ide-img,bus=scsi0.0,id=sys-ide-img -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11
---
# ./vm_stop 
ioctl: Invalid argument
=====================================
virtio-blk:
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/home/juli/rhel-6.6-20140603.n.0.raw,if=none,id=img,rerror=stop,werror=stop -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/data-disk.qcow2,if=none,id=ide-img,rerror=stop,werror=stop,media=disk -device virtio-blk-pci,drive=ide-img,bus=pci.0,id=sys-ide-img -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11
---
# ./vm_stop 
ioctl: Inappropriate ioctl for device
==========================================

As above show, can not use vm_stop.c for scsi-hd and virtio-blk. In my opinion, we should modify ioctl() for scsi-hd and virtio-blk in file vm_stop.c.

Hi Ulrich,

  Could you help to modify vm_stop.c for scsi-hd and virtio-blk. Thx.


Best Regards,
Jun Li

Comment 23 Jun Li 2014-06-26 07:34:59 UTC
Add components version for comment #22:
qemu-kvm version:
qemu-kvm-0.12.1.2-2.428.el6.x86_64
Host kernel version:
2.6.32-486.el6.x86_64
Guest kernel version:
2.6.32-483.el6.x86_64

Comment 29 Jeff Nelson 2014-07-29 15:56:58 UTC
Fix included in qemu-kvm-0.12.1.2-2.432.el6

Comment 31 Jun Li 2014-08-06 02:45:57 UTC
Verify this bz need to verify IDE/virtio-scsi/virtio-blk interface.
===========
IDE:
---
1, boot guest using following cli.
# qemu-img create -f qcow2 /home/data-disk.qcow2 10G
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/mnt/linux_img/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,rerror=stop,werror=stop,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/data-disk.qcow2,if=none,id=ide-img,rerror=stop,werror=stop,media=disk -device ide-drive,drive=ide-img,bus=ide.0,unit=1,id=sys-ide-img -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11

2, compile vm_stop.c and run it inside guest.
# cat vm_stop.c 
/*
 *  * program 'vm_stop.c' - compile with 'make vm_stop'
 *   */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <linux/hdreg.h>

char args[7];

main()
{
    int fd;

    if ((fd = open("/dev/sda", O_RDONLY)) == -1) {
        perror("open");
        _exit(1);
    }

    /*
 *      * lba is 0xfffffff (268435455)
 *           * To reproduce the problem, the size of the virtual disk image
 *                * must be less than 268435455 * 512 = 137438953472 bytes (128G).
 *                     */
    args[6] = 0x4f;     // select : lba bits 24 .. 27
    args[5] = 0xff;     // hcyl   : lba bits 16 .. 23
    args[4] = 0xff;     // lcyl   : lba bits  8 .. 15
    args[3] = 0xff;     // sector : lba bits  0 ..  7
    args[2] = 0x01;     // number of sectors
    args[1] = 0x00;
    args[0] = WIN_READ;

    if (ioctl(fd, HDIO_DRIVE_TASK, args) == -1) {
        perror("ioctl");
        _exit(1);
    }

    if (close(fd) == -1) {
        perror("close");
        _exit(1);
    }
}
---
# make vm_stop
# ./vm_stop
-----------
Results:
After step2, guest and qemu-kvm works well, no error in guest dmesg, guest cli will give as following info:
# ./vm_stop 
ioctl: Input/output error
(qemu) info status 
VM status: running
==================
virtio scsi:
---
1, boot guest with following cli:
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/mnt/linux_img/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,rerror=stop,werror=stop,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/data-disk.qcow2,if=none,id=ide-img,rerror=stop,werror=stop,media=disk -device virtio-scsi-pci,id=scsi0 -device scsi-hd,drive=ide-img,bus=scsi0.0,id=sys-ide-img -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11
---
2, Build the modified 'sg_simple16' program, ref https://bugzilla.redhat.com/attachment.cgi?id=907758.
----
3, Run the modified 'sg_simple16' program in the guest.
# dmesg |grep sda
sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
...
# ./sg_simple16 /dev/sg1 20971520
READ_16 command error: SCSI status: Check Condition 
 Fixed format, current;  Sense key: Illegal Request
 Additional sense: Invalid field in cdb
 Raw sense data (in hex):
        70 00 05 00 00 00 00 0a  00 00 00 00 24 00 00 00    
        00 00  
---
Results:
After step3, guest and qemu-kvm works well, no error in guest dmesg.
(qemu) info status 
VM status: running
=============================
virtio block:
Cli:
# /usr/libexec/qemu-kvm -m 2G -enable-kvm -drive file=/mnt/linux_img/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,rerror=stop,werror=stop,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=/home/juli/rhel6-data.img,if=none,id=drive-virtio-disk1,format=raw,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -monitor stdio -vnc :1 -boot menu=on -netdev tap,id=tap0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=tap0,id=sys-tap0,mac=24:be:05:14:11:11
----
Create a virtual disk with 130M.
# qemu-img create -f raw /home/juli/rhel6-data.img 130M
---
Steps as attachment 912144 [details] show. 
---
Run following cli inside guest(As 270336/2/1024=132M):
# dd iflag=direct if=/dev/vdb of=/dev/null bs=512 count=1 skip=270336
---
Then, guest works well, no error in guest dmesg. When execute "dd" inside guest, will get following info:
# dd iflag=direct if=/dev/vdb of=/dev/null bs=512 count=1 skip=270336
dd: reading `/dev/vdb': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0017959 s, 0.0 kB/s
---
(qemu) info status 
VM status: running
=========================
Based on above testing, all works well for ide-hd, scsi-hd and virtio-blk interface disks.

Version of qemu:
qemu-kvm-0.12.1.2-2.435.el6.x86_64

Comment 32 Qunfang Zhang 2014-08-11 02:40:56 UTC
Setting to VERIFIED according to comment 31.

Comment 33 Markus Armbruster 2014-08-15 17:40:57 UTC
Upstream needed the followup-fix "ide: only constrain read/write requests to drive size, not other types".
https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg02005.html

Background: http://bugs.debian.org/757927

RHEL-6 doesn't need it, because we can DMA only read and write: we don't have commit 4e1e005.

Comment 34 errata-xmlrpc 2014-10-14 06:55:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html