Bug 608548 - QEMU doesn't respect hardware sector size of underlying block device when doing O_DIRECT
QEMU doesn't respect hardware sector size of underlying block device when doi...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.0
All Linux
medium Severity high
: rc
: 6.1
Assigned To: chellwig@redhat.com
Virtualization Bugs
: Reopened
: 616509 (view as bug list)
Depends On:
Blocks: 580566 580954 616480
  Show dependency treegraph
 
Reported: 2010-06-28 00:13 EDT by Johnny Liu
Modified: 2013-01-09 17:47 EST (History)
12 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.124.el6
Doc Type: Bug Fix
Doc Text:
Cause: qemu did not align memory properly for O_DIRECT support Consequence: I/O to device with large sector sizes like CDROMs dit not work in cache=none mode Fix: qemu was changed to use properly aligned memory for I/O requests Result: I/O to devices with large sector sizes like CDROMs work in cache=none mode
Story Points: ---
Clone Of:
: 616480 (view as bug list)
Environment:
Last Closed: 2011-05-19 07:28:32 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Screenshot - installation failed form cdrom device. (17.65 KB, image/png)
2010-06-28 00:13 EDT, Johnny Liu
no flags Details

  None (edit)
Description Johnny Liu 2010-06-28 00:13:47 EDT
Created attachment 427301 [details]
Screenshot - installation failed form cdrom device.

Description of problem:
When trying to install guest form cdrom device, installation failed.
The following error message is seen in virt-viewer:
Boot failed: Could not read from CDROM (code 0003)
No bootable device
The screen-shot will be attached.

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.82.el6.x86_64
libvirt-0.8.1-10.el6.x86_64
seabios-0.5.1-0.15.20100108git669c991.el6.x86_64
# uname -a
Linux dhcp-66-70-179.nay.redhat.com 2.6.32-37.el6.x86_64 #1 SMP Sun Jun 20 19:29:35 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux



How reproducible:
Always

Steps to Reproduce:
1. Run virt-install to install guest from cdrom device.
# virt-install -n cdrom_installation -r 1024 --disk path=/var/lib/libvirt/images/cdrom_installation.img,size=5 -c /dev/cdrom
2. Check the message in pop-up virt-viewer windows.

  
Actual results:
Installation failed.
The following error message is seen in virt-viewer:
Boot failed: Could not read from CDROM (code 0003)
No bootable device

NOTE:
I can make sure my DVD is a good media, because I can install host OS from this DVD media successfully.

Expected results:
Installation from cdrom device should be finished successfully.

Additional info:
If install the guest from ISO file, the installation works fine.
Comment 1 Johnny Liu 2010-06-28 00:25:45 EDT
# virsh dumpxml cdrom_installation
<domain type='kvm' id='4'>
  <name>cdrom_installation</name>
  <uuid>093b482e-a02e-f5cf-563d-95d226d422b8</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='cdrom'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/cdrom_installation.img'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' cache='none'/>
      <source dev='/dev/cdrom'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:be:fb:69'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
  <seclabel type='dynamic' model='selinux'>
    <label>system_u:system_r:svirt_t:s0:c487,c559</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c487,c559</imagelabel>
  </seclabel>
</domain>


Log in /var/log/libvirt/qemu/cdrom_installation.log:
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name cdrom_installation -uuid 093b482e-a02e-f5cf-563d-95d226d422b8 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/cdrom_installation.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -no-reboot -boot d -drive file=/var/lib/libvirt/images/cdrom_installation.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/dev/cdrom,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,cache=none -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:be:fb:69,bus=pci.0,addr=0x4 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
char device redirected to /dev/pts/1
Comment 3 RHEL Product and Program Management 2010-06-28 00:42:56 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 Hugh Brock 2010-06-28 12:24:35 EDT
Don't see how this could be a libvirt BZ, reassigning to QEMU
Comment 5 Johnny Liu 2010-06-29 05:31:21 EDT
Today I install a fresh HOST os from rel-eng build (RHEL6.0-20100622.1), this issue dose NOT occur. So it look like a environment issue, the nightly build is not stable. So I close this bug.
Comment 6 Johnny Liu 2010-06-30 04:55:09 EDT
The problem is reproduced again after updated some packages, such as: libvirt, qemu-kvm, python-virtinst, selinux-policy, virt-manager, an so on.

I yum these package from brewweb repo.
# cat /etc/yum.repos.d/brewroot.repo 
[brewroot]
name = brewroot_repo
baseurl = http://porkchop.devel.redhat.com/brewroot/repos/RHEL-6-build/latest/$basearch
enabled = 1
gpgcheck = 0



Not only one tester encountered this issue at our side, but now we still did not find the root cause, so I reopen this bug for tracking this issue.
Comment 7 dyuan 2010-06-30 23:31:46 EDT
reproduce the bug with 

<disk type='block' device='cdrom'>
  <driver cache='none'/> 

in domain xml, when set it to writethrough or writeback or default, boot successfully via cdrom.

It's the new feature of virt-manager-0.8.4-4.el6.

no-cache should be the default (refer to the comment in bug 594080),
but I'm not sure why it became 'default' occasionally, that make us very faint to reproduce it before.
Comment 8 dyuan 2010-06-30 23:33:21 EDT
reproduce the bug with 

<disk type='block' device='cdrom'>
  <driver cache='none'/> 

in domain xml, when set it to writethrough or writeback or default, boot successfully via cdrom.

kyla.

It's the new feature of virt-manager-0.8.4-4.el6.

no-cache should be the default (refer to the comment in bug 594080),
but I'm not sure why it became 'default' occasionally, that make us very faint to reproduce it before.

dyuan.
Comment 9 Dor Laor 2010-07-06 10:36:52 EDT

*** This bug has been marked as a duplicate of bug 558256 ***
Comment 10 Johnny Liu 2010-07-06 21:48:41 EDT
I do not think this bug is a a duplicate of bug 558256.

In this bug, the guest even can NOT
Comment 11 Johnny Liu 2010-07-06 21:55:56 EDT
I do not think this bug is a a duplicate of bug 558256.

Obviously in BZ#558256, guest can find the bootable media, and already start the installation process. 
But in this bug, the guest even can NOT find a bootable media, the result is worse than 558256.

Maybe the summary is not clear, so I changed it to be more clear.
Comment 12 RHEL Product and Program Management 2010-07-15 10:08:41 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 13 Markus Armbruster 2010-07-15 12:12:18 EDT
"Boot failed: Could not read from CDROM (code 0003)" is from seabios/boot.c.
It's almost certainly not a duplicate of bug 558256.
Comment 14 Jes Sorensen 2010-07-20 08:08:18 EDT
I can reproduce it here when trying to use virt-manager to install from a
real DVD on RHEL6 as of July 19. If I go into virt-manager and change the
cache mode for the DVD drive to writeback it works fine.

I'll look at whats wrong with qemu for this, but I believe libvirt shouldn't
set DVD drives as cache=none in the first place either.

Jes
Comment 15 Jes Sorensen 2010-07-20 10:20:57 EDT
Ok so the problem is as follows: cache=none means the device, in this case
the CDROM/DVD is opened with O_DIRECT. With O_DIRECT all I/O needs to be
sector size aligned, and all request must be a multiple of the sector size.
This works fine for files on a file system and raw disk partitions where the
sector size is 512 bytes, but for a CDROM/DVD drive the sector size is normally
2048 bytes.

Checking whether we are on a CDROM/DVD device, and checking the block size
is pretty complicated, plus the 512 byte sector size is hard coded all over
the place in QEMU, so it is not really feasible to try and fix this in QEMU
itself. The only reasonable solution for RHEL6 is to make sure that libvirt
doesn't set cache=none on CDROM/DVD drives.

Jes
Comment 21 Dor Laor 2010-07-21 05:47:56 EDT
Christoph tells that we should fix the alignment to 4k for supporting 4k block io physical disks
Comment 22 Jes Sorensen 2010-07-23 03:58:35 EDT
Moving to 6.1 as 616480 is a workaround for the CDROM problem.

Change bug description to match actual problem that QEMU expects all
devices to have 512 byte sectors with O_DIRECT, which doesn't for devices
with 2K or 4K sectors.
Comment 23 Jes Sorensen 2010-07-23 04:00:49 EDT
Reassign to Christoph as he said he was looking into solving the problem
the right way.
Comment 24 chellwig@redhat.com 2010-09-20 15:25:55 EDT
I've submitted a patch to allow O_DIRECT I/O for non-512byte block sizes, including automatically enabling it for CDROMs. I'll backport it as soon as it's been accepted upstream.
Comment 27 chellwig@redhat.com 2010-12-01 09:13:21 EST
*** Bug 616509 has been marked as a duplicate of this bug. ***
Comment 31 Shirley Zhou 2010-12-23 05:32:43 EST
Reproduce this bug with 113 with cdrom option:
-drive file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,cache=none 
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 

Boot failed with error code 0003.

Verify this bug with qemu-kvm-0.12.1.2-2.128.el6.x86_64.rpm as same cdrom option, guest can booting from CD-Rom ok.

So this bug has been fixed.
Comment 33 Miya Chen 2011-03-09 00:01:00 EST
move to verified based on comment#31
Comment 34 chellwig@redhat.com 2011-05-05 10:41:31 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: qemu did not align memory properly for O_DIRECT support

Fix: qemu was changed to use properly aligned memory for I/O requests

Consequence: I/O to device with large sector sizes like CDROMs dit not work in cache=none mode
Result: I/O to devices with large sector sizes like CDROMs work in cache=none mode
Comment 35 chellwig@redhat.com 2011-05-05 10:42:14 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,6 +1,7 @@
 Cause: qemu did not align memory properly for O_DIRECT support
 
+Consequence: I/O to device with large sector sizes like CDROMs dit not work in cache=none mode
+
 Fix: qemu was changed to use properly aligned memory for I/O requests
 
-Consequence: I/O to device with large sector sizes like CDROMs dit not work in cache=none mode
 Result: I/O to devices with large sector sizes like CDROMs work in cache=none mode
Comment 36 errata-xmlrpc 2011-05-19 07:28:32 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Comment 37 errata-xmlrpc 2011-05-19 08:46:07 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html

Note You need to log in before you can comment on or make changes to this bug.