Bug 997839 - GOS installation fails on a virtio-blk device(img file) created on a XFS filesystem with 4k sector size.
GOS installation fails on a virtio-blk device(img file) created on a XFS file...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Kevin Wolf
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-16 05:34 EDT by Inbaraj
Modified: 2013-09-18 05:48 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-18 05:48:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
VM Anaconda installer logs (24.73 KB, application/x-gzip)
2013-08-19 07:34 EDT, Inbaraj
no flags Details
host messages,qemu logs,libvirt logs,failure screen shot (102.32 KB, application/x-gzip)
2013-08-19 07:48 EDT, Inbaraj
no flags Details
screenshot for guest installration error. (32.54 KB, image/png)
2013-08-23 04:33 EDT, Sibiao Luo
no flags Details

  None (edit)
Description Inbaraj 2013-08-16 05:34:26 EDT
Description of problem:

While trying to install an Guest operating system (linux)  on a virtio blk device (disk image), installation failed with I/O error. The disk image is created from an DMMP device formatted with XFS file system.

Version-Release number of selected component:
OS-Hypervisor: RHEL 6.4 KVM
XFS version: xfsprogs-3.1.1-10.el6.x86_64
qemu-kvm-0.12.1.2-2.355.el6.x86_64
kernel: 2.6.32-358.6.1.el6.x86_64              

How reproducible:
Every time.

steps to reproduce the issue :
1) map a lun to  the hypervisor.
2) Discover the device and configure multipath.
2) create a XFS file system on the DM-MP device using mkfs.xfs command.
4) Mount the device  and create an image file for 30g on this mount point.
5) Using virt-manager create a new VM and assign the image file as the virtio blk device for installation. 

Expected result:

Partition creation should pass and guest installation should complete successfully.

Actual result:

Installation fails during partition creation with an i/o error.


Additional info:

Default XFS file system aligns the sector size to the physical block size of the device. So the above issue is seen , when a img file is created out of a XFS formatted device with sector size of 4096 .
Whereas the issue is not seen after tweaking the sector size  to 512 .

Issue seen with the default XFS file system values:
[root@ibmx3550-210-108 ~]# xfs_info /mnt/test2
meta-data=/dev/mapper/test_xfs-dmmp_lvm_img_part isize=256    agcount=8, agsize=163839 blks
         =                       sectsz=4096  attr=2, projid32bit=0
data     =                       bsize=4096   blocks=1310712, imaxpct=25
         =                       sunit=1      swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Issue is not seen on XFS file system with sector size 512..
[root@ibmx3550-210-108 ~]# xfs_info /mnt/test1
meta-data=/dev/mapper/360a98000324669384a2443744a78474f isize=256    agcount=16, agsize=491519 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=7864304, imaxpct=25
         =                       sunit=1      swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=3840, version=2
         =                       sectsz=512   sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Comment 1 Inbaraj 2013-08-19 07:34:49 EDT
Created attachment 788013 [details]
VM Anaconda installer logs
Comment 2 Inbaraj 2013-08-19 07:48:16 EDT
Created attachment 788018 [details]
host messages,qemu logs,libvirt logs,failure screen shot
Comment 4 Sibiao Luo 2013-08-23 04:32:18 EDT
Reproduce this issue on latest qemu-kvm and kernel version with sectsz=4096, btw, sectsz=512 did not met this issue.

Host info:
2.6.32-412.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.397.el6.x86_64
xfsprogs-3.1.1-10.el6_4.1.x86_64
Guest info:
rhel6.4GA x84_64bit

How reproducible:
100%

steps to reproduce the issue:
1) map a lun to  the hypervisor.
# ls -l /dev/disk/by-path/ip-10.66.90.100\:3260-iscsi-iqn.2001-05.com.equallogic\:0-8a0906-0971f7d03-1dff49b26885073d-s2-sluo-172259-lun-0 
lrwxrwxrwx. 1 root root 9 Aug 22 03:27 /dev/disk/by-path/ip-10.66.90.100:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-0971f7d03-1dff49b26885073d-s2-sluo-172259-lun-0 -> ../../sdb
2) Discover the device and configure multipath.
# yum install device-mapper.x86_64 device-mapper-multipath.x86_64
# modprobe dm-multipath
# modprobe dm-round-robin
# lsmod | grep multipath
dm_multipath           17711  2 dm_round_robin
dm_mod                 82743  14 dm_multipath,dm_mirror,dm_log
 
# service multipathd start
Starting multipathd daemon:                                [  OK  ]
# chkconfig multipathd on  //ensure sure that the multipath daemon starts on bootup
# /sbin/mpathconf --enable
# ls -l /etc/multipath.conf
-rw-------. 1 root root 2754 Aug 23 01:34 /etc/multipath.conf
# cat /etc/multipath.conf
blacklist {
	wwid 1ATA_ST3500413AS_S2A29GR7
}
defaults {
user_friendly_names no
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
}
multipaths {
    multipath {
    wwid 36090a038d0f771093d078568b249ff1d //wwid of /dev/sdb
    alias multipath-disk
    path_grouping_policy multibus
    path_selector "round-robin 0"
    }
}
# service multipathd reload
Reloading multipathd:                                      [  OK  ]
# fdisk -l
...
Disk /dev/sdb: 107.4 GB, 107379425280 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/multipath-disk: 107.4 GB, 107379425280 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
2) create a XFS file system with sectsz=4096 on the DM-MP device using mkfs.xfs command.
# mkfs.xfs -s size=4096 /dev/mapper/multipath-disk -f
meta-data=/dev/mapper/multipath-disk isize=256    agcount=4, agsize=6553920 blks
         =                       sectsz=4096  attr=2, projid32bit=0
data     =                       bsize=4096   blocks=26215680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=12800, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
4) Mount the device  and create an image file for 30g on this mount point.
# mount /dev/mapper/multipath-disk /mnt/
# mount
...
/dev/mapper/multipath-disk on /mnt type xfs (rw)
# qemu-img create -f raw /mnt/bug997839testing.raw 30G
Formatting '/mnt/bug997839testing.raw', fmt=raw size=32212254720 
# xfs_info /mnt/bug997839testing.raw
meta-data=/dev/mapper/multipath-disk isize=256    agcount=4, agsize=6553920 blks
         =                       sectsz=4096  attr=2, projid32bit=0
data     =                       bsize=4096   blocks=26215680, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=12800, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
5) Using qemu-kvm create a new VM and assign the image file as the virtio blk device for installation. 
# /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 316a0cb5-6ea7-4920-902c-afc06184b292 -rtc base=localtime,clock=host,driftfix=slew -drive file=/mnt/bug997839testing.raw,if=none,id=drive-ide-disk,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x3,drive=drive-ide-disk,id=ide-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=e1000-net-pci0,mac=2C:41:38:B6:40:11,bus=pci.0,addr=0x5 -k en-us -boot menu=on -qmp tcp:0:5555,server,nowait -spice disable-ticketing,port=5931 -monitor stdio -drive file=/home/RHEL6.4-20130130.0-Server-x86_64-DVD1.iso,if=none,id=drive-cd-disk,format=raw,cache=none,media=cdrom -device ide-drive,bus=ide.0,unit=1,drive=drive-cd-disk,id=cd-disk,bootindex=0

Results:
after step 5, qemu will prompt "block I/O error in device 'drive-ide-disk': Invalid argument (22)" in HMP monitor.
(qemu) main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 45.702000 ms, bitrate 6564102564 bps (6260.016026 Mbps)
inputs_connect: inputs channel client create
red_dispatcher_set_cursor_peer: 
block I/O error in device 'drive-ide-disk': Invalid argument (22)
block I/O error in device 'drive-ide-disk': Invalid argument (22)
block I/O error in device 'drive-ide-disk': Invalid argument (22)

And guest also prompt some error and cann't continue install, i will attach the screenshot later.


Base on above, mark qa_ack+ to it, Please correct me if any mistake.

Best Regards,
sluo
Comment 5 Sibiao Luo 2013-08-23 04:33:50 EDT
Created attachment 789500 [details]
screenshot for guest installration error.
Comment 7 Kevin Wolf 2013-09-18 05:48:15 EDT
You need to use the option logical_block_size=4096 for the virtio-blk-pci device,
which will present a 4k sectors disk to the guest. However, this will mean that
you can't boot from the disk but only use it as a data disk.

The other options is to use a cache mode that doesn't involve O_DIRECT, for
example cache=writeback. In this case, qemu can write even blocks smaller than
the XFS sector size.

Note You need to log in before you can comment on or make changes to this bug.