Description of problem: When using storage with sector size of 4k, qemu and qemu-img fail to probe the alignment requirement for direct I/O, and fail with EINVAL when accesing storage. Several flows may fail: - Provisioning a VM on 4k storage, fail when the installer try to create filesystems - Coping disk image from 4k storage, fail when reading from source image - Copying disk to 4k storage, fail when writing to the target disk I reproduced the failures with: - xfs on loop devices using 4k sector size - gluster backed by xfs, on vdo device (exposing 4k sector size) The root cause for both issues is alignment probing. The issue was fixed upstream in this commit: https://github.com/qemu/qemu/commit/a6b257a08e3d72219f03e461a52152672fec0612 This is the RHEL version of the these Fedora bugs: - Bug 1737256 - Provisioning VM on 4k gluster storage fails with "Invalid argument" - qemu fail to detect block size - Bug 1738657 - qemu-img convert fail to read with "Invalid argument" on gluster storage with 4k sector size I merged both bugs for RHEL since we understand now that both issue are the same. Version-Release number of selected component (if applicable): Tested with qemu/qemu-img 4.1 rc2 on Fedora 29 Tested with qemu-rhev/qemu-img-rhev on CentOS 7.6 How reproducible: Always Note: copying disk depends on the disk content, not all disk fail. Steps to Reproduce - provisioning - xfs on loop device: 1. Create loop device with 4 sector size: losetup -f backing-file --show --sector-size=4096 2. Create xfs file system mkfs -t xfs /dev/loop0 3. Mount mkdir /tmp/loop0 mount /dev/loop0 /tmp/loop0 4. Create new disk qemu-img create -f raw /tmp/loop0/disk.img 5. Start a VM: qemu-system-x86_64 -accel kvm -m 2048 -smp 2 \ -drive file=/tmp/loop0/disk.img,format=raw,cache=none \ -cdrom Fedora-Server-dvd-x86_64-29-1.2.iso 6. Try to install with default options. The installer fails in few seconds when trying to create filesystem on the root logical volume. Steps to Reproduce - copying disk from 4k storage - xfs on loop device: 1. Create a new image on 4k storage. One way is to use virt-builder: virt-builder fedora-29 -o disk.img 2. Copy the disk to target image elsewhere: qemu-img convert -f raw -O raw -t none -T none /tmp/loop0/disk.img \ disk-clone.img Will fail with EINVAL when reading from the image: qemu-img: error while reading sector XXX: Invalid argument Steps to Reproduce - copying disk to 4k storage - xfs on loop device: 1. Create a new image on 4k storage. One way is to use virt-builder: virt-builder fedora-29 -o /tmp/loop0/disk.img 2. Copy the disk to target image on the 4k storage: qemu-img convert -f raw -O raw -t none -T none disk.img \ /tmp/loop0/disk-clone.img Will fail with EINVAL when writing to target image: qemu-img: error while writing sector XXX: Invalid argument Steps to reproduce - gluster/xfs/vdo storage Creating this storage is more complex. I reproduced this using 3 vms, deployed using these scripts: - https://github.com/oVirt/vdsm/blob/master/contrib/deploy-gluster.sh - https://github.com/oVirt/vdsm/blob/master/contrib/create-vdo-brick.sh - https://github.com/oVirt/vdsm/blob/master/contrib/create-gluster-volume.sh You need also to set this gluster volume option: gluster volume set gv0 performance.strict-o-direct on Once all gluster nodes are up, mount the storage: mkdir /tmp/gv0 mount -t glusterfs gluster1:/gv0 /tmp/gv0 Now you can reproduce using the same flows explained above for loop device, replacing /tmp/loop0 with /tmp/gv0.
See also bug 1743365 for RHEL 7.8.
I can reproduce this bug in scenarios of installation and convert. Tested with: qemu-kvm-4.1.0-4.module+el8.1.0+4020+16089f93 kernel-4.18.0-134.el8 Scenario 1(Installation) 1. Create a raw image on the xfs with 4k sectors # qemu-img create -f raw base.img 20G 2. Install guest on it # /usr/libexec/qemu-kvm \ -name 'guest-rhel8.0' \ -machine q35 \ -nodefaults \ -vga qxl \ -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=unsafe,media=cdrom,file=/mnt/hucheng/RHEL8.0.0-BaseOS-x86_64.iso \ -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=base.img \ -device virtio-blk-pci,id=virtio_blk_pci0,drive=drive_image1,bus=pcie.0,addr=05,bootindex=0 \ -vnc :0 \ -monitor stdio \ -m 8192 \ -smp 8 \ -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9 \ -netdev tap,id=idxgXAlm \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/hucheng/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ Result: When the process in 'create xfs', install will hit error. And there is I/O error in QMP. The PrtSc is attached. {"timestamp": {"seconds": 1566385831, "microseconds": 937572}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive_image1", "nospace": false, "node-name": "#block398", "reason": "Invalid argument", "operation": "read", "action": "report"}} {"timestamp": {"seconds": 1566385831, "microseconds": 938011}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive_image1", "nospace": false, "node-name": "#block398", "reason": "Invalid argument", "operation": "write", "action": "report"}} Scenario 2(Convert) 1. Create a test image on the xfs with 4k sectors # dd if=/dev/urandom of=test.img bs=1M count=2048 2. Convert the image to the xfs with 512 bytes sectors # qemu-img convert -f raw -O raw test.img /home/tgt.img -t none -T none -p (100.00/100%) 3. Convert back to xfs with 4k sectors # qemu-img convert -f raw -O raw /home/tgt.img tgt.img -t none -T none -p qemu-img: error while writing sector 4190208: Invalid argument Additional info: 1. Disk info. # fdisk -l Disk /dev/sda: 372 GiB, 399431958528 bytes, 780140544 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0xb768558d Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 2099199 2097152 1G 83 Linux /dev/sda2 2099200 780140543 778041344 371G 8e Linux LVM Disk /dev/sdb: 558.4 GiB, 599550590976 bytes, 1170997248 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 44B4C16F-E8F3-4AB0-9ED8-EB99E1452090 Device Start End Sectors Size Type /dev/sdb1 34 19531250 19531217 9.3G Microsoft basic data Disk /dev/sdc: 558.4 GiB, 599550590976 bytes, 146374656 sectors Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0xa2466bb0 Device Boot Start End Sectors Size Id Type /dev/sdc1 * 256 262399 262144 1G 83 Linux /dev/sdc2 262400 146374655 146112256 557.4G 8e Linux LVM # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 372G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 371G 0 part ├─rhel_dell--per740xd--01-root 253:0 0 50G 0 lvm / ├─rhel_dell--per740xd--01-swap 253:1 0 31.4G 0 lvm [SWAP] └─rhel_dell--per740xd--01-home 253:5 0 289.6G 0 lvm /home sdb 8:16 0 558.4G 0 disk └─sdb1 8:17 0 9.3G 0 part sdc 8:32 0 558.4G 0 disk ├─sdc1 8:33 0 1G 0 part └─sdc2 8:34 0 557.4G 0 part ├─rhel_dell--per740xd--0101-swap 253:2 0 31.4G 0 lvm ├─rhel_dell--per740xd--0101-home 253:3 0 476G 0 lvm /mnt └─rhel_dell--per740xd--0101-root 253:4 0 50G 0 lvm 2. Mount info # mount | grep mnt /dev/mapper/rhel_dell--per740xd--0101-home on /mnt type xfs (rw,relatime,seclabel,attr2,inode64,noquota) 3. The director I operated. # pwd /mnt/test
Created attachment 1606458 [details] PrtSc for install error
*** This bug has been marked as a duplicate of bug 1749134 ***