Bug 2120275

Summary: Wrong max_sectors_kb and Maximum transfer length on the pass-through device [rhel-9.1]
Product: Red Hat Enterprise Linux 9 Reporter: qing.wang <qinwang>
Component: qemu-kvmAssignee: Kevin Wolf <kwolf>
qemu-kvm sub component: virtio-blk,scsi QA Contact: qing.wang <qinwang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bhull, coli, jinzhao, juzhang, kkiwi, kwolf, lijin, mgandhi, mrezanin, pvlasin, qzhang, virt-maint, xuwei, zhguo
Version: 9.1Keywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: qemu-kvm-7.0.0-12.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2121430 (view as bug list) Environment:
Last Closed: 2022-11-15 09:55:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2121430    

Description qing.wang 2022-08-22 12:05:04 UTC
Description of problem:
This issue come from Bug 2096251 - RHEL8.4 with qemu sg_io passthrough devices advertise a failing max_sectors_kb size of 1280 for the LUN 

The problem only happens on hardware that does not support the Block Limits VPD page

For example:

Check the host FC disk

[root@localhost tmp]# for tmp_f in /sys/block/sdd/queue/[mlp]*; do echo -n $tmp_f: ; cat $tmp_f; done
/sys/block/sdd/queue/logical_block_size:512
/sys/block/sdd/queue/max_discard_segments:1
/sys/block/sdd/queue/max_hw_sectors_kb:32767
/sys/block/sdd/queue/max_integrity_segments:0
/sys/block/sdd/queue/max_sectors_kb:1280
/sys/block/sdd/queue/max_segments:64
/sys/block/sdd/queue/max_segment_size:65536
/sys/block/sdd/queue/minimum_io_size:512
/sys/block/sdd/queue/physical_block_size:512
[root@localhost tmp]# sg_inq --page bl /dev/sdd
    inquiry: Numerical argument out of domain
sg_inq failed: Numerical argument out of domain

[root@localhost tmp]# sg_vpd -a /dev/sdd
Supported VPD pages VPD page:
  Supported VPD pages [sv]
  Unit serial number [sn]
  Device identification [di]
  0xc0
  0xe0
  0xe3


There is no Block Limits VPD page.

pass-through the disk as scsi-block 

It get wrong Maximum transfer length result in io-error.

[root@testg ~]# for tmp_f in /sys/block/sda/queue/[mlp]*; do echo -n $tmp_f: ; cat $tmp_f; done
/sys/block/sda/queue/logical_block_size:512
/sys/block/sda/queue/max_discard_segments:1
/sys/block/sda/queue/max_hw_sectors_kb:32767
/sys/block/sda/queue/max_integrity_segments:0
/sys/block/sda/queue/max_sectors_kb:1280
/sys/block/sda/queue/max_segments:126
/sys/block/sda/queue/max_segment_size:65536
/sys/block/sda/queue/minimum_io_size:512
/sys/block/sda/queue/physical_block_size:512
[root@testg ~]# sg_inq --page bl /dev/sda
VPD INQUIRY: Block limits page (SBC)
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 0 blocks [not reported]
  Maximum transfer length: 4194303 blocks   << invalid value 
  Optimal transfer length: 0 blocks [not reported]
  Maximum prefetch transfer length: 0 blocks [ignored]
  Maximum unmap LBA count: 0 [Unmap command not implemented]
  Maximum unmap block descriptor count: 0 [Unmap command not implemented]
  Optimal unmap granularity: 0 blocks [not reported]
  Unmap granularity alignment valid: false
  Unmap granularity alignment: 0 [invalid]
  Maximum write same length: 0x3fffff blocks
  Maximum atomic transfer length: 0 blocks [not reported]
  Atomic alignment: 0 [unaligned atomic writes permitted]
  Atomic transfer length granularity: 0 [no granularity requirement
  Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
  Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]




Version-Release number of selected component (if applicable):
qemu-kvm-7.0.0-9.el9.x86_64


How reproducible:


Steps to Reproduce:
1.Boot vm with pass-through disk

/usr/libexec/qemu-kvm \
  -name src_vm1 \
  -machine q35,accel=kvm,usb=off,dump-guest-core=off \
  -m 8G \
  -smp 2 \
  -cpu host,vmx,+kvm_pv_unhalt \
  -device pcie-root-port,id=pcie.0-root-port-2,slot=2,bus=pcie.0,multifunction=on \
  -device pcie-root-port,id=pcie.0-root-port-2-1,chassis=3,bus=pcie.0,addr=0x2.0x1 \
  -device pcie-root-port,id=pcie.0-root-port-2-2,chassis=4,bus=pcie.0,addr=0x2.0x2 \
  -device pcie-root-port,id=pcie.0-root-port-3,slot=3,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-4,slot=4,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-5,slot=5,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-7,slot=7,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-8,slot=8,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-9,slot=9,bus=pcie.0 \
  -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2-1,addr=0x0 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie.0-root-port-2-2,addr=0x0,iothread=iothread0 \
  -blockdev \
  driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel79-64-virtio-scsi.qcow2,node-name=drive_image1 \
  -device scsi-hd,id=os1,drive=drive_image1,bootindex=0 \
  \
  \
  -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
  \
  -blockdev node-name=host_device_stg,driver=host_device,aio=native,filename=/dev/$disk,cache.direct=on,cache.no-flush=off,discard=unmap \
  -blockdev node-name=drive_stg,driver=raw,cache.direct=on,cache.no-flush=off,file=host_device_stg \
  -device scsi-block,id=stg,drive=drive_stg,rerror=stop,werror=stop \
  \
  -vnc \
  :5 \
  -qmp tcp:0:5955,server,nowait \
  -monitor stdio \
  \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b5,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0-root-port-5,addr=0x0 \
  -netdev tap,id=idxgXAlm


2.check disk info in guest

[root@testg ~]# 
for tmp_f in /sys/block/sdb/queue/[mlp]*; do echo -n $tmp_f: ; cat $tmp_f; done

[root@testg ~]# sg_inq --page bl /dev/sdb

3.

Actual results:
[root@testg ~]# for tmp_f in /sys/block/sdb/queue/[mlp]*; do echo -n $tmp_f: ; cat $tmp_f; done
/sys/block/sdb/queue/logical_block_size:512
..
/sys/block/sdb/queue/max_sectors_kb:1280
/sys/block/sdb/queue/max_segments:126

[root@testg ~]# sg_inq --page bl /dev/sdb
VPD INQUIRY: Block limits page (SBC)
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 0 blocks [not reported]
  Maximum transfer length: 4194303 blocks   << invalid value 

Expected results:
The max_sectors_kb >=  4 * max_segments 
The maximum transfer length / 2 = max_sectors_kb

[root@testg mnt]# grep -H . /sys/block/*/queue/max_sectors_kb
/sys/block/sdb/queue/max_sectors_kb:256


[root@testg mnt]# sg_inq --page 176 /dev/sdb
VPD INQUIRY: Block limits page (SBC)
  Maximum compare and write length: 0 blocks [Command not implemented]
  Optimal transfer length granularity: 0 blocks [not reported]
  Maximum transfer length: 512 blocks


Additional info:

Comment 2 Kevin Wolf 2022-08-24 10:13:21 UTC
A scratch build with the fix is available at:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47311108

Comment 9 Yanan Fu 2022-08-29 07:04:33 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 13 qing.wang 2022-08-30 08:06:59 UTC
Passed block sanity test and manually verification below:


Red Hat Enterprise Linux release 9.1 Beta (Plow)
5.14.0-152.el9.x86_64
qemu-kvm-7.0.0-12.el9.x86_64
seabios-bin-1.16.0-4.el9.noarch
edk2-ovmf-20220526git16779ede2d36-3.el9.noarch


Get FC disk on HOST

/sys/block/sdb/queue/max_sectors_kb:1280
/sys/block/sdb/queue/max_segments:64

VPD INQUIRY: Block limits page (SBC)
  Maximum transfer length: 0 blocks [not reported]



pass-through to guest

[ "$1" == "" ] && disk=sdb || disk=$1

/usr/libexec/qemu-kvm \
  -name src_vm1 \
  -machine q35,accel=kvm,usb=off,dump-guest-core=off \
  -m 8G \
  -smp 2 \
  -cpu host,vmx,+kvm_pv_unhalt \
  -device pcie-root-port,id=pcie.0-root-port-2,slot=2,bus=pcie.0,multifunction=on \
  -device pcie-root-port,id=pcie.0-root-port-2-1,chassis=3,bus=pcie.0,addr=0x2.0x1 \
  -device pcie-root-port,id=pcie.0-root-port-2-2,chassis=4,bus=pcie.0,addr=0x2.0x2 \
  -device pcie-root-port,id=pcie.0-root-port-3,slot=3,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-4,slot=4,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-5,slot=5,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-7,slot=7,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-8,slot=8,bus=pcie.0 \
  -device pcie-root-port,id=pcie.0-root-port-9,slot=9,bus=pcie.0 \
  -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2-1,addr=0x0 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie.0-root-port-2-2,addr=0x0,iothread=iothread0 \
  -blockdev \
  driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel840-64-virtio-scsi.qcow2,node-name=drive_image1 \
  -device scsi-hd,id=os1,drive=drive_image1,bootindex=0 \
  \
  \
  -device virtio-scsi-pci,id=scsi1,bus=pcie.0-root-port-8,addr=0x0 \
  \
  -blockdev node-name=host_device_stg,driver=host_device,aio=native,filename=/dev/$disk,cache.direct=on,cache.no-flush=off,discard=unmap \
  -blockdev node-name=drive_stg,driver=raw,cache.direct=on,cache.no-flush=off,file=host_device_stg \
  -device scsi-block,id=stg,drive=drive_stg,rerror=stop,werror=stop \
  \
  -vnc \
  :5 \
  -qmp tcp:0:5955,server,nowait \
  -monitor stdio \
  \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b5,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0-root-port-5,addr=0x0 \
  -netdev tap,id=idxgXAlm




Get disk info in guest

/sys/block/sdb/queue/max_sectors_kb:256
/sys/block/sdb/queue/max_segments:254

  Maximum transfer length: 512 blocks


change max_sectors_kb to 64 on host then reboot guest

echo 64 > /sys/block/sdb/queue/max_sectors_kb


/sys/block/sdb/queue/max_sectors_kb:64
/sys/block/sdb/queue/max_segments:254

  Maximum transfer length: 128 blocks
  

change max_sectors_kb to 1024 on host then reboot guest
echo 1024 > /sys/block/sdb/queue/max_sectors_kb

/sys/block/sdb/queue/max_sectors_kb:256
/sys/block/sdb/queue/max_segments:254

  Maximum transfer length: 512 blocks

Comment 15 errata-xmlrpc 2022-11-15 09:55:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7967