Bug 2225439

Summary:	[vdpa-blk] read-only=on option not work on driver virtio-blk-vhost-vdpa
Product:	Red Hat Enterprise Linux 9	Reporter:	qing.wang <qinwang>
Component:	qemu-kvm	Assignee:	Stefano Garzarella <sgarzare>
qemu-kvm sub component:	virtio-blk,scsi	QA Contact:	qing.wang <qinwang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	aliang, chayang, coli, jinzhao, juzhang, kwolf, lijin, qizhu, sgarzare, stefanha, virt-maint, xuwei, zhenyzha
Version:	9.3	Keywords:	Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu-kvm-8.0.0-11.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-11-07 08:28:05 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description qing.wang 2023-07-25 10:17:26 UTC

Description of problem:
Boot VM  with read-only=on option on the blockdev 
node driver virtio-blk-vhost-vdpa

VM boot failed due to
qemu-kvm: -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,read-only=on: blkio_start failed: Bad file descriptor (os error 9): Bad file descriptor



Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-340.el9.x86_64
qemu-kvm-8.0.0-8.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.3.0-2.el9.x86_64
libblkio-1.3.0-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:

1. parare vhost vdpa disks on host

  modprobe vhost-vdpa
  modprobe vdpa-sim-blk
  vdpa dev add mgmtdev vdpasim_blk name blk0
  vdpa dev add mgmtdev vdpasim_blk name blk1
  vdpa dev list -jp
  ls /dev/vhost-vdpa*
  [ $? -ne 0 ] && echo "wrong create vdpa device"

2. boot VM with read-pnly=on on virtio-blk-vhost-vdpa driver

/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,discard=unmap,detect-zeroes=on \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
  \
  -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,read-only=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \



3.

Actual results:
VM boot failed due to
qemu-kvm: -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,read-only=on: blkio_start failed: Bad file descriptor (os error 9): Bad file descriptor
Expected results:
boot succeed and the disk should read only in guest
Additional info:

Comment 1 Stefano Garzarella 2023-07-25 12:18:11 UTC

Patch posted upstream: https://lore.kernel.org/qemu-devel/20230725111155.85426-1-sgarzare@redhat.com/

Comment 4 qing.wang 2023-08-09 12:38:02 UTC

Failed on 
Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-348.el9.x86_64
qemu-kvm-8.0.0-11.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.5.0-5.el9.x86_64
virtio-win-prewhql-0.1-240.iso



/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,discard=unmap,detect-zeroes=on \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device virtio-blk-pci,iothread=iothread0,share-rw=on,serial=data0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
  \
  -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,read-only=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device scsi-hd,id=stg1,drive=fmt_stg1,serial=data1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \



QEMU 8.0.0 monitor - type 'help' for more information
(qemu) qemu-kvm: -device scsi-hd,id=stg1,drive=fmt_stg1,serial=data1,bootindex=2: Block node is read-only

Comment 5 Stefano Garzarella 2023-08-09 12:45:35 UTC

Sorry, but the output is telling that the blockdev is read-only as we asked, so I think it is validating the issue, no?

If `scsi-hd` is supposed to work also with a read-only blockdev, then we should open another BZ, because this is a different issue.
Can you try with another read-only blockdev (e.g. file) ?

Comment 6 Stefano Garzarella 2023-08-09 12:48:56 UTC

I just tried the following using a "file" blockdev and I had the same output, so I think this is the expected behavior:

/usr/libexec/qemu-kvm \
  ... \
  -device scsi-hd,drive=dr1,bus=virtio_scsi_ccw0.0 \
  -blockdev file,node-name=file_stg0,filename=/tmp/storage0.qcow2,read-only=true \
  -blockdev qcow2,node-name=drive_stg0,file=file_stg0 \
  ...
qemu-kvm: -blockdev qcow2,node-name=drive_stg0,file=file_stg0: Block node is read-only

Comment 7 Stefano Garzarella 2023-08-09 13:01:08 UTC

I pasted a wrong example, the right one is this:

/usr/libexec/qemu-kvm \
  ... \
  -device scsi-hd,drive=dr1,bus=virtio_scsi_ccw0.0 \
  -blockdev file,node-name=prot_stg0,filename=/tmp/storage0.qcow2,read-only=true \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device scsi-hd,drive=fmt_stg0,bus=virtio_scsi_ccw0.0 \
  ...
qemu-kvm: -device scsi-hd,drive=fmt_stg0,bus=virtio_scsi_ccw0.0: Block node is read-only

Comment 8 qing.wang 2023-08-09 13:13:08 UTC

Sorry, my mistake, the read-only=on alse need to add to format node


-blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,auto-read-only=on,force-share=off,read-only=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1,read-only=on \
  -device scsi-hd,id=stg1,drive=fmt_stg1,serial=data1,bootindex=2 \


It boot succeed , login the guest

mkfs.xfs /dev/sdb
mkfs.xfs: cannot open /dev/sdb: Read-only file system


It get expected result.

Comment 9 Yanan Fu 2023-08-10 11:18:30 UTC

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 12 qing.wang 2023-08-14 02:40:05 UTC

Pass test on 


Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-352.el9.x86_64
qemu-kvm-8.0.0-11.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.5.0-5.el9.x86_64
virtio-win-prewhql-0.1-240.iso




/usr/libexec/qemu-kvm \
 -name testvm \
 -machine q35,memory-backend=mem \
 -object memory-backend-memfd,id=mem,size=6G,share=on \
 -m 6G \
 -smp 2 \
 -cpu host,+kvm_pv_unhalt \
 -device ich9-usb-ehci1,id=usb1 \
 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
 -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
 -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
 -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
 -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
 -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
 -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \
 -object iothread,id=iothread0 \
 -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
 -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \
 -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \
 -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,discard=unmap,detect-zeroes=on \
 -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
 -device virtio-blk-pci,iothread=iothread0,share-rw=on,serial=data0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
 -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,auto-read-only=on,force-share=off,read-only=on \
 -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1,read-only=on \
 -device scsi-hd,id=stg1,drive=fmt_stg1,serial=data1,bootindex=2 \
 -vnc :5 \
 -monitor stdio \
 -qmp tcp:0:5955,server=on,wait=off \
 -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
 -netdev tap,id=nicpci \
 -boot menu=on,reboot-timeout=1000,strict=off \
 -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
 -serial chardev:socket-serial \
 -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
 -device isa-debugcon,chardev=file-bios,iobase=0x402 \
 -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
 -mon chardev=socket-qmp,mode=control \
 -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
 -mon chardev=socket-hmp,mode=readline





login the guest

mkfs.xfs /dev/sdb
mkfs.xfs: cannot open /dev/sdb: Read-only file system

Comment 14 errata-xmlrpc 2023-11-07 08:28:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368