Bug 1848881 - nvme:// block driver can exhaust IOMMU DMAs, hanging the VM, possible data loss (raw format)
Summary: nvme:// block driver can exhaust IOMMU DMAs, hanging the VM, possible data lo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.5
Assignee: Philippe Mathieu-Daudé
QA Contact: Tingting Mao
URL:
Whiteboard:
: 1934172 (view as bug list)
Depends On:
Blocks: 1900136 1948358 1957194
TreeView+ depends on / blocked
 
Reported: 2020-06-19 07:19 UTC by Xueqiang Wei
Modified: 2021-11-16 07:57 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 07:49:57 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
gdb log (60.26 KB, text/plain)
2020-06-22 12:06 UTC, Xueqiang Wei
no flags Details

Description Xueqiang Wei 2020-06-19 07:19:29 UTC
Description of problem:

qemu-kvm: VFIO_MAP_DMA failed: Invalid argument when boot a guest with NVMe device

Version-Release number of selected component (if applicable):
kernel-4.18.0-214.el8.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420


How reproducible:
100%

Steps to Reproduce:
1. create a qcow2 image on NVMe deviece
# qemu-img create -f qcow2 nvme://0000:bc:00.0/1 20G

2. boot a guest with the new image
# cat boot.sh
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/monitor-qmpmonitor1-20200522-125204-4Vi7sqOR,server,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/monitor-catch_monitor-20200522-125204-4Vi7sqOR,server,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idX2dIhI \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/serial-serial0-20200522-125204-4Vi7sqOR,server,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200522-125204-4Vi7sqOR,path=/var/tmp/avocado_xpeuo28b/seabios-20200522-125204-4Vi7sqOR,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200522-125204-4Vi7sqOR,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1 \
    -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1 \
    -device scsi-hd,id=nvme1,drive=drive_nvme1 \

3. dd test in guest
# dd if=/dev/zero of=/dev/sdb bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 1.53942 s, 681 MB/s

# dmesg |grep error
#

4. reset the guest in qmp monitor
# nc -U /var/tmp/avocado_xpeuo28b/monitor-qmpmonitor1-20200522-125204-4Vi7sqOR
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 5}, "package": "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420"}, "capabilities": ["oob"]}}
{"execute": "qmp_capabilities"}
{"return": {}}

{"execute": "system_reset"}
{"timestamp": {"seconds": 1592549695, "microseconds": 279896}, "event": "RESET", "data": {"guest": false, "reason": "host-qmp-system-reset"}}
{"timestamp": {"seconds": 1592549695, "microseconds": 310687}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}

5. check dmesg in host and guest
Host:
# dmesg |grep error
#

Guest:
# lsblk
NAME                             MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda                                8:0    0  20G  0 disk 
├─sda1                             8:1    0   1G  0 part /boot
└─sda2                             8:2    0  19G  0 part 
  ├─rhel_bootp--73--226--76-root 253:0    0  17G  0 lvm  /
  └─rhel_bootp--73--226--76-swap 253:1    0   2G  0 lvm  [SWAP]
sdb                                8:16   0  20G  0 disk 
# dmesg |grep error
#


Actual results:
after step 2, hit it in hmp monitor
# sh boot.sh 
QEMU 5.0.0 monitor - type 'help' for more information
(qemu) qemu-kvm: VFIO_MAP_DMA failed: Invalid argument
(qemu) c

after step 3, disk works well, not found error in dmesg
after step 4, not found error in qmp monitor
after step 5, not found error in dmesg


Expected results:
Not hit it and NVMe device works well


Additional info:

Comment 2 Xueqiang Wei 2020-06-22 12:01:57 UTC
Failed to install a guest on it, VFIO_MAP_DMA failed: No space left on device.


Details:

1. # qemu-img create -f qcow2  nvme://0000:bc:00.0/1 20G
Formatting 'nvme://0000:bc:00.0/1', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. # qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

3. sh install.sh
# cat install.sh
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2 \
    -m 8G  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-qmpmonitor1-20200622-044212-f3M3b8EC,id=qmp_id_qmpmonitor1,server,nowait  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-catch_monitor-20200622-044212-f3M3b8EC,id=qmp_id_catch_monitor,server,nowait  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idWpRInM \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/serial-serial0-20200622-044212-f3M3b8EC,id=chardev_serial0,server,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200622-044212-f3M3b8EC,path=/var/tmp/avocado_6rbwb5nb/seabios-20200622-044212-f3M3b8EC,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200622-044212-f3M3b8EC,iobase=0x402 \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0 \
    -device virtio-net-pci,mac=9a:ce:07:93:d2:57,id=idVu6rLy,netdev=idIumYb0,bus=pci.0,addr=0x5  \
    -netdev tap,id=idIumYb0,vhost=on \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.3.0-20200616.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=2,write-cache=on,bus=ide.0,unit=0 \
    -blockdev node-name=file_unattended,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/images/rhel830-64/ks.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_unattended,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_unattended \
    -device ide-cd,id=unattended,drive=drive_unattended,bootindex=3,write-cache=on,bus=ide.0,unit=1  \
    -kernel '/home/kvm_autotest_root/images/rhel830-64/vmlinuz'  \
    -append 'inst.sshd ksdevice=link inst.repo=cdrom:/dev/sr0 inst.ks=cdrom:/dev/sr1:/ks.cfg nicdelay=60 biosdevname=0 net.ifnames=0 console=ttyS0,115200 console=tty0'  \
    -initrd '/home/kvm_autotest_root/images/rhel830-64/initrd.img'  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=d,strict=off  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \



after step 3:
# sh install.sh 
QEMU 5.0.0 monitor - type 'help' for more information
(qemu) qemu-kvm: VFIO_MAP_DMA failed: Invalid argument
(qemu) c
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
(qemu) 
(qemu) info status 
VM status: paused (io-error)
(qemu) c
VFIO_MAP_DMA failed: No space left on device
qemu-kvm: /builddir/build/BUILD/qemu-5.0.0/block/block-backend.c:1968: blk_get_aio_context: Assertion `ctx == blk->ctx' failed.
install.sh: line 44: 117833 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -m 30720 -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2 -cpu 'Skylake-Server',+kvm_pv_unhalt -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-qmpmonitor1-20200622-044212-f3M3b8EC,id=qmp_id_qmpmonitor1,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-catch_monitor-20200622-044212-f3M3b8EC,id=qmp_id_catch_monitor,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idWpRInM -chardev socket,path=/var/tmp/avocado_6rbwb5nb/serial-serial0-20200622-044212-f3M3b8EC,id=chardev_serial0,server,nowait -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20200622-044212-f3M3b8EC,path=/var/tmp/avocado_6rbwb5nb/seabios-20200622-044212-f3M3b8EC,server,nowait -device isa-debugcon,chardev=seabioslog_id_20200622-044212-f3M3b8EC,iobase=0x402 -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -object iothread,id=iothread1 -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off -blockdev node-name=drive_image1,driver=raw,cache.direct=off,cache.no-flush=off,file=nvme_image1 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0 -device virtio-net-pci,mac=9a:ce:07:93:d2:57,id=idVu6rLy,netdev=idIumYb0,bus=pci.0,addr=0x5 -netdev tap,id=idIumYb0,vhost=on -blockdev node-name=file_cd1,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.3.0-20200616.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 -device ide-cd,id=cd1,drive=drive_cd1,bootindex=2,write-cache=on,bus=ide.0,unit=0 -blockdev node-name=file_unattended,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/images/rhel830-64/ks.iso,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_unattended,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_unattended -device ide-cd,id=unattended,drive=drive_unattended,bootindex=3,write-cache=on,bus=ide.0,unit=1 -kernel '/home/kvm_autotest_root/images/rhel830-64/vmlinuz' -append 'inst.sshd ksdevice=link inst.repo=cdrom:/dev/sr0 inst.ks=cdrom:/dev/sr1:/ks.cfg nicdelay=60 biosdevname=0 net.ifnames=0 console=ttyS0,115200 console=tty0' -initrd '/home/kvm_autotest_root/images/rhel830-64/initrd.img' -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=d,strict=off -no-shutdown -enable-kvm -monitor stdio

Comment 3 Xueqiang Wei 2020-06-22 12:06:33 UTC
Created attachment 1698283 [details]
gdb log

Comment 6 Xueqiang Wei 2020-12-02 15:06:12 UTC
According to Comment 2, tested on rhel8.4-av, hit another issue.


Versions:
kernel-4.18.0-255.el8.x86_64
qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9



# sh install.sh 
QEMU 5.1.92 monitor - type 'help' for more information
(qemu) c
(qemu) error: kvm run failed No such file or directory
RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000150 RDX=000000008000001c
RSI=0000000000000000 RDI=0000000000000150 RBP=0000000000000000 RSP=ffffa48d00d1be28
R8 =ffff995677816748 R9 =0000000000000000 R10=0000000000000000 R11=00000001055dca40
R12=0000000000000000 R13=ffffffff896fd070 R14=ffff995547c01410 R15=ffffffff890dd450
RIP=ffffffff89064d98 RFL=00000296 [--S-AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00c00000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =0000 0000000000000000 ffffffff 00c00000
FS =0000 0000000000000000 ffffffff 00c00000
GS =0000 ffff995677800000 ffffffff 00c00000
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0040 fffffe0000003000 0000206f 00008b00 DPL=0 TSS64-busy
GDT=     fffffe0000001000 0000007f
IDT=     fffffe0000000000 00000fff
CR0=80050033 CR2=ffff99567ffff000 CR3=0000000077e0a001 CR4=003606f0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000fffe0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=40 00 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 08 89 f0 89 f9 <0f> 30 31 c0 0f 1f 44 00 00 48 83 c4 08 c3 48 c1 e2 20 89 f6 89 44 24 04 48 09 d6 89 c2 e8

(qemu) info status 
VM status: paused (internal-error)



# cat install.sh
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2 \
    -m 8G  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-qmpmonitor1-20200622-044212-f3M3b8EC,id=qmp_id_qmpmonitor1,server,nowait  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/monitor-catch_monitor-20200622-044212-f3M3b8EC,id=qmp_id_catch_monitor,server,nowait  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idWpRInM \
    -chardev socket,path=/var/tmp/avocado_6rbwb5nb/serial-serial0-20200622-044212-f3M3b8EC,id=chardev_serial0,server,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200622-044212-f3M3b8EC,path=/var/tmp/avocado_6rbwb5nb/seabios-20200622-044212-f3M3b8EC,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200622-044212-f3M3b8EC,iobase=0x402 \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=1,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0 \
    -device virtio-net-pci,mac=9a:ce:07:93:d2:57,id=idVu6rLy,netdev=idIumYb0,bus=pci.0,addr=0x5  \
    -netdev tap,id=idIumYb0,vhost=on \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.4.0-20200905.n.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=2,write-cache=on,bus=ide.0,unit=0 \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=d,strict=off  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \

Comment 7 Xueqiang Wei 2020-12-02 16:50:14 UTC
According to Description, tested on rhel.8.4-av, failed to install guest, also hit this bug.

Versions:
kernel-4.18.0-255.el8.x86_64
qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9


# sh install_q35.sh  
QEMU 5.1.92 monitor - type 'help' for more information
(qemu) c
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device

(qemu) info status 
VM status: paused (io-error)


# cat install_q35.sh
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/monitor-qmpmonitor1-20200522-125204-4Vi7sqOR,server,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/monitor-catch_monitor-20200522-125204-4Vi7sqOR,server,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idX2dIhI \
    -chardev socket,nowait,path=/var/tmp/avocado_xpeuo28b/serial-serial0-20200522-125204-4Vi7sqOR,server,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200522-125204-4Vi7sqOR,path=/var/tmp/avocado_xpeuo28b/seabios-20200522-125204-4Vi7sqOR,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200522-125204-4Vi7sqOR,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap \
    -device scsi-hd,id=nvme1,drive=drive_nvme1,bootindex=1 \
    -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=6 \
    -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-6,addr=0x0 \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.4.0-20200905.n.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on,bootindex=0 \

Comment 8 Xueqiang Wei 2020-12-02 17:49:52 UTC
Tested with raw format and q35 machine type, guest installs successfully.

Versions:
kernel-4.18.0-255.el8.x86_64
qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9



# qemu-img create -f raw nvme://0000:bc:00.0/1 30G

# qemu-img info nvme://0000:bc:00.0/1 
image: nvme://0000:bc:00.0/1
file format: raw
virtual size: 745 GiB (800166076416 bytes)
disk size: unavailable


# sh install_q35.sh 
QEMU 5.1.92 monitor - type 'help' for more information
(qemu) c
(qemu) info block
drive_nvme1: nvme://0000:bc:00.0/1 (raw)
    Attached to:      nvme1
    Cache mode:       writeback

drive_cd1: /home/kvm_autotest_root/iso/linux/RHEL-8.4.0-20200905.n.0-x86_64-dvd1.iso (raw, read-only)
    Attached to:      cd1
    Removable device: locked, tray closed
    Cache mode:       writeback, direct
(qemu) 



cmd lines:
-device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
-device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
-blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_nvme1,driver=raw,file=nvme_image1,read-only=off,discard=unmap \
-device scsi-hd,id=nvme1,drive=drive_nvme1,bootindex=0 \


check message in guest after the installation:
# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0 745.2G  0 disk 
├─sda1          8:1    0     1G  0 part /boot
└─sda2          8:2    0 744.2G  0 part 
  ├─rhel-root 253:0    0    70G  0 lvm  /
  ├─rhel-swap 253:1    0   7.4G  0 lvm  [SWAP]
  └─rhel-home 253:2    0 666.8G  0 lvm  /home
sr0            11:0    1   8.7G  0 rom  /run/media/xuwei/RHEL-8-4-0-BaseOS-x86_64

# dmesg | grep error
# 



Tested with raw format and pc machine type, hit the same issue with Comment 6.

Comment 9 Philippe Mathieu-Daudé 2020-12-03 13:59:22 UTC
(In reply to Xueqiang Wei from comment #8)
> Tested with raw format and q35 machine type, guest installs successfully.
> 
> # qemu-img create -f raw nvme://0000:bc:00.0/1 30G
> 
> # qemu-img info nvme://0000:bc:00.0/1 
> image: nvme://0000:bc:00.0/1
> file format: raw
> virtual size: 745 GiB (800166076416 bytes)
> disk size: unavailable

Thanks, this might be a hint about what is going on.

Comment 10 Philippe Mathieu-Daudé 2020-12-03 22:20:22 UTC
(In reply to Xueqiang Wei from comment #8)
> Tested with raw format and q35 machine type, guest installs successfully.
> 
...
> # qemu-img create -f raw nvme://0000:bc:00.0/1 30G

You ask 30G but the 'raw' format doesn't contain metadata, ...

> 
> # qemu-img info nvme://0000:bc:00.0/1 
> image: nvme://0000:bc:00.0/1
> file format: raw
> virtual size: 745 GiB (800166076416 bytes)

... so the full disk is used.

> check message in guest after the installation:
> # lsblk
> NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda             8:0    0 745.2G  0 disk 

As confirmed within guest  ^^^

Comment 11 Xueqiang Wei 2020-12-04 03:17:12 UTC
(In reply to Philippe Mathieu-Daudé from comment #10)
> (In reply to Xueqiang Wei from comment #8)
> > Tested with raw format and q35 machine type, guest installs successfully.
> > 
> ...
> > # qemu-img create -f raw nvme://0000:bc:00.0/1 30G
> 
> You ask 30G but the 'raw' format doesn't contain metadata, ...
> 
> > 
> > # qemu-img info nvme://0000:bc:00.0/1 
> > image: nvme://0000:bc:00.0/1
> > file format: raw
> > virtual size: 745 GiB (800166076416 bytes)
> 
> ... so the full disk is used.
> 
> > check message in guest after the installation:
> > # lsblk
> > NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> > sda             8:0    0 745.2G  0 disk 
> 
> As confirmed within guest  ^^^


Yes, I have confirmed it with Kevin before.

Feedback from Kevin:

"Yes, this is the expected behavior. qemu-img cannot change the size of
a block device, so creating a raw image just checks that the block
device has _at least_ the requested size, but it can be larger.

qcow2, in contrast, has metadata and can provide the exact size."


# qemu-img create -f qcow2 nvme://0000:bc:00.0/1 30G
Formatting 'nvme://0000:bc:00.0/1', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=32212254720 lazy_refcounts=off refcount_bits=16

# qemu-img info nvme://0000:bc:00.0/1 
image: nvme://0000:bc:00.0/1
file format: qcow2
virtual size: 30 GiB (32212254720 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false


# qemu-img create -f raw  nvme://0000:bc:00.0/1 30G
Formatting 'nvme://0000:bc:00.0/1', fmt=raw size=32212254720

# qemu-img info nvme://0000:bc:00.0/1 
image: nvme://0000:bc:00.0/1
file format: raw
virtual size: 745 GiB (800166076416 bytes)
disk size: unavailable

Comment 12 Philippe Mathieu-Daudé 2020-12-09 15:43:51 UTC
(In reply to Xueqiang Wei from comment #8)
> Tested with raw format and q35 machine type, guest installs successfully.
> 
> # qemu-img create -f raw nvme://0000:bc:00.0/1 30G
> Formatting 'nvme://0000:bc:00.0/1', fmt=raw size=32212254720

                                              ^^^^^^^^^^^^^^^^
As the 'raw' format doesn't provide metadata, this size ...

> cmd lines:
> -blockdev
> node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-
> only=on,discard=unmap \
> -blockdev
> node-name=drive_nvme1,driver=raw,file=nvme_image1,read-only=off,
> discard=unmap \

... has to be passed along to the blockdev. Otherwise the full drive capacity is used:

-blockdev node-name=drive_nvme1,driver=raw,size=32212254720,...
                                           ^^^^^^^^^^^^^^^^

Comment 13 Xueqiang Wei 2020-12-09 17:32:41 UTC
(In reply to Philippe Mathieu-Daudé from comment #12)
> (In reply to Xueqiang Wei from comment #8)
> > Tested with raw format and q35 machine type, guest installs successfully.
> > 
> > # qemu-img create -f raw nvme://0000:bc:00.0/1 30G
> > Formatting 'nvme://0000:bc:00.0/1', fmt=raw size=32212254720
> 
>                                               ^^^^^^^^^^^^^^^^
> As the 'raw' format doesn't provide metadata, this size ...
> 
> > cmd lines:
> > -blockdev
> > node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-
> > only=on,discard=unmap \
> > -blockdev
> > node-name=drive_nvme1,driver=raw,file=nvme_image1,read-only=off,
> > discard=unmap \
> 
> ... has to be passed along to the blockdev. Otherwise the full drive
> capacity is used:
> 
> -blockdev node-name=drive_nvme1,driver=raw,size=32212254720,...
>                                            ^^^^^^^^^^^^^^^^


Please check the comment in https://bugzilla.redhat.com/show_bug.cgi?id=1900136#c5,  tested with size=32212254720 hit the issue that can't install guest successfully.

Comment 17 Xueqiang Wei 2021-03-02 22:06:23 UTC
*** Bug 1934172 has been marked as a duplicate of this bug. ***

Comment 18 Philippe Mathieu-Daudé 2021-06-11 15:31:22 UTC
Fix posted to mainstream:
https://www.mail-archive.com/qemu-block@nongnu.org/msg86159.html

Comment 19 Klaus Heinrich Kiwi 2021-07-13 16:35:07 UTC
According to Phil, the Patch series was re-spun, but will need another one

Comment 21 John Ferlan 2021-07-29 10:14:48 UTC
Setting ITM=25 to get release+ since all 3 downstream acks are present - feel free to adjust higher based on your load.

Comment 23 Yanan Fu 2021-07-30 01:38:24 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 27 Tingting Mao 2021-08-03 12:08:02 UTC
Hi Philippe,

Tried to verify this bug as below, but still hit 'qemu-kvm: VFIO_MAP_DMA failed: No space left on device' when installing guest over the NVMe disk with qcow2 format + 20G size + q35. 

And installation hangs when installing of the disk with qcow2 format + 20G size + pc.

While installing guest over the NVMe disk with raw format + 20G size + q35, it works well.

Could you please help to check? Thanks.


Tested with:
qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc
kernel-modules-4.18.0-325.el8.x86_64


Setup NVMe disk:
1. Unbind the host NVMe controller from host
# echo 0000:bc:00.0 > /sys/bus/pci/devices/0000\:bc\:00.0/driver/unbind

2.Bind the host NVMe controller to the host vfio-pci driver
# echo 144d a822 > /sys/bus/pci/drivers/vfio-pci/new_id


Scenario1 - Install guest on the NVMe disk (Qcow2 with 20G size + q35).
1. Create the NVMe disk with qcow2+20G.
# qemu-img create -f qcow2 nvme://0000:bc:00.0/1 20G
# qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

2. Install guest on the disk
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap \
    -device scsi-hd,id=nvme1,drive=drive_nvme1 \
    -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=6 \
    -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-6,addr=0x0 \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.5.0-20210714.n.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \

Result:
Still hit the error hint in HMP when the installation finished and shutdown the guest for first time. And hit the error hint in HMP when rebooting the guest after installation for the second and third time.
# sh qemu-install.sh 
QEMU 6.0.0 monitor - type 'help' for more information
(qemu) c
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device



Scenario2 - Install guest on the NVMe disk (Raw with 20G size)
1. Over-write the image with raw format
# qemu-img create -f raw nvme://0000:bc:00.0/1 20G
# qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: raw
virtual size: 745 GiB (800166076416 bytes)
disk size: unavailable

2. Install guest on it.
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_nvme1,driver=raw,file=nvme_image1,read-only=off,discard=unmap,size=21474836480 \ ----------------------------------------> Specified the size here.
    -device scsi-hd,id=nvme1,drive=drive_nvme1 \
    -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x1.0x6,bus=pcie.0,chassis=6 \
    -device virtio-scsi-pci,id=virtio_scsi_pci2,bus=pcie-root-port-6,addr=0x0 \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.5.0-20210714.n.0-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \


Result:
Tried twice, all installed successfully.



Scenario3 - Install guest on the NVMe disk (Qcow2 with 20G size + pc).
1. Install guest as below.
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2 \
    -m 8G  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0 \
    -device virtio-net-pci,mac=9a:ce:07:93:d2:57,id=idVu6rLy,netdev=idIumYb0,bus=pci.0,addr=0x5  \
    -netdev tap,id=idIumYb0,vhost=on \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.4.0-20210503.1-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,write-cache=on,bus=ide.0,unit=0 \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=d,strict=off  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \

Result:
The installation hang when the process is 439/1374 for my first time test, and when the process is 784/1374 for my second time test.

Comment 30 Philippe Mathieu-Daudé 2021-08-06 14:12:03 UTC
(In reply to Tingting Mao from comment #27)
> Scenario1 - Install guest on the NVMe disk (Qcow2 with 20G size + q35).
> 1. Create the NVMe disk with qcow2+20G.
> Result:
> Still hit the error hint in HMP when the installation finished and shutdown
> the guest for first time. And hit the error hint in HMP when rebooting the
> guest after installation for the second and third time.
> # sh qemu-install.sh 
> QEMU 6.0.0 monitor - type 'help' for more information
> (qemu) c
> (qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device

This is a simple warning, not an error. Now when this happens the driver
flushes all the temporary DMA mappings and keeps going. The installation
should succeed.

> Scenario2 - Install guest on the NVMe disk (Raw with 20G size)
> 1. Over-write the image with raw format
> Result:
> Tried twice, all installed successfully.

Good.

> Scenario3 - Install guest on the NVMe disk (Qcow2 with 20G size + pc).
> 1. Install guest as below.
> Result:
> The installation hang when the process is 439/1374 for my first time test,
> and when the process is 784/1374 for my second time test.

Not even a "qemu-kvm: VFIO_MAP_DMA failed: No space left on device" warning?
This is not good. It could another bug now uncovered.

For the raw vs qcow2 format I'll refer to Stefan, I'm not sure the driver is
ready for non-raw formats.

Comment 31 Stefan Hajnoczi 2021-08-16 14:34:25 UTC
(In reply to Philippe Mathieu-Daudé from comment #30)
> (In reply to Tingting Mao from comment #27)
> > Scenario3 - Install guest on the NVMe disk (Qcow2 with 20G size + pc).
> > 1. Install guest as below.
> > Result:
> > The installation hang when the process is 439/1374 for my first time test,
> > and when the process is 784/1374 for my second time test.
> 
> Not even a "qemu-kvm: VFIO_MAP_DMA failed: No space left on device" warning?
> This is not good. It could another bug now uncovered.
> 
> For the raw vs qcow2 format I'll refer to Stefan, I'm not sure the driver is
> ready for non-raw formats.

After the hang, does the HMP command "info status" show "VM status: running" and the HMP command "info block" show no I/O error status? If yes, then the guest is still running and I can try to debug it with GDB if you can send me connection details to the host machine.

Comment 32 Tingting Mao 2021-08-17 09:28:40 UTC
I would like to confirm firstly that whether the NVMe driver is ready for non-raw formats, like qcow2 or luks now? Or just for raw format only?

Comment 33 Stefan Hajnoczi 2021-08-17 10:27:22 UTC
(In reply to Tingting Mao from comment #32)
> I would like to confirm firstly that whether the NVMe driver is ready for
> non-raw formats, like qcow2 or luks now? Or just for raw format only?

The QEMU NVMe driver was intended to be used primarily with the raw format for performance, so we haven't tested non-raw. In principle the driver should work with non-raw formats though! If you hit issues with non-raw formats then it's worth filing BZes so it can be fixed.

Thanks!

Comment 34 Klaus Heinrich Kiwi 2021-08-17 11:40:57 UTC
Based on the answers from the last few comments, are we ready to move this bug to verified?

Comment 35 Tingting Mao 2021-08-18 08:54:48 UTC
Also tried installation on the NVMe disk (Raw with 20G size + pc) as below, works fine. Agree to set this bug as verified. And as for the issues with non-raw formats, will file new bugs to trace later. Thanks.


Scenario4 - Install guest on the NVMe disk (Raw with 20G size + pc)
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2 \
    -m 8G  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,cache.direct=off,cache.no-flush=off,file=nvme_image1,size=21474836480 \ -----------------------------------------> Define the 20G size here.
    -device virtio-blk-pci,id=image1,drive=drive_image1,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0 \
    -device virtio-net-pci,mac=9a:ce:07:93:d2:57,id=idVu6rLy,netdev=idIumYb0,bus=pci.0,addr=0x5  \
    -netdev tap,id=idIumYb0,vhost=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=d,strict=off  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=native,filename=/home/kvm_autotest_root/iso/linux/RHEL-8.4.0-20210503.1-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,write-cache=on,bus=ide.0,unit=0 \

Results:
Installation successed.

Comment 38 errata-xmlrpc 2021-11-16 07:49:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684


Note You need to log in before you can comment on or make changes to this bug.