Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2007932

Summary: Core Dumped When Executing system_reset Repeatedly While The Guest Over NVMe block Is Booting
Product: Red Hat Enterprise Linux 9 Reporter: Tingting Mao <timao>
Component: qemu-kvmAssignee: Stefan Hajnoczi <stefanha>
qemu-kvm sub component: Storage QA Contact: Tingting Mao <timao>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: medium    
Priority: medium CC: coli, eblake, hreitz, jinzhao, juzhang, kkiwi, sgarzare, stefanha, virt-maint, xuwei
Version: 9.0Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2021454 (view as bug list) Environment:
Last Closed: 2022-08-11 08:36:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2021454    
Attachments:
Description Flags
Screenshot of guest after rebooting none

Description Tingting Mao 2021-09-26 09:13:09 UTC 
Description of problem:
As subject.


Version-Release number of selected component (if applicable):
qemu-kvm-6.1.0-2.el9
kernel-5.14.0-0.rc7.54.el9.x86_64


How reproducible:
3/3


Steps to Reproduce:
1. Check the system image file over NVMe(The image was installed on rhel9 OS.)
# qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
 
2. Boot guest from the image file
# cat qemu.sh
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap \
    -device scsi-hd,id=nvme1,drive=drive_nvme1 \
 
3. Execute system_reset via HMP repeatly. If you don't hit core dumped, try more times please.
# sh qemu.sh
QEMU 6.1.0 monitor - type 'help' for more information
(qemu) c
(qemu) sy
sync-profile      system_powerdown  system_reset      system_wakeup    
(qemu) sys
system_powerdown  system_reset      system_wakeup    
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) system_reset
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
 
(qemu)
(qemu)
(qemu) system_reset
(qemu) corrupted size vs. prev_size
qemu.sh: line 33: 19661 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 15360 -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 -cpu 'Haswell-noTSX',+kvm_pv_unhalt -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -object iothread,id=iothread1 -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=idEFQ4i1,vhost=on -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -enable-kvm -monitor stdio -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off -mon chardev=qmp_id_qmpmonitor1,mode=control -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap -device scsi-hd,id=nvme1,drive=drive_nvme1


Actual results:
As above. Qemu core dumped.

Expected results:
Qemu works fine without core dumped.


Additional info:
(gdb) bt
#0  0x00007fc6b488e763 in pthread_kill.5 () from /lib64/libc.so.6
#1  0x00007fc6b4841686 in raise () from /lib64/libc.so.6
#2  0x00007fc6b482b7d3 in abort () from /lib64/libc.so.6
#3  0x00007fc6b4882a07 in __libc_message () from /lib64/libc.so.6
#4  0x00007fc6b489872c in malloc_printerr () from /lib64/libc.so.6
#5  0x00007fc6b48992f6 in unlink_chunk.constprop () from /lib64/libc.so.6
#6  0x00007fc6b489baa9 in _int_malloc () from /lib64/libc.so.6
#7  0x00007fc6b489c1bf in _int_memalign () from /lib64/libc.so.6
#8  0x00007fc6b489c86a in _mid_memalign.constprop.0 () from /lib64/libc.so.6
#9  0x00007fc6b489ddf3 in posix_memalign () from /lib64/libc.so.6
#10 0x0000562a2b261c5c in nvme_co_prw (bs=0x562a2d31f610, offset=4569038848, bytes=4096, qiov=0x7fc2803c3ee0, is_write=false, flags=0) at ../util/oslib-posix.c:210
#11 0x0000562a2b2268b9 in bdrv_driver_preadv (bs=0x562a2d31f610, offset=4569038848, bytes=4096, qiov=0x7fc6b488e763 <pthread_kill.5+67>, qiov_offset=<optimized out>, flags=0) at ../block/io.c:1190
#12 0x0000562a2b225fd9 in bdrv_aligned_preadv (child=0x562a2d330ed0, req=<optimized out>, offset=<optimized out>, bytes=<optimized out>, align=<optimized out>, qiov=<optimized out>, qiov_offset=0, flags=0)
    at ../block/io.c:1577
#13 0x0000562a2b22561f in bdrv_co_preadv_part (child=<optimized out>, offset=<optimized out>, bytes=<optimized out>, qiov=<optimized out>, qiov_offset=<optimized out>, flags=<optimized out>)
    at ../block/io.c:1848
#14 0x0000562a2b236a49 in qcow2_co_preadv_task (bs=0x562a2d332050, subc_type=<optimized out>, host_offset=4569038848, offset=<optimized out>, bytes=<optimized out>, qiov=<optimized out>, qiov_offset=64512)
    at ../block/qcow2.c:2297
#15 qcow2_co_preadv_task_entry (task=<optimized out>) at ../block/qcow2.c:2313
#16 0x0000562a2b29a085 in aio_task_co (opaque=0x7fc6a0056f90) at ../block/aio_task.c:45
#17 0x0000562a2b3c3026 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at ../util/coroutine-ucontext.c:173
#18 0x00007fc6b4856820 in ?? () from /lib64/libc.so.6
#19 0x00007fc6b0e2e940 in ?? ()
#20 0x0000000000000000 in ?? ()
Comment 1 Klaus Heinrich Kiwi 2021-09-28 11:39:30 UTC 
Philippe, can you take this one? It's a high-priority crash.
Comment 2 Philippe Mathieu-Daudé 2021-09-28 17:17:46 UTC 
(In reply to Tingting Mao from comment #0)
> 1. Check the system image file over NVMe(The image was installed on rhel9
> OS.)
> # qemu-img info nvme://0000:bc:00.0/1
> image: nvme://0000:bc:00.0/1
> file format: qcow2
> virtual size: 20 GiB (21474836480 bytes)
> disk size: unavailable
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
>     extended l2: false

How was this created / formated?

> 2. Boot guest from the image file
> # cat qemu.sh
> /usr/libexec/qemu-kvm \
...
>     -blockdev
> node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-
> only=on,discard=unmap \
>     -blockdev
> node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,
> discard=unmap \

> (qemu) system_reset
> (qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
>  
> (qemu)
> (qemu)
> (qemu) system_reset
> (qemu) corrupted size vs. prev_size

So we have a memory corruption (likely) or double free (unlikely IMO).

What is interesting is this happens *after* exhausting the IOMMU DMAs.
Comment 3 Tingting Mao 2021-09-29 03:02:31 UTC 
(In reply to Philippe Mathieu-Daudé from comment #2)
> (In reply to Tingting Mao from comment #0)
> > 1. Check the system image file over NVMe(The image was installed on rhel9
> > OS.)
> > # qemu-img info nvme://0000:bc:00.0/1
> > image: nvme://0000:bc:00.0/1
> > file format: qcow2
> > virtual size: 20 GiB (21474836480 bytes)
> > disk size: unavailable
> > cluster_size: 65536
> > Format specific information:
> >     compat: 1.1
> >     compression type: zlib
> >     lazy refcounts: false
> >     refcount bits: 16
> >     corrupt: false
> >     extended l2: false
> 
> How was this created / formated?
Before formated, the image is:
# qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: raw
virtual size: 745 GiB (800166076416 bytes)
disk size: unavailable


The image is formated by:
# qemu-img create -f qcow2 nvme://0000:bc:00.0/1 20G
Formatting 'nvme://0000:bc:00.0/1', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16

# qemu-img info nvme://0000:bc:00.0/1
image: nvme://0000:bc:00.0/1
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false


> > 2. Boot guest from the image file
> > # cat qemu.sh
> > /usr/libexec/qemu-kvm \
> ...
> >     -blockdev
> > node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-
> > only=on,discard=unmap \
> >     -blockdev
> > node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,
> > discard=unmap \
> 
> > (qemu) system_reset
> > (qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
> >  
> > (qemu)
> > (qemu)
> > (qemu) system_reset
> > (qemu) corrupted size vs. prev_size
> 
> So we have a memory corruption (likely) or double free (unlikely IMO).
> 
> What is interesting is this happens *after* exhausting the IOMMU DMAs.
Comment 4 Philippe Mathieu-Daudé 2021-09-29 18:12:49 UTC 
I opened an upstream ticket to track this bug with a broader audience: https://gitlab.com/qemu-project/qemu/-/issues/647
Comment 5 Tingting Mao 2021-11-09 10:01:32 UTC 
Still hit the issue in rhel8.6.


Tested with:
qemu-kvm-6.1.50-4.scrmod+el8.6.0+13148+60ec5265.wrb211103
kernel-4.18.0-348.4.el8.kpq0.x86_64


Steps:
Boot guest from the NVMe image installed well, and execute system_reset in HMP repeatly.
# sh qemu.sh 
QEMU 6.1.50 monitor - type 'help' for more information
(qemu) cont
(qemu) sys
system_powerdown  system_reset      system_wakeup     
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) free(): invalid size
qemu.sh: line 33:  5258 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 15360 -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 -cpu 'Haswell-noTSX',+kvm_pv_unhalt -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -object iothread,id=iothread1 -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=idEFQ4i1,vhost=on -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -enable-kvm -monitor stdio -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off -mon chardev=qmp_id_qmpmonitor1,mode=control -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap -device scsi-hd,id=nvme1,drive=drive_nvme1

Note:
The CML to boot:
# cat qemu.sh 
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1c:0c:0d:e3:4c,id=idjmZXQS,netdev=idEFQ4i1,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEFQ4i1,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -chardev socket,server=on,path=/var/tmp/monitor-qmpmonitor1-20210721-024113-AsZ7KYro,id=qmp_id_qmpmonitor1,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0,iothread=iothread1 \
    -blockdev node-name=nvme_image1,driver=nvme,device=0000:bc:00.0,namespace=1,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_nvme1,driver=qcow2,file=nvme_image1,read-only=off,discard=unmap \
    -device scsi-hd,id=nvme1,drive=drive_nvme1 \
Comment 9 Philippe Mathieu-Daudé 2021-12-20 22:11:24 UTC 
(In reply to Tingting Mao from comment #5)
> Steps:
> Boot guest from the NVMe image installed well, and execute system_reset in
> HMP repeatly.
> # sh qemu.sh 
> QEMU 6.1.50 monitor - type 'help' for more information
> (qemu) cont
> (qemu) sys
> system_powerdown  system_reset      system_wakeup     
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) free(): invalid size
> qemu.sh: line 33:  5258 Aborted                 (core dumped)

Can you run with '-D trace.log -trace nvme\*' and attach the resulting trace.log file?
Comment 10 Tingting Mao 2021-12-23 07:41:25 UTC 
Tried several times in in latest qemu, still fails to install but the results are a little different.


Tested with:
qemu-kvm-6.2.0-1.el9
kernel-5.14.0-22.el9.x86_64


Steps:
1. Create qcow2 image over NVMe
# qemu-img create -f qcow2 nvme://0000:06:00.0/1 20G

2. Install guest on the image
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idzArAFk \
    -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=unmap,device=0000:06:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=off \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idwZT70w,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20211221.5-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -enable-kvm \
    -monitor stdio \


Results:
After installation, restart the guest, the guest can not reboot and there is "uncompression error\n -- System halted" error hint info in the guest screen. And reset the guest in HMP repeatedly, the qemu core dumped.
(qemu) c  
(qemu) system_reset 
(qemu) 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) q
corrupted double-linked list
qemu.sh: line 41: 375111 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object memory-backend-ram,size=30720M,id=mem-machine_mem -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idzArAFk -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=unmap,device=0000:06:00.0,namespace=1,cache.direct=off,cache.no-flush=off -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=off -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=idwZT70w,vhost=on -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20211221.5-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on -vnc :0 -rtc base=utc,clock=host,driftfix=slew -enable-kvm -monitor stdio


Note:
Sometimes, the process gets hang during the installation and below are the output info in HMP
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) c
(qemu) qcow2: Marking image as corrupt: Cluster allocation offset 0x14ff9e45e00 unaligned (L2 offset: 0x390000, L2 index: 0x120e); further corruption events will be suppressed
Comment 13 Philippe Mathieu-Daudé 2021-12-30 22:06:46 UTC 
(In reply to Tingting Mao from comment #10)

> 2. Install guest on the image
> /usr/libexec/qemu-kvm \
...
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
>     -blockdev
> node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=unmap,
> device=0000:06:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
>     -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=off,cache.no-
> flush=off,file=nvme_image1 \
>     -device scsi-hd,id=image1,drive=drive_image1,write-cache=off \
>     -device
> pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,
> chassis=4 \
>     -device
> virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-
> root-port-3,addr=0x0  \
>     -netdev tap,id=idwZT70w,vhost=on \
>     -blockdev
> node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,
> filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20211221.5-x86_64-dvd1.
> iso,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-
> flush=off,file=file_cd1 \
>     -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
...
> 
> Results:
> After installation, restart the guest, the guest can not reboot and there is
> "uncompression error\n -- System halted" error hint info in the guest
> screen. And reset the guest in HMP repeatedly, the qemu core dumped.
> (qemu) c  
> (qemu) system_reset 
> (qemu) system_reset 
> (qemu) q
> corrupted double-linked list

This is likely an effect of what follows:

> Note:
> Sometimes, the process gets hang during the installation and below are the
> output info in HMP
> QEMU 6.2.0 monitor - type 'help' for more information
> (qemu) c
> (qemu) qcow2: Marking image as corrupt: Cluster allocation offset
> 0x14ff9e45e00 unaligned (L2 offset: 0x390000, L2 index: 0x120e); further
> corruption events will be suppressed

Cc'ing Hanna and Eric for this error.
Comment 15 Klaus Heinrich Kiwi 2022-01-03 14:00:55 UTC 
Hanna, can you take this one?

Also, submitter, any word on what version did this same test last succeed? Please set the regression flag if this is indeed a regression.

Thanks,

 -Klaus
Comment 17 Hanna Czenczek 2022-01-06 10:57:42 UTC 
(In reply to Klaus Heinrich Kiwi from comment #15)
> Hanna, can you take this one?
I wonder about that just as much as you do: I have virtually no experience with the nvme block driver, and I also have no hardware on which I could test this.  (Phil seemed to have been in a much better position, but still couldn’t track this bug down in three months.)

Cc-ing Stefan, as he seems to have looked at some recent block/nvme.c patches.


So here are all my thoughts off of just the reports in this BZ:

I agree with Phil that memory corruption seems to be the underlying problem.

A qcow2 corruption was reported, but (1) qcow2 corruptions are notoriously difficult to track down, and (2) since we’re talking about a memory corruption, it isn’t difficult to imagine how that could cause corruption of the qcow2 file.

In comment 13, Phil seems to imply this scenario:
(A) The qcow2 file becomes corrupted (probably due to a bug in the qcow2 driver?), and then this file corruption causes memory corruption in the qcow2 driver.

I find this unlikely, because it implies the existence of two separate bugs: One where the qcow2 driver produces a corrupted qcow2 files, and another where the qcow2 driver cannot gracefully handle corrupted qcow2 files.
(Of note is that AFAIK there is some known bug in qcow2 when it comes to rewriting discarded clusters, but that isn’t known to cause corruption, and certainly not memory corruption.)

Also, (without further information) the crash actually cannot be the result of the “Marking image as corrupt” event.  When an image is marked corrupt, it is effectively closed (by setting bs->drv to NULL, so that no further driver callbacks can be invoked, which makes the node unusable) and when you try to reopen it, R/W access will not be allowed (until `qemu-img check -r all` is run).  Therefore, installation cannot complete, and from the description “After installation, restart the guest” I take it that the crash only affects cases where installation was completed.
(It is entirely imaginable that said setting bs->drv to NULL would cause a crash across multiple reboots, but then the “Marking image as corrupt” event would need to precede every single crash, which it doesn’t.)

On top, a memory corruption can manifest itself in various ways: For example with an assertion error in the memory management code (as originally reported), or simply by abnormal behavior of the program (e.g. writing corrupted data).  I don’t find it hard to imagine how a memory corruption bug somewhere could sometimes manifest itself in a crash, and sometimes in a corrupted qcow2 file.

So I propose other scenarios:
(B) The qcow2 driver has a memory corruption bug, which can sometimes manifest itself in qcow2 file corruption, and sometimes in an assertion failure in the memory management code.
(C) The nvme driver has a memory corruption bug, which can sometimes manifest itself in corrupted data being written, and sometimes in an assertion failure in the memory management code.

I find (C) more likely than (B), because:
(1) Personal bias to blame unknown code (bad reason, I know, but you should know I have this bias)
(2) I feel like the qcow2 driver is generally better tested than the nvme driver
(3) In both cases you need an explanation why the nvme+qcow2 combination fails, but not (B) file-posix+qcow2 or (C) nvme+raw.  Since I know nothing about the nvme code, I may be very wrong, but I feel like from qcow2’s perspective, the nvme driver should not behave very differently from the file-posix driver, so I find it difficult to imagine why a memory corruption (and file corruption) caused by the qcow2 driver would only show up with the nvme driver, but not with file-posix.  On the other hand, it is clear why the nvme+raw combination will not produce file corruption events, because there is no metadata, so you will not get any error from qemu if corrupted data is written.  Also, the qcow2 block driver behaves much differently from the raw block driver, which could make bugs in the nvme code visible that do not appear when using a raw format.


Obviously, all of this is conjecture, but I don’t know what more I can do, other than perhaps inspect all of the nvme block driver code.
Comment 18 Hanna Czenczek 2022-01-06 11:01:19 UTC 
(In reply to Hanna Reitz from comment #17)
> (Of note is that AFAIK there is some known bug in qcow2 when it comes to
> rewriting discarded clusters, but that isn’t known to cause corruption, and
> certainly not memory corruption.)

Hi,

Can you test whether the bug also appears with discard=ignore instead of discard=unmap?


Thanks!

Hanna
Comment 19 Tingting Mao 2022-01-10 11:17:39 UTC 
Created attachment 1849818 [details]
Screenshot of guest after rebooting

Hi Hanna,

Still hit the issue with discard=ignore.


Tested with:
qemu-kvm-6.2.0-2.el9
kernel-5.14.0-39.el9.x86_64


Steps:
1. Install guest via qemu-kvm
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idzArAFk \
    -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=ignore,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=off \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idwZT70w,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=ignore,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20220103.2-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -enable-kvm \
    -monitor stdio \

2. Reboot after installion
### Clicked the 'reboot' bottum in the guest screen


Results:
After step2, the guest can not reboot and there is "uncompression error\n -- System halted" error hint info in the guest screen as the attachment.

Plus, I tried execute system_reset via HMP as below. The host is broken and can not ssh/ping to. 
(qemu) system_reset 
(qemu) system_reset 
(qemu) system_reset 
(qemu) q
Comment 20 Tingting Mao 2022-01-10 12:13:48 UTC 
While tried with luks format, hit qemu core dumped.


Tested with:
qemu-kvm-6.2.0-2.el9
kernel-5.14.0-39.el9.x86_64


Steps:
1. Create luks image over NVMe block
# qemu-img create -f luks --object secret,id=sec0,data=redhat nvme://0000:bc:00.0/1 -o key-secret=sec0 20G

2. Install guest over the luks image
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idzArAFk \
    -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object secret,id=sec0,data=redhat \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=ignore,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=luks,key-secret=sec0,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=off \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idwZT70w,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=ignore,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20220103.2-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -enable-kvm \
    -monitor stdio \


Results:
Hit the qemu coredumped during the installation process.
(qemu) corrupted size vs. prev_size
qemuluks.sh: line 42:  1695 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object memory-backend-ram,size=30720M,id=mem-machine_mem -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idzArAFk -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object secret,id=sec0,data=redhat -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=ignore,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off -blockdev node-name=drive_image1,driver=luks,key-secret=sec0,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=off -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=idwZT70w,vhost=on -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=ignore,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20220103.2-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on -vnc :0 -rtc base=utc,clock=host,driftfix=slew -enable-kvm -monitor stdio


Additional info:
(gdb) bt
#0  0x00007fd94eaa07fc in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fd94ea53676 in raise () from /lib64/libc.so.6
#2  0x00007fd94ea3d7d3 in abort () from /lib64/libc.so.6
#3  0x00007fd94ea949d7 in __libc_message () from /lib64/libc.so.6
#4  0x00007fd94eaaa7ec in malloc_printerr () from /lib64/libc.so.6
#5  0x00007fd94eaab3b6 in unlink_chunk.constprop () from /lib64/libc.so.6
#6  0x00007fd94eaadcd1 in _int_malloc () from /lib64/libc.so.6
#7  0x00007fd94eaae26f in _int_memalign () from /lib64/libc.so.6
#8  0x00007fd94eaae91a in _mid_memalign.constprop.0 () from /lib64/libc.so.6
#9  0x00007fd94eaafea3 in posix_memalign () from /lib64/libc.so.6
#10 0x000055fb9fc7e325 in qemu_try_blockalign (bs=<optimized out>, size=<optimized out>) at ../util/oslib-posix.c:210
#11 0x000055fb9fc71520 in block_crypto_co_pwritev (bs=<optimized out>, offset=68300800, bytes=131072, qiov=0x55fba2c0d460, flags=BDRV_REQ_FUA) at ../block/crypto.c:486
#12 0x000055fb9fc7a51d in bdrv_driver_pwritev (bs=0x55fba1bee050, offset=<optimized out>, bytes=131072, qiov=0x55fba2c0d460, qiov_offset=<optimized out>, flags=BDRV_REQ_FUA) at ../block/io.c:1264
#13 0x000055fb9fc7bdb0 in bdrv_aligned_pwritev (child=0x55fba3018e10, req=0x7fd1305e5f98, offset=68300800, bytes=<optimized out>, align=<optimized out>, qiov=0x55fba2c0d460, qiov_offset=0, 
    flags=<optimized out>) at ../block/io.c:2126
#14 0x000055fb9fc7b1a3 in bdrv_co_pwritev_part (child=<optimized out>, offset=<optimized out>, bytes=<optimized out>, qiov=<optimized out>, qiov_offset=<optimized out>, flags=<optimized out>)
    at ../block/io.c:2314
#15 0x000055fb9fc66db2 in blk_co_do_pwritev_part (blk=0x55fba303fb10, offset=68300800, bytes=131072, qiov=0x7fd94eaa07fc <__pthread_kill_implementation+284>, qiov_offset=140536437434576, flags=0)
    at ../block/block-backend.c:1283
#16 0x000055fb9fc67227 in blk_aio_write_entry (opaque=0x55fba2da8ca0) at ../block/block-backend.c:1467
#17 0x000055fb9fe22a66 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at ../util/coroutine-ucontext.c:173
#18 0x00007fd94ea68810 in ?? () from /lib64/libc.so.6
#19 0x00007fd94da3e570 in ?? ()
#20 0x0000000000000000 in ?? ()


The complete dump info:
http://fileshare.englab.nay.redhat.com/pub/section2/kvm/timao/bugs/2007932/core.qemu-kvm.0.7a5c6fa7bbbc4120bce24d71fc21d37a.1695.1641816146000000.zst
Comment 21 Stefan Hajnoczi 2022-01-10 13:16:19 UTC 
(In reply to Tingting Mao from comment #20)
> While tried with luks format, hit qemu core dumped.

Please try this modified qemu-kvm-6.1.0.el9 RPM that has AddressSanitizer enabled. I hope it will catch the heap corruption and making finding the root cause easier:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42297678

No special command-line options are necessary, just run the new qemu-kvm executable instead of the old one. The AddressSanitizer documentation is here:
https://clang.llvm.org/docs/AddressSanitizer.html

If a memory corruption is found the qemu-kvm process will exit and a report will be printed.
Comment 22 Tingting Mao 2022-01-11 07:52:39 UTC 
Hi Stefan,

Tried with the qemu packages of https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42297678, there are some error info printed in the screen but can go on executing. And the result is listed below.


Tested with:
qemu-kvm-6.1.0-2.el9.coroutine_pool_timer
kernel-5.14.0-39.el9.x86_64


Steps:
1. Create luks image over NVMe.
# qemu-img create -f luks --object secret,id=sec0,data=redhat nvme://0000:bc:00.0/1 -o key-secret=sec0 20G
Formatting 'nvme://0000:bc:00.0/1', fmt=luks size=21474836480 key-secret=sec0
../util/vfio-helpers.c:279:41: runtime error: member access within misaligned address 0x5651733190b4 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x5651733190b4: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:279:41: runtime error: load of misaligned address 0x5651733190bc for type '__u32' (aka 'unsigned int'), which requires 8 byte alignment
0x5651733190bc: note: pointer points here
  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:287:58: runtime error: member access within misaligned address 0x5651733190b4 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x5651733190b4: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:58 in 
../util/vfio-helpers.c:287:42: runtime error: member access within misaligned address 0x5651733190c4 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x5651733190c4: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:42 in 
../util/vfio-helpers.c:287:73: runtime error: load of misaligned address 0x5651733190c4 for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x5651733190c4: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:73 in 
../util/vfio-helpers.c:288:56: runtime error: member access within misaligned address 0x5651733190b4 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x5651733190b4: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:56 in 
../util/vfio-helpers.c:288:40: runtime error: member access within misaligned address 0x5651733190c4 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x5651733190c4: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:40 in 
../util/vfio-helpers.c:288:71: runtime error: load of misaligned address 0x5651733190cc for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x5651733190cc: note: pointer points here
  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff  ff ff 00 00 00 00 00 00
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:71 in 

2. Install guest with below CML
# /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,server=on,path=/tmp/avocado_chdzoghs/monitor-qmpmonitor1-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=qmp_id_catch_monitor,server=on,path=/tmp/avocado_chdzoghs/monitor-catch_monitor-20211222-043056-XUr3fyof,wait=off  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idzArAFk \
    -chardev socket,id=chardev_serial0,server=on,path=/tmp/avocado_chdzoghs/serial-serial0-20211222-043056-XUr3fyof,wait=off \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211222-043056-XUr3fyof,path=/tmp/avocado_chdzoghs/seabios-20211222-043056-XUr3fyof,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211222-043056-XUr3fyof,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object secret,id=sec0,data=redhat \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=nvme_image1,driver=nvme,auto-read-only=on,discard=ignore,device=0000:bc:00.0,namespace=1,cache.direct=off,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=luks,key-secret=sec0,read-only=off,cache.direct=off,cache.no-flush=off,file=nvme_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=off \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:4a:05:2e:bd:42,id=idleb5Da,netdev=idwZT70w,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idwZT70w,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=ignore,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL-9.0.0-20220103.2-x86_64-dvd1.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -enable-kvm \
    -monitor stdio \


Results:
For first time, did not hit the abort of qemu. But the guest can not reboot after installation, and the screenshot of the guest is the same as the attachment in Comment19.
../util/vfio-helpers.c:279:41: runtime error: member access within misaligned address 0x557aad031924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x557aad031924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:279:41: runtime error: load of misaligned address 0x557aad03192c for type '__u32' (aka 'unsigned int'), which requires 8 byte alignment
0x557aad03192c: note: pointer points here
  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:287:58: runtime error: member access within misaligned address 0x557aad031924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x557aad031924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:58 in 
../util/vfio-helpers.c:287:42: runtime error: member access within misaligned address 0x557aad031934 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x557aad031934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:42 in 
../util/vfio-helpers.c:287:73: runtime error: load of misaligned address 0x557aad031934 for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x557aad031934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:73 in 
../util/vfio-helpers.c:288:56: runtime error: member access within misaligned address 0x557aad031924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x557aad031924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:56 in 
../util/vfio-helpers.c:288:40: runtime error: member access within misaligned address 0x557aad031934 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x557aad031934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:40 in 
../util/vfio-helpers.c:288:71: runtime error: load of misaligned address 0x557aad03193c for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x557aad03193c: note: pointer points here
  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff  ff ff 00 00 00 00 00 00
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:71 in 
QEMU 6.1.0 monitor - type 'help' for more information
(qemu) 
(qemu) 
(qemu) c
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device

(qemu) 
(qemu) 


For the second time, hit the abort of qemu as below.
../util/vfio-helpers.c:279:41: runtime error: member access within misaligned address 0x55b8d1fa1924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x55b8d1fa1924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:279:41: runtime error: load of misaligned address 0x55b8d1fa192c for type '__u32' (aka 'unsigned int'), which requires 8 byte alignment
0x55b8d1fa192c: note: pointer points here
  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:279:41 in 
../util/vfio-helpers.c:287:58: runtime error: member access within misaligned address 0x55b8d1fa1924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x55b8d1fa1924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:58 in 
../util/vfio-helpers.c:287:42: runtime error: member access within misaligned address 0x55b8d1fa1934 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x55b8d1fa1934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:42 in 
../util/vfio-helpers.c:287:73: runtime error: load of misaligned address 0x55b8d1fa1934 for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x55b8d1fa1934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:287:73 in 
../util/vfio-helpers.c:288:56: runtime error: member access within misaligned address 0x55b8d1fa1924 for type 'struct vfio_iommu_type1_info_cap_iova_range', which requires 8 byte alignment
0x55b8d1fa1924: note: pointer points here
  ff ff 00 00 01 00 01 00  00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:56 in 
../util/vfio-helpers.c:288:40: runtime error: member access within misaligned address 0x55b8d1fa1934 for type 'struct vfio_iova_range', which requires 8 byte alignment
0x55b8d1fa1934: note: pointer points here
  00 00 00 00 00 00 00 00  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:40 in 
../util/vfio-helpers.c:288:71: runtime error: load of misaligned address 0x55b8d1fa193c for type '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x55b8d1fa193c: note: pointer points here
  00 00 00 00 ff ff df fe  00 00 00 00 00 00 f0 fe  00 00 00 00 ff ff ff ff  ff ff 00 00 00 00 00 00
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../util/vfio-helpers.c:288:71 in 
QEMU 6.1.0 monitor - type 'help' for more information
(qemu) c
(qemu) qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device
qemu-kvm: VFIO_MAP_DMA failed: No space left on device

(qemu) UndefinedBehaviorSanitizer:DEADLYSIGNAL
==30662==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address (pc 0x7f081d944cba bp 0x7f081daa2b00 sp 0x7efff2fafad0 T30662)
==30662==The signal is caused by a READ memory access.
==30662==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x7f081d944cba  (/lib64/libc.so.6+0x9ccba)
    #1 0x7f081d94526e  (/lib64/libc.so.6+0x9d26e)
    #2 0x7f081d945919  (/lib64/libc.so.6+0x9d919)
    #3 0x7f081d946ea2  (/lib64/libc.so.6+0x9eea2)
    #4 0x55b8cf645a31  (/usr/libexec/qemu-kvm+0x1b1ba31)
    #5 0x55b8cf371c85  (/usr/libexec/qemu-kvm+0x1847c85)
    #6 0x55b8cf387246  (/usr/libexec/qemu-kvm+0x185d246)
    #7 0x55b8cf38adcb  (/usr/libexec/qemu-kvm+0x1860dcb)
    #8 0x55b8cf389330  (/usr/libexec/qemu-kvm+0x185f330)
    #9 0x55b8cf35a419  (/usr/libexec/qemu-kvm+0x1830419)
    #10 0x55b8cf35b312  (/usr/libexec/qemu-kvm+0x1831312)
    #11 0x55b8cf67b421  (/usr/libexec/qemu-kvm+0x1b51421)
    #12 0x7f081d8ff80f  (/lib64/libc.so.6+0x5780f)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/lib64/libc.so.6+0x9ccba) 
==30662==ABORTING


Note:
For the "qemu-kvm: VFIO_MAP_DMA failed: No space left on device" is fixed in qemu6.2 by Bug #1998027.
Comment 23 Stefan Hajnoczi 2022-01-17 09:50:11 UTC 
The unaligned access warnings are not related to this bug.

It looks like the heap corruption was not detected by the sanitizer. The SIGSEGV that was caught is too late (I looked up the addresses and they are in a posix_memalign() call).

I think this is likely a bug in nvme_cmd_map_qiov(). I found at least one issue when inspecting the code, it could cause memory corruption. I'll send a fix and post a build that you can test.
Comment 30 Stefan Hajnoczi 2022-02-02 12:23:25 UTC 
I agreed with Hanna that I'll work on this bug.
Comment 31 Stefan Hajnoczi 2022-02-03 15:55:34 UTC 
I was unable to reproduce this with qemu-kvm-6.2.0-4.el9 and qemu.git/master.

Please provide access to a machine where I can investigate this bug. Thank you!
Comment 40 Stefan Hajnoczi 2022-05-17 12:24:29 UTC 
Setting ITR to '---'. Let's postpone this for now because the QEMU userspace NVMe is low priority.
Comment 41 Tingting Mao 2022-08-11 08:36:34 UTC 
Not hit the issue in latest rhel9, so close it now.

Tested with:
qemu-kvm-7.0.0-9.el9