Bug 1736792

Summary: QEMU core dumped if boot guest with nvdimm backed by /dev/dax0.0 and option pmem=off
Product: Red Hat Enterprise Linux 8 Reporter: Yumei Huang <yuhuang>
Component: qemu-kvmAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED DEFERRED QA Contact: Yumei Huang <yuhuang>
Severity: high Docs Contact:
Priority: high    
Version: 8.1CC: chayang, juzhang, rbalakri, stefanha, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1736788 Environment:
Last Closed: 2019-08-13 17:50:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1736788    
Bug Blocks: 1736789    

Description Yumei Huang 2019-08-02 02:51:46 UTC
Hit same issue on slow train.

qemu-kvm-2.12.0-82.module+el8.1.0+3738+0d8c0249
kernel-4.18.0-116.el8.x86_64

+++ This bug was initially created as a clone of Bug #1736788 +++

Description of problem:

Boot guest with nvdimm device backed by /dev/dax0.0, set pmem=off, qemu core dumped with 'Bus error' during guest boot.

Version-Release number of selected component (if applicable):
qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3
host kernel: 4.18.0-116.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. Emulate /dev/pmem by adding memmap=4G!4G to host kernel line
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-116.el8.x86_64 root=/dev/mapper/rhel_hp--dl385g10--02-root ro crashkernel=auto resume=/dev/mapper/rhel_hp--dl385g10--02-swap rd.lvm.lv=rhel_hp-dl385g10-02/root rd.lvm.lv=rhel_hp-dl385g10-02/swap console=ttyS0,115200n81 memmap=4G!4G

2. Create /dev/dax0.0 by ndctl

# ndctl create-namespace -m dax -e namespace0.0 -f -v -a 4096
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"4635cd69-c9f1-4886-a9f6-b8f7aa4d4fc2",
  "daxregion":{
    "id":0,
    "size":"3.94 GiB (4.23 GB)",
    "align":4096,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"3.94 GiB (4.23 GB)"
      }
    ]
  },
  "align":4096
}
[root@hp-dl385g10-02 ~]# ndctl list
[
  {
    "dev":"namespace0.0",
    "mode":"devdax",
    "map":"dev",
    "size":4225761280,
    "uuid":"4635cd69-c9f1-4886-a9f6-b8f7aa4d4fc2",
    "chardev":"dax0.0",
    "align":4096
  }
]

3. Boot guest with /dev/dax0.0, set pmem=off

# /usr/libexec/qemu-kvm -M pc,nvdimm   \
-m 1G,slots=256,maxmem=40G  \
-object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=off \
-device nvdimm,memdev=mem2,id=nv2,label-size=2M \
/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \
-monitor stdio -vnc :0

Actual results:
QEMU core dumped.
(qemu) Bus error (core dumped)

Expected results:
No core dump.

Additional info:
1. if set pmem=on, qemu quit and print error message,
qemu-kvm: -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=on: size property 4294967296 is larger than pmem file "/dev/dax0.0" size 4225761280

2. if set object size<4G, e.g. -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=2G,align=128M,pmem=off, guest can boot up well.

3. Backtrace:
(gdb) bt
#0  0x00007fd2f514a97a in pthread_sigmask () at /lib64/libpthread.so.0
#1  0x00005609c99d5540 in sigbus_reraise ()
#2  0x00005609c99d55a3 in  ()
#3  0x00007fd2f514ddc0 in <signal handler called> () at /lib64/libpthread.so.0
#4  0x00007fd2f4ed66a5 in __memmove_avx_unaligned_erms () at /lib64/libc.so.6
#5  0x00005609c9b00f07 in nvdimm_dsm_write ()
#6  0x00005609c99eae03 in memory_region_write_accessor ()
#7  0x00005609c99e8fb6 in access_with_adjusted_size ()
#8  0x00005609c99ece70 in memory_region_dispatch_write ()
#9  0x00005609c9993923 in flatview_write_continue ()
#10 0x00005609c9993b46 in flatview_write ()
#11 0x00005609c9997ccf in address_space_write ()
#12 0x00005609c99ff324 in kvm_cpu_exec ()
#13 0x00005609c99d7316 in qemu_kvm_cpu_thread_fn ()
#14 0x00005609c9cec774 in qemu_thread_start ()
#15 0x00007fd2f51432de in start_thread () at /lib64/libpthread.so.0
#16 0x00007fd2f4e74463 in clone () at /lib64/libc.so.6

Comment 2 Ademar Reis 2019-08-13 17:50:23 UTC
(In reply to Yumei Huang from comment #0)
> Hit same issue on slow train.
> 
> qemu-kvm-2.12.0-82.module+el8.1.0+3738+0d8c0249
> kernel-4.18.0-116.el8.x86_64
> 
> +++ This bug was initially created as a clone of Bug #1736788 +++

...

> 
> 3. Boot guest with /dev/dax0.0, set pmem=off
> 
> # /usr/libexec/qemu-kvm -M pc,nvdimm   \
> -m 1G,slots=256,maxmem=40G  \
> -object
> memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,
> pmem=off \
> -device nvdimm,memdev=mem2,id=nv2,label-size=2M \
> /home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \
> -monitor stdio -vnc :0

The handling of these options in libvirt was introduced in libvirt-5.0, while RHEL is using libvirt-4.5 (RHEL-AV gets rebased, but not RHEL).

Given we don't support direct invocation of qemu in RHEL, I'm closing this BZ. If this can be reproduced using libvirt from RHEL, please reopen.