Bug 1736788 - QEMU core dumped if boot guest with nvdimm backed by /dev/dax0.0 and option pmem=off
Summary: QEMU core dumped if boot guest with nvdimm backed by /dev/dax0.0 and option p...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Stefan Hajnoczi
QA Contact: Yumei Huang
URL:
Whiteboard:
: 1736789 (view as bug list)
Depends On:
Blocks: 1736789 1736792
TreeView+ depends on / blocked
 
Reported: 2019-08-02 02:32 UTC by Yumei Huang
Modified: 2020-01-20 05:50 UTC (History)
4 users (show)

Fixed In Version: qemu-kvm-4.1.0-11.module+el8.1.0+4250+4f5fbfdc
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1736789 1736792 (view as bug list)
Environment:
Last Closed: 2019-11-06 07:18:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:18:45 UTC

Description Yumei Huang 2019-08-02 02:32:45 UTC
Description of problem:

Boot guest with nvdimm device backed by /dev/dax0.0, set pmem=off, qemu core dumped with 'Bus error' during guest boot.

Version-Release number of selected component (if applicable):
qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3
host kernel: 4.18.0-116.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. Emulate /dev/pmem by adding memmap=4G!4G to host kernel line
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-116.el8.x86_64 root=/dev/mapper/rhel_hp--dl385g10--02-root ro crashkernel=auto resume=/dev/mapper/rhel_hp--dl385g10--02-swap rd.lvm.lv=rhel_hp-dl385g10-02/root rd.lvm.lv=rhel_hp-dl385g10-02/swap console=ttyS0,115200n81 memmap=4G!4G

2. Create /dev/dax0.0 by ndctl

# ndctl create-namespace -m dax -e namespace0.0 -f -v -a 4096
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"4635cd69-c9f1-4886-a9f6-b8f7aa4d4fc2",
  "daxregion":{
    "id":0,
    "size":"3.94 GiB (4.23 GB)",
    "align":4096,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"3.94 GiB (4.23 GB)"
      }
    ]
  },
  "align":4096
}
[root@hp-dl385g10-02 ~]# ndctl list
[
  {
    "dev":"namespace0.0",
    "mode":"devdax",
    "map":"dev",
    "size":4225761280,
    "uuid":"4635cd69-c9f1-4886-a9f6-b8f7aa4d4fc2",
    "chardev":"dax0.0",
    "align":4096
  }
]

3. Boot guest with /dev/dax0.0, set pmem=off

# /usr/libexec/qemu-kvm -M pc,nvdimm   \
-m 1G,slots=256,maxmem=40G  \
-object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=off \
-device nvdimm,memdev=mem2,id=nv2,label-size=2M \
/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \
-monitor stdio -vnc :0

Actual results:
QEMU core dumped.
(qemu) Bus error (core dumped)

Expected results:
No core dump.

Additional info:
1. if set pmem=on, qemu quit and print error message,
qemu-kvm: -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=on: size property 4294967296 is larger than pmem file "/dev/dax0.0" size 4225761280

2. if set object size<4G, e.g. -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=2G,align=128M,pmem=off, guest can boot up well.

3. Backtrace:
(gdb) bt
#0  0x00007fd2f514a97a in pthread_sigmask () at /lib64/libpthread.so.0
#1  0x00005609c99d5540 in sigbus_reraise ()
#2  0x00005609c99d55a3 in  ()
#3  0x00007fd2f514ddc0 in <signal handler called> () at /lib64/libpthread.so.0
#4  0x00007fd2f4ed66a5 in __memmove_avx_unaligned_erms () at /lib64/libc.so.6
#5  0x00005609c9b00f07 in nvdimm_dsm_write ()
#6  0x00005609c99eae03 in memory_region_write_accessor ()
#7  0x00005609c99e8fb6 in access_with_adjusted_size ()
#8  0x00005609c99ece70 in memory_region_dispatch_write ()
#9  0x00005609c9993923 in flatview_write_continue ()
#10 0x00005609c9993b46 in flatview_write ()
#11 0x00005609c9997ccf in address_space_write ()
#12 0x00005609c99ff324 in kvm_cpu_exec ()
#13 0x00005609c99d7316 in qemu_kvm_cpu_thread_fn ()
#14 0x00005609c9cec774 in qemu_thread_start ()
#15 0x00007fd2f51432de in start_thread () at /lib64/libpthread.so.0
#16 0x00007fd2f4e74463 in clone () at /lib64/libc.so.6

Comment 1 Yumei Huang 2019-08-02 06:33:14 UTC
Add full backtrace:

(gdb) bt full
#0  0x00007f898f68797a in pthread_sigmask () at /lib64/libpthread.so.0
#1  0x000055a5a2cb9540 in sigbus_reraise ()
    at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/cpus.c:1125
        set = {__val = {64, 0 <repeats 15 times>}}
        action = 
          {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
#2  0x000055a5a2cb95a3 in sigbus_handler
    (n=<optimized out>, siginfo=<optimized out>, ctx=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/cpus.c:1134
#3  0x00007f898f68adc0 in <signal handler called> () at /lib64/libpthread.so.0
#4  0x00007f898f413600 in __memmove_avx_unaligned_erms () at /lib64/libc.so.6
#5  0x000055a5a2de4f07 in nvdimm_dsm_get_label_data
    (dsm_mem_addr=1073737728, in=0x7f887021c1f0, nvdimm=0x55a5a60377a0) at hw/acpi/nvdimm.c:765
        nvc = 0x55a5a5367e20
        get_label_data = 0x7f887021c1fc
        status = 0
        get_label_data_out = 0x7f88702e7000
        size = 4084
        __func__ = "nvdimm_dsm_get_label_data"
        nvdimm = 0x55a5a60377a0
        state = <optimized out>
        in = 0x7f887021c1f0
        dsm_mem_addr = 1073737728
#6  0x000055a5a2de4f07 in nvdimm_dsm_device (dsm_mem_addr=1073737728, in=0x7f887021c1f0)
    at hw/acpi/nvdimm.c:841
        nvdimm = 0x55a5a60377a0
        state = <optimized out>
        in = 0x7f887021c1f0
        dsm_mem_addr = 1073737728
#7  0x000055a5a2de4f07 in nvdimm_dsm_write
    (opaque=<optimized out>, addr=<optimized out>, val=1073737728, size=<optimized out>)
    at hw/acpi/nvdimm.c:905
        state = <optimized out>
        in = 0x7f887021c1f0
--Type <RET> for more, q to quit, c to continue without paging--c
        dsm_mem_addr = 1073737728
#8  0x000055a5a2ccee03 in memory_region_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/memory.c:502
        tmp = <optimized out>
#9  0x000055a5a2cccfb6 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7f8983408548, size=size@entry=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access_fn=access_fn@entry=0x55a5a2ccedb0 <memory_region_write_accessor>, mr=0x55a5a53ace80, attrs=...) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/memory.c:568
        access_mask = 4294967295
        access_size = 4
        i = <optimized out>
        r = 0
#10 0x000055a5a2cd0e70 in memory_region_dispatch_write (mr=0x55a5a53ace80, addr=0, data=<optimized out>, size=4, attrs=...) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/memory.c:1496
#11 0x000055a5a2c77923 in flatview_write_continue (fv=0x7f88707f2560, addr=2584, attrs=..., buf=0x7f8994ab5000 <error: Cannot access memory at address 0x7f8994ab5000>, len=4, addr1=<optimized out>, l=<optimized out>, mr=0x55a5a53ace80) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/exec.c:3279
        ptr = <optimized out>
        val = <optimized out>
        result = 0
        release_lock = true
#12 0x000055a5a2c77b46 in flatview_write (fv=0x7f88707f2560, addr=2584, attrs=..., buf=0x7f8994ab5000 <error: Cannot access memory at address 0x7f8994ab5000>, len=4) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/exec.c:3318
        l = 4
        addr1 = 0
        mr = <optimized out>
        result = 0
#13 0x000055a5a2c7bccf in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/exec.c:3408
        result = 0
        fv = <optimized out>
#14 0x000055a5a2ce3324 in kvm_handle_io (count=1, size=4, direction=<optimized out>, data=<optimized out>, attrs=..., port=2584) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/accel/kvm/kvm-all.c:1790
        i = 0
        ptr = <optimized out>
        attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#15 0x000055a5a2ce3324 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/accel/kvm/kvm-all.c:2036
        attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0, target_tlb_bit0 = 0, target_tlb_bit1 = 0, target_tlb_bit2 = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = <optimized out>
#16 0x000055a5a2cbb316 in qemu_kvm_cpu_thread_fn (arg=0x55a5a53ed9c0) at /usr/src/debug/qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64/cpus.c:1281
        cpu = 0x55a5a53ed9c0
        r = <optimized out>
#17 0x000055a5a2fd0774 in qemu_thread_start (args=0x55a5a5411440) at util/qemu-thread-posix.c:502
        __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        qemu_thread_args = 0x55a5a5411440
        start_routine = 0x55a5a2cbb260 <qemu_kvm_cpu_thread_fn>
        arg = 0x55a5a53ed9c0
        r = <optimized out>
#18 0x00007f898f6802de in start_thread () at /lib64/libpthread.so.0
#19 0x00007f898f3b1463 in clone () at /lib64/libc.so.6

Comment 3 Ademar Reis 2019-08-02 15:31:20 UTC
*** Bug 1736789 has been marked as a duplicate of this bug. ***

Comment 4 Stefan Hajnoczi 2019-08-30 09:34:52 UTC
I've sent a patch upstream and will backport it.

The issue is that QEMU does not detect the pmem size so it trusts size=4G, which is incorrect because this test creates a device that is smaller.  This leads to SIGBUS when QEMU accesses beyond the end of the pmem device.  The patch I have sent extends the size check to handle the pmem device and reject size=4G.

Comment 8 Yumei Huang 2019-09-25 06:52:04 UTC
Verify:
qemu-kvm-4.1.0-11.module+el8.1.0+4250+4f5fbfdc
kernel-4.18.0-145.el8.x86_64

Same steps as comment 0, no core dumped, qemu quit with an error message.

# /usr/libexec/qemu-kvm -M pc,nvdimm   -m 1G,slots=256,maxmem=40G  -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=off -device nvdimm,memdev=mem2,id=nv2,label-size=2M /home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 -monitor stdio -vnc :0
qemu-kvm: -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=off: backing store (null) size 0xfbe00000 does not match 'size' option 0x100000000


Also tried with pmem=on, got same error message.

# /usr/libexec/qemu-kvm -M pc,nvdimm   -m 1G,slots=256,maxmem=40G  -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=on -device nvdimm,memdev=mem2,id=nv2,label-size=2M /home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 -monitor stdio -vnc :0
qemu-kvm: -object memory-backend-file,id=mem2,share,mem-path=/dev/dax0.0,size=4G,align=128M,pmem=on: backing store (null) size 0xfbe00000 does not match 'size' option 0x100000000

Comment 10 errata-xmlrpc 2019-11-06 07:18:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.