Bug 1724008

Summary:	QEMU core dumped "memory_region_get_ram_ptr: Assertion `mr->ram_block' failed"
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Yumei Huang <yuhuang>
Component:	qemu-kvm	Assignee:	Stefan Hajnoczi <stefanha>
Status:	CLOSED ERRATA	QA Contact:	Yumei Huang <yuhuang>
Severity:	high	Docs Contact:
Priority:	high
Version:	8.1	CC:	aarcange, chayang, ddepaula, juzhang, pagupta, philmd, stefanha, virt-maint, xuelian.guo
Target Milestone:	rc	Keywords:	Regression
Target Release:	8.1	Flags:	knoel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu-kvm-4.1.0-11.module+el8.1.0+4250+4f5fbfdc	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-06 07:17:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yumei Huang 2019-06-26 02:38:37 UTC

Description of problem:

Emulate /dev/pmem by adding memmap=4G!2G to host kernel line, use it as vNVDIMM backend to boot guest, if set option pmem=on, qemu core dumped.

Version-Release number of selected component (if applicable):
qemu-kvm-4.0.0-4.module+el8.1.0+3356+cda7f1ee
kernel-4.18.0-107.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. Add memmap=4G!2G to host kernel line to emulate /dev/pmem*

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-107.el8.x86_64 root=/dev/mapper/rhel_hp--dl385g10--02-root ro crashkernel=auto resume=/dev/mapper/rhel_hp--dl385g10--02-swap rd.lvm.lv=rhel_hp-dl385g10-02/root rd.lvm.lv=rhel_hp-dl385g10-02/swap console=ttyS0,115200n81 memmap=4G!2G

# ls /dev/pmem*
/dev/pmem0  /dev/pmem2  /dev/pmem4


2. Boot guest with /dev/pmem, set option pmem=on

#  /usr/libexec/qemu-kvm  -object memory-backend-file,id=mem0,mem-path=/dev/pmem4,size=1G,share,pmem=on


Actual results:
QEMU core dumped.
qemu-kvm: /builddir/build/BUILD/qemu-4.0.0/memory.c:2155: memory_region_get_ram_ptr: Assertion `mr->ram_block' failed.
Aborted (core dumped)

Expected results:
No core dump.

Additional info:
1. Works well with qemu-kvm-2.12.0-78.module+el8.1.0+3434+46ed87c2

Comment 1 Yumei Huang 2019-06-26 02:41:27 UTC

Backtrace:

(gdb) bt
#0  0x00007f1ffe9c18af in raise () at /lib64/libc.so.6
#1  0x00007f1ffe9abcc5 in abort () at /lib64/libc.so.6
#2  0x00007f1ffe9abb99 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3  0x00007f1ffe9b9e66 in .annobin_assert.c_end () at /lib64/libc.so.6
#4  0x0000562944139a0e in memory_region_get_ram_ptr ()
#5  0x000056294423a0bc in host_memory_backend_memory_complete ()
#6  0x000056294436669f in user_creatable_add_type ()
#7  0x000056294436686a in user_creatable_add_opts ()
#8  0x00005629443669e1 in user_creatable_add_opts_foreach ()
#9  0x00005629444419c2 in qemu_opts_foreach ()
#10 0x00005629440d9cf3 in main ()

Comment 2 Yumei Huang 2019-06-26 08:42:22 UTC

Full backtrace:
(gdb) bt full
#0  0x00007f1ffe9c18af in raise () at /lib64/libc.so.6
#1  0x00007f1ffe9abcc5 in abort () at /lib64/libc.so.6
#2  0x00007f1ffe9abb99 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3  0x00007f1ffe9b9e66 in .annobin_assert.c_end () at /lib64/libc.so.6
#4  0x0000562944139a0e in memory_region_get_ram_ptr (mr=<optimized out>, mr@entry=0x56294635a060)
    at /usr/src/debug/qemu-kvm-4.0.0-4.module+el8.1.0+3356+cda7f1ee.x86_64/memory.c:2155
        ptr = <optimized out>
        offset = <optimized out>
        __PRETTY_FUNCTION__ = "memory_region_get_ram_ptr"
#5  0x000056294423a0bc in host_memory_backend_memory_complete (uc=<optimized out>, errp=0x7ffccd36cb70) at backends/hostmem.c:324
        lastbit = <optimized out>
        maxnode = <optimized out>
        flags = <optimized out>
        backend = 0x56294635a000
        __func__ = "host_memory_backend_memory_complete"
        bc = <optimized out>
        local_err = 0x0
        ptr = <optimized out>
        sz = <optimized out>
#6  0x000056294436669f in user_creatable_add_type
    (type=type@entry=0x56294630b060 "memory-backend-file", id=id@entry=0x56294630b040 "mem0", qdict=qdict@entry=0x5629463999d0, v=v@entry=0x56294639a9f0, errp=errp@entry=0x562944cb7b10 <error_fatal>) at qom/object_interfaces.c:86
        obj = 0x56294635a000
        klass = <optimized out>
        e = <optimized out>
        local_err = 0x0
        __func__ = "user_creatable_add_type"
        __PRETTY_FUNCTION__ = "user_creatable_add_type"
#7  0x000056294436686a in user_creatable_add_opts (opts=opts@entry=0x56294630aff0, errp=errp@entry=0x562944cb7b10 <error_fatal>)
    at qom/object_interfaces.c:127
        v = 0x56294639a9f0
        pdict = 0x5629463999d0
        obj = <optimized out>
        id = 0x56294630b040 "mem0"
        type = 0x56294630b060 "memory-backend-file"
        __func__ = "user_creatable_add_opts"
#8  0x00005629443669e1 in user_creatable_add_opts_foreach
    (opaque=0x562944224ce0 <object_create_delayed>, opts=0x56294630aff0, errp=0x562944cb7b10 <error_fatal>)
    at qom/object_interfaces.c:150
        type_opt_predicate = 0x562944224ce0 <object_create_delayed>
        obj = 0x0
        type = <optimized out>
#9  0x00005629444419c2 in qemu_opts_foreach
    (list=<optimized out>, func=0x5629443669a0 <user_creatable_add_opts_foreach>, opaque=0x562944224ce0 <object_create_delayed>, errp=0x562944cb7b10 <error_fatal>) at util/qemu-option.c:1171
        loc = {kind = LOC_CMDLINE, num = 2, ptr = 0x7ffccd36cec0, prev = 0x562944cb7b30 <std_loc>}
        opts = 0x56294630aff0
        rc = 0
        __PRETTY_FUNCTION__ = "qemu_opts_foreach"
#10 0x00005629440d9cf3 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4442
        i = <optimized out>
        snapshot = 0
        linux_boot = <optimized out>
        initrd_filename = 0x0
        kernel_filename = 0x0
        kernel_cmdline = <optimized out>
        boot_order = 0x5629444eade0 "cad"
        boot_once = <optimized out>
        ds = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = 0x0
        accel_opts = 0x0
        olist = <optimized out>
        optind = 3
        optarg = 0x7ffccd36d4a7 "memory-backend-file,id=mem0,mem-path=/dev/pmem4,size=1G,share,pmem=on"
        loadvm = 0x0
        machine_class = 0x562946341760
        cpu_model = 0x0
        vga_model = 0x0
        qtest_chrdev = 0x0
        qtest_log = 0x0
        incoming = 0x0
        userconfig = <optimized out>
        nographic = false
        display_remote = <optimized out>
        log_mask = <optimized out>
        log_file = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = 0x0
        main_loop_err = 0x0
        err = 0x0
        list_data_dirs = <optimized out>
        dir = <optimized out>
        bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffccd36cd50}
        __func__ = "main"

Comment 6 Yumei Huang 2019-08-19 08:46:41 UTC

Reproduced with qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1.

# /usr/libexec/qemu-kvm  -object memory-backend-file,id=mem0,mem-path=/dev/pmem0,size=1G,share,pmem=on
qemu-kvm: /builddir/build/BUILD/qemu-4.1.0-rc4/memory.c:2203: memory_region_get_ram_ptr: Assertion `mr->ram_block' failed.
Aborted (core dumped)

Comment 7 Ademar Reis 2019-08-19 14:36:18 UTC

*** Bug 1732683 has been marked as a duplicate of this bug. ***

Comment 8 Yumei Huang 2019-09-09 09:07:00 UTC

*** Bug 1732683 has been marked as a duplicate of this bug. ***

Comment 9 Stefan Hajnoczi 2019-09-18 15:10:57 UTC

I have posted a backport for bz1736788 which also fixed this BZ.  I'm not marking it as a duplicate because the root cause is different and this BZ should still be verified.

Comment 12 Yumei Huang 2019-09-25 11:03:04 UTC

Verify:
qemu-kvm-4.1.0-11.module+el8.1.0+4250+4f5fbfdc
kernel-4.18.0-145.el8.x86_64

Same steps as comment 0, generate /dev/pmem0 by modify host kernel line, then boot guest with /dev/pmem0. No core dumped, only got a warning, and guest can boot up normally. 

# /usr/libexec/qemu-kvm -M pc,nvdimm  \
-object memory-backend-file,id=mem0,mem-path=/dev/pmem0,size=4G,share,pmem=on \
-device nvdimm,memdev=mem0,id=nv0 ...
Warning: requesting persistence across crashes for backend file /dev/pmem0 failed. Proceeding without persistence, data might become corrupted in case of host crash.
QEMU 4.1.0 monitor - type 'help' for more information
(qemu) 


Also tried with a real nvdimm device on host, but still got the same warning. Stefan, would you please help confirm if it is expected? Thanks.

Comment 15 Stefan Hajnoczi 2019-10-04 12:31:31 UTC

Thanks for the update, Pankaj!

Comment 16 belinda 2019-10-24 08:50:22 UTC

Duplicated issue BUG 1732683 still reproduced on RHEL8.1 rc1:
Kernel version: 4.18.0-147.el8.x86_64
Qemu version: 4.1.0 (qemu-kvm-4.1.0-7.module+el8.1.0+4177+896cb282)

Comment 17 Yumei Huang 2019-10-24 09:09:07 UTC

(In reply to belinda from comment #16)
> Duplicated issue BUG 1732683 still reproduced on RHEL8.1 rc1:
> Kernel version: 4.18.0-147.el8.x86_64
> Qemu version: 4.1.0 (qemu-kvm-4.1.0-7.module+el8.1.0+4177+896cb282)

Hi Belinda,

This bug is fixed in qemu-kvm-4.1.0-11.module+el8.1.0+4250+4f5fbfdc, please test with fixed version, thanks.

Comment 19 errata-xmlrpc 2019-11-06 07:17:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723