Bug 1842691 - dump-guest-memory failed due to Python Exception <class 'gdb.MemoryError'>
Summary: dump-guest-memory failed due to Python Exception <class 'gdb.MemoryError'>
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Developer Toolset
Classification: Red Hat
Component: gdb
Version: DTS 10.0 RHEL 7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: alpha
: 9.1
Assignee: Keith Seitz
QA Contact: Martin Cermak
URL:
Whiteboard:
: 1847164 (view as bug list)
Depends On: 1777751
Blocks: 1785126 1847164
TreeView+ depends on / blocked
 
Reported: 2020-06-01 21:00 UTC by Keith Seitz
Modified: 2020-12-01 12:15 UTC (History)
17 users (show)

Fixed In Version: 9.2-7.el7
Doc Type: No Doc Update
Doc Text:
Clone Of: 1777751
Environment:
Last Closed: 2020-12-01 12:15:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Keith Seitz 2020-06-01 21:00:57 UTC
+++ This bug was initially created as a clone of Bug #1777751 +++

Description of problem:

dump-guest-memory failed due to Python Exception <class 'gdb.MemoryError'> during test qemu dump


Try it with seabios, not hit it. But hit it with OVMF.


Version-Release number of selected component (if applicable):
Host:
kernel-3.10.0-1113.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-1113.el7.x86_64
kernel-debuginfo-3.10.0-1113.el7.x86_64
kernel-debug-3.10.0-1113.el7.x86_64
qemu-kvm-rhev-2.12.0-38.el7
spice-server-0.14.2-1.el8.x86_64
seavgabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch
seabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch
seabios-1.11.1-4.module+el8.1.0+4066+0f1aadab.x86_64
OVMF-20180508-6.gitee3198e672e2.el7.noarch

Guest:
kernel-3.10.0-1113.el7.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Set 'ulimit -c unlimited' and 'echo "/var/core.%p-%u-%g-%s-%t-%h-%e" >
/proc/sys/kernel/core_pattern' in the host terminal

2. boot guest with dump-guest-core=on via seabios
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35,dump-guest-core=on \
    -nodefaults \
    -device qxl-vga,bus=pcie.0,addr=0x1 \
    -m 5457  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_6oWAIJ/monitor-qmpmonitor1-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_6oWAIJ/monitor-catch_monitor-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idgXePLi \
    -chardev socket,path=/var/tmp/avocado_6oWAIJ/serial-serial0-20191126-064055-ae4I6Jkf,nowait,id=chardev_serial0,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20191126-064055-ae4I6Jkf,path=/var/tmp/avocado_6oWAIJ/seabios-20191126-064055-ae4I6Jkf,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20191126-064055-ae4I6Jkf,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=0x2 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2.seabios \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:15:f1:28:fb:2f,id=idjEhtS7,netdev=id2p3VOQ,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=id2p3VOQ,vhost=on \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/linux/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d  \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=xhci,bus=pcie_extra_root_port_0 \
    -spice port=5931,disable-ticketing \
    -drive id=drive_stg,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/usbdevice.qcow2 \
    -device usb-storage,id=stg,drive=drive_stg,serial=TARGET_DISK0,bus=xhci.0,port=1 \
    -qmp tcp:0:4444,server,nowait \

3.Kill the qemu-kvm process with SIGSEGV, as root:
# pkill -SEGV -x qemu-kvm
(qemu) Segmentation fault (core dumped)

4.Open the "coredump" file with gdb, and load the extension.
# gdb /var/core.7971-0-0-11-1574850340-hp-dl385pg8-03.lab.eng.pek2.redhat.com-qemu-kvm
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-117.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 7971]
[New LWP 8098]
......
[New LWP 8134]
[New LWP 8095]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -machine q35,dump-guest-core=on -'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd97ddf6cff in ppoll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install boost-iostreams-1.53.0-28.el7.x86_64 boost-random-1.53.0-28.el7.x86_64 boost-system-1.53.0-28.el7.x86_64 boost-thread-1.53.0-28.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 cyrus-sasl-scram-2.1.26-23.el7.x86_64 elfutils-libelf-0.176-4.el7.x86_64 elfutils-libs-0.176-4.el7.x86_64 expat-2.1.0-11.el7.x86_64 glib2-2.56.1-5.el7.x86_64 glibc-2.17-307.el7.x86_64 glusterfs-api-6.0-20.el7.x86_64 glusterfs-libs-6.0-20.el7.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.29-9.el7_6.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-46.el7.x86_64 libacl-2.2.51-15.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-63.el7.x86_64 libcacard-2.7.0-1.el7.x86_64 libcap-2.22-11.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libcurl-7.29.0-56.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libdrm-2.4.97-2.el7.x86_64 libepoxy-1.5.2-1.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibumad-22.1-3.el7.x86_64 libibverbs-22.1-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libiscsi-1.9.0-7.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libmount-2.23.2-63.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 librados2-10.2.5-4.el7.x86_64 librbd1-10.2.5-4.el7.x86_64 librdmacm-22.1-3.el7.x86_64 libseccomp-2.3.1-3.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-3.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-63.el7.x86_64 libwayland-server-1.15.0-1.el7.x86_64 lz4-1.7.5-3.el7.x86_64 lzo-2.06-8.el7.x86_64 mesa-libgbm-18.3.4-5.el7.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 numactl-libs-2.0.12-5.el7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 opus-1.0.2-6.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 snappy-1.1.0-3.el7.x86_64 spice-server-0.14.0-8.el7.x86_64 systemd-libs-219-73.el7.x86_64 usbredir-0.7.1-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py

5.extract the guest vmcore with the new extension
(gdb) set height 0
(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fd81c600000 added       1
00000000000c0000 00000000000ca000 00007fd81c6c0000 added       2
00000000000ca000 00000000000cd000 00007fd81c6ca000 joined      2
00000000000cd000 00000000000e8000 00007fd81c6cd000 joined      2
00000000000e8000 00000000000f0000 00007fd81c6e8000 joined      2
00000000000f0000 0000000000100000 00007fd81c6f0000 joined      2
0000000000100000 0000000080000000 00007fd81c700000 joined      2
00000000f4000000 00000000f8000000 00007fd818000000 added       3
00000000f8000000 00000000fc000000 00007fd813c00000 added       4
00000000fc614000 00000000fc616000 00007fd817e00000 added       5
00000000fffc0000 0000000100000000 00007fd81c400000 added       6
0000000100000000 00000001d5100000 00007fd89c600000 added       7
dumping range at 00007fd81c600000 for length 00000000000a0000
dumping range at 00007fd81c6c0000 for length 000000007ff40000
dumping range at 00007fd818000000 for length 0000000004000000
dumping range at 00007fd813c00000 for length 0000000004000000
dumping range at 00007fd817e00000 for length 0000000000002000
dumping range at 00007fd81c400000 for length 0000000000040000
dumping range at 00007fd89c600000 for length 00000000d5100000
(gdb)
(gdb) bt
#0  0x00007fd97ddf6cff in ppoll () at /lib64/libc.so.6
#1  0x000055e9e7c381d9 in qemu_poll_ns (__ss=0x0, __timeout=0x7ffd6e9dddc0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x000055e9e7c381d9 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=451675965) at util/qemu-timer.c:334
#3  0x000055e9e7c390be in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:233
#4  0x000055e9e7c390be in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
#5  0x000055e9e78d8467 in main () at vl.c:2050
#6  0x000055e9e78d8467 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4813

6. change seabios to OVMF and boot guest with dump-guest-core=on
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35,dump-guest-core=on \
    -nodefaults \
    -device qxl-vga,bus=pcie.0,addr=0x1 \
    -m 5457  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_6oWAIJ/monitor-qmpmonitor1-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_6oWAIJ/monitor-catch_monitor-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idgXePLi \
    -chardev socket,path=/var/tmp/avocado_6oWAIJ/serial-serial0-20191126-064055-ae4I6Jkf,nowait,id=chardev_serial0,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20191126-064055-ae4I6Jkf,path=/var/tmp/avocado_6oWAIJ/seabios-20191126-064055-ae4I6Jkf,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20191126-064055-ae4I6Jkf,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=0x2 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:15:f1:28:fb:2f,id=idjEhtS7,netdev=id2p3VOQ,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=id2p3VOQ,vhost=on \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/linux/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d  \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.secboot.fd \
    -drive if=pflash,format=raw,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2.fd \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=xhci,bus=pcie_extra_root_port_0 \
    -spice port=5931,disable-ticketing \
    -drive id=drive_stg,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/usbdevice.qcow2 \
    -device usb-storage,id=stg,drive=drive_stg,serial=TARGET_DISK0,bus=xhci.0,port=1 \
    -qmp tcp:0:4444,server,nowait \

7.Kill the qemu-kvm process with SIGSEGV, as root:
# pkill -SEGV -x qemu-kvm
(qemu) Segmentation fault (core dumped)

8.Open the "coredump" file with gdb, and load the extension.
# gdb /var/core.7051-0-0-11-1574848207-hp-dl385pg8-03.lab.eng.pek2.redhat.com-qemu-kvm
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-117.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 7051]
[New LWP 7206]
......
[New LWP 7073]
[New LWP 7053]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -machine q35,dump-guest-core=on -'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fed990c6cff in ppoll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install boost-iostreams-1.53.0-28.el7.x86_64 boost-random-1.53.0-28.el7.x86_64 boost-system-1.53.0-28.el7.x86_64 boost-thread-1.53.0-28.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 cyrus-sasl-scram-2.1.26-23.el7.x86_64 elfutils-libelf-0.176-4.el7.x86_64 elfutils-libs-0.176-4.el7.x86_64 expat-2.1.0-11.el7.x86_64 glib2-2.56.1-5.el7.x86_64 glibc-2.17-307.el7.x86_64 glusterfs-api-6.0-20.el7.x86_64 glusterfs-libs-6.0-20.el7.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.29-9.el7_6.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-46.el7.x86_64 libacl-2.2.51-15.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-63.el7.x86_64 libcacard-2.7.0-1.el7.x86_64 libcap-2.22-11.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libcurl-7.29.0-56.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libdrm-2.4.97-2.el7.x86_64 libepoxy-1.5.2-1.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibumad-22.1-3.el7.x86_64 libibverbs-22.1-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libiscsi-1.9.0-7.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libmount-2.23.2-63.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 librados2-10.2.5-4.el7.x86_64 librbd1-10.2.5-4.el7.x86_64 librdmacm-22.1-3.el7.x86_64 libseccomp-2.3.1-3.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-3.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-63.el7.x86_64 libwayland-server-1.15.0-1.el7.x86_64 lz4-1.7.5-3.el7.x86_64 lzo-2.06-8.el7.x86_64 mesa-libgbm-18.3.4-5.el7.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 numactl-libs-2.0.12-5.el7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 opus-1.0.2-6.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 snappy-1.1.0-3.el7.x86_64 spice-server-0.14.0-8.el7.x86_64 systemd-libs-219-73.el7.x86_64 usbredir-0.7.1-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py

9.extract the guest vmcore with the new extension
(gdb) set height 0
(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fec37800000 added       1
00000000000a0000 00000000000b0000 00007fec32c00000 added       2
00000000000c0000 00000000000c4000 00007fec378c0000 added       3
00000000000c4000 00000000000e0000 00007fec36e04000 added       4
00000000000e0000 0000000000100000 00007fec37200000 added       5
0000000000100000 000000007f000000 00007fec37900000 added       6
0000000090000000 0000000094000000 00007fec2e800000 added       7
0000000094000000 0000000098000000 00007fec32c00000 added       8
0000000098600000 0000000098602000 00007fec32a00000 added       9
0000000100000000 00000001d5100000 00007fecb7800000 added      10
dumping range at 00007fec37800000 for length 00000000000a0000
dumping range at 00007fec32c00000 for length 0000000000010000
dumping range at 00007fec378c0000 for length 0000000000004000
dumping range at 00007fec36e04000 for length 000000000001c000
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fec36e04000:
Error occurred in Python command: Cannot access memory at address 0x7fec36e04000
(gdb)
(gdb) bt
#0  0x00007fed990c6cff in ppoll () at /lib64/libc.so.6
#1  0x000055d82aa6a1d9 in qemu_poll_ns (__ss=0x0, __timeout=0x7ffed73caca0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x000055d82aa6a1d9 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=2999255243) at util/qemu-timer.c:334
#3  0x000055d82aa6b0be in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:233
#4  0x000055d82aa6b0be in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
#5  0x000055d82a70a467 in main () at vl.c:2050
#6  0x000055d82a70a467 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4813



Actual results:

After step 5, not hit this issue.
After step 9, hit it. 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fec36e04000:
Error occurred in Python command: Cannot access memory at address 0x7fec36e04000


Expected results:
after step 5 and step 9, all work well.


Additional info:

--- Additional comment from RHEL Program Management on 2019-11-28 09:09:11 UTC ---

Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Xueqiang Wei on 2019-11-28 09:14:43 UTC ---

Hi Andre,

I paste your feedback. Do you have anything to add? Thank you.


I have done some research, and it seems to me it could be either a gdb
or linux change.

It's always "pc.rom" ramblock that fails (and perhaps other after)

(gdb) p *address_space_memory.current_map.ranges[4].mr.ram_block
$1 = {
  rcu = {
    next = 0x0,
    func = 0x0
  },
  mr = 0x5622cfe00f80,
  host = 0x7fa597200000 <error: Cannot access memory at address 0x7fa597200000>,
  colo_cache = 0x0,
  offset = 2152202240,
  used_length = 131072,
  max_length = 131072,
  resized = 0x0,
  flags = 16,
  idstr = "pc.rom", '\000' <repeats 249 times>,
  next = {
    le_next = 0x5622d0962c80,
    le_prev = 0x5622cff846e0
  },
  ramblock_notifiers = {
    lh_first = 0x0
  },
  fd = -1,
  page_size = 4096,
  bmap = 0x0,
  receivedmap = 0x0,
  clear_bmap = 0x0,
  clear_bmap_shift = 0 '\000'
}

(gdb) maintenance info sections
...
 [214]     0x7fa597200000->0x7fa597200000 at 0x8b188000: load171 ALLOC

So the section is there, and it is supposed to be allocated. But since
there is no content, gdb might be avoiding allocation? We will need
gdb developpers help.

--- Additional comment from Marc-Andre Lureau on 2019-11-28 09:37:42 UTC ---

Reassigning to gdb, please reply on comment 2 (the ALLOC section not being readable). Thanks

--- Additional comment from David Smith on 2019-12-03 20:47:22 UTC ---

Moving out to 7.9.

--- Additional comment from Keith Seitz on 2019-12-06 17:39:02 UTC ---

What is dump-guest-memory? Is this included in some package that I haven't installed?

I've grabbed the script off github, but that does not appear to work with the ancient version of GDB on rhel7.
At least I think it doesn't. I cannot get the reproducer working. [I'm no virtualization expert obviously.
So I've simply installed rhel-7.8 beta into a VM and hacked at the qemu-kvm commands above until something ran.
Then I sent the qemu-kvm process a segv signal and used the resulting core file.]

From my session:

(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
Python Exception <class 'gdb.error'> There is no member named nonvolatile.: 
Error occurred in Python command: There is no member named nonvolatile.

That could be anything, though. I've never used any of these programs/scripts.

Have you tried DTS8 or DTS9 gdb? That might be faster to resolve.

--- Additional comment from Keith Seitz on 2019-12-06 19:39:59 UTC ---

After digging a little further...

I've managed to reproduce something similar to the original complaint:

(gdb) dump-guest-memory vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fb059c00000 added       1
00000000000c0000 00000000000e0000 00007fb059400000 added       2
00000000000e0000 0000000000100000 00007fb059600000 added       3
0000000000100000 0000000080000000 00007fb059d00000 added       4
dumping range at 00007fb059c00000 for length 00000000000a0000
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fb059c00000:
Error occurred in Python command: Cannot access memory at address 0x7fb059c00000

This happens regardless of seabios or ovmf, but probably does not matter. To make get here,
I've removed line 423 of the original script from github (the part that does memory_region["nonvolatile"]).

From "maint info sections" for the above address,

0x7fb059c00000->0x7fb059c00000 at 0x08571000: load30 ALLOC

This is all computed from BFD:

      addr = bfd_section_vma (asect);
      endaddr = addr + bfd_section_size (asect);

These are the addresses shown in the output. Since both are the same (0x7fb059c00000),
bfd_section_size for the section is 0. BFD simply does not know anything more.

I will coordinate with BFD maintainers and see if there is anything we can/should do.

--- Additional comment from Marc-Andre Lureau on 2019-12-06 20:33:33 UTC ---

thanks!

--- Additional comment from Laszlo Ersek on 2020-02-11 08:55:27 UTC ---

Per Kevin Buettner's analysis in <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c4>, the host kernel's process coredump logic is what decides to omit the never-written-to region from the qemu process coredump. The python script should expect and handle this possibility.

Kevin proposed a patch in <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c3>; we should upstream it and backport it.

Moving this BZ back to qemu-kvm-rhev.

--- Additional comment from John Ferlan on 2020-02-12 14:47:10 UTC ---

Laszlo I assume this is yours, so I'll directly assign. I also changed to rhel-7.9.0? since that's the next release to be considered.

--- Additional comment from Laszlo Ersek on 2020-02-13 22:19:32 UTC ---

Hi John,

(In reply to John Ferlan from comment #9)
> Laszlo I assume this is yours, so I'll directly assign. I also changed
> to rhel-7.9.0? since that's the next release to be considered.

no, it doesn't belong in my queue:

$ scripts/get_maintainer.pl -f scripts/dump-guest-memory.py

> "Marc-André Lureau" <marcandre.lureau> (supporter:Dump)
> Eduardo Habkost <ehabkost> (odd fixer:Python scripts)
> Cleber Rosa <crosa> (odd fixer:Python scripts)
> qemu-devel (open list:All patches CC here)

I have contributed exactly one commit to "scripts/dump-guest-memory.py",
namely the initial one (3e16d14fd93c, "Python-lang gdb script to extract
x86_64 guest vmcore from qemu coredump", 2014-01-26). The only reason I
learned of this BZ is that Kevin Buettner wrote me an email about his
analysis under bug 1785126 (i.e. the RHEL8 instance of the same
problem).

I think Kevin must have found my email simply from the banner that was
added to the script in the original commit:

> diff --git a/scripts/dump-guest-memory.py b/scripts/dump-guest-memory.py
> new file mode 100644
> index 000000000000..1ed8b67883a3
> --- /dev/null
> +++ b/scripts/dump-guest-memory.py
> @@ -0,0 +1,339 @@
> +# This python script adds a new gdb command, "dump-guest-memory". It
> +# should be loaded with "source dump-guest-memory.py" at the (gdb)
> +# prompt.
> +#
> +# Copyright (C) 2013, Red Hat, Inc.
> +#
> +# Authors:
> +#   Laszlo Ersek <lersek>

(Kevin also notified, in the same email, Janosch Frank
<frankja.ibm.com>, whose name+email are also in the banner,
from commit 368e3adc8928, "scripts/dump-guest-memory.py: Introduce
multi-arch support", 2016-01-26).

So I figured I'd capture Kevin's info here (in the RH Bugzilla instance)
the best I could. Doesn't mean I plan (or even can) get more involved in
it :) Given Marc-André's upstream supporter role for the script, I
suggest asking him.

For now I'm returning the bug to the backlog. Thanks!
Laszlo

--- Additional comment from Laszlo Ersek on 2020-03-05 15:43:58 UTC ---

Moving to gdb per <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c14>. Thanks!

--- Additional comment from David Smith on 2020-06-01 20:54:06 UTC ---

Since this one didn't make it upstream in time, I'm removing it from RHEL 7.9. We should be able to fix this for the DTS 10.0 version of gdb.

Comment 2 Keith Seitz 2020-06-01 21:07:33 UTC
Moving to DTS10.

Comment 3 David Smith 2020-06-30 20:06:28 UTC
*** Bug 1847164 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2020-12-01 12:15:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: devtoolset-10-gdb), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5295


Note You need to log in before you can comment on or make changes to this bug.