+++ This bug was initially created as a clone of Bug #1785126 +++

+++ This bug was initially created as a clone of Bug #1777751 +++

Description of problem:

dump-guest-memory failed due to Python Exception <class 'gdb.MemoryError'> during test qemu dump


Try it with seabios, not hit it. But hit it with OVMF.


Version-Release number of selected component (if applicable):
Host:
kernel-3.10.0-1113.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-1113.el7.x86_64
kernel-debuginfo-3.10.0-1113.el7.x86_64
kernel-debug-3.10.0-1113.el7.x86_64
qemu-kvm-rhev-2.12.0-38.el7
spice-server-0.14.2-1.el8.x86_64
seavgabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch
seabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch
seabios-1.11.1-4.module+el8.1.0+4066+0f1aadab.x86_64
OVMF-20180508-6.gitee3198e672e2.el7.noarch

Guest:
kernel-3.10.0-1113.el7.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Set 'ulimit -c unlimited' and 'echo "/var/core.%p-%u-%g-%s-%t-%h-%e" >
/proc/sys/kernel/core_pattern' in the host terminal

2. boot guest with dump-guest-core=on via seabios
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35,dump-guest-core=on \
    -nodefaults \
    -device qxl-vga,bus=pcie.0,addr=0x1 \
    -m 5457  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_6oWAIJ/monitor-qmpmonitor1-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_6oWAIJ/monitor-catch_monitor-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idgXePLi \
    -chardev socket,path=/var/tmp/avocado_6oWAIJ/serial-serial0-20191126-064055-ae4I6Jkf,nowait,id=chardev_serial0,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20191126-064055-ae4I6Jkf,path=/var/tmp/avocado_6oWAIJ/seabios-20191126-064055-ae4I6Jkf,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20191126-064055-ae4I6Jkf,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=0x2 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2.seabios \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:15:f1:28:fb:2f,id=idjEhtS7,netdev=id2p3VOQ,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=id2p3VOQ,vhost=on \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/linux/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d  \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=xhci,bus=pcie_extra_root_port_0 \
    -spice port=5931,disable-ticketing \
    -drive id=drive_stg,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/usbdevice.qcow2 \
    -device usb-storage,id=stg,drive=drive_stg,serial=TARGET_DISK0,bus=xhci.0,port=1 \
    -qmp tcp:0:4444,server,nowait \

3.Kill the qemu-kvm process with SIGSEGV, as root:
# pkill -SEGV -x qemu-kvm
(qemu) Segmentation fault (core dumped)

4.Open the "coredump" file with gdb, and load the extension.
# gdb /var/core.7971-0-0-11-1574850340-hp-dl385pg8-03.lab.eng.pek2.redhat.com-qemu-kvm
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-117.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 7971]
[New LWP 8098]
......
[New LWP 8134]
[New LWP 8095]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -machine q35,dump-guest-core=on -'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd97ddf6cff in ppoll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install boost-iostreams-1.53.0-28.el7.x86_64 boost-random-1.53.0-28.el7.x86_64 boost-system-1.53.0-28.el7.x86_64 boost-thread-1.53.0-28.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 cyrus-sasl-scram-2.1.26-23.el7.x86_64 elfutils-libelf-0.176-4.el7.x86_64 elfutils-libs-0.176-4.el7.x86_64 expat-2.1.0-11.el7.x86_64 glib2-2.56.1-5.el7.x86_64 glibc-2.17-307.el7.x86_64 glusterfs-api-6.0-20.el7.x86_64 glusterfs-libs-6.0-20.el7.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.29-9.el7_6.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-46.el7.x86_64 libacl-2.2.51-15.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-63.el7.x86_64 libcacard-2.7.0-1.el7.x86_64 libcap-2.22-11.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libcurl-7.29.0-56.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libdrm-2.4.97-2.el7.x86_64 libepoxy-1.5.2-1.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibumad-22.1-3.el7.x86_64 libibverbs-22.1-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libiscsi-1.9.0-7.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libmount-2.23.2-63.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 librados2-10.2.5-4.el7.x86_64 librbd1-10.2.5-4.el7.x86_64 librdmacm-22.1-3.el7.x86_64 libseccomp-2.3.1-3.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-3.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-63.el7.x86_64 libwayland-server-1.15.0-1.el7.x86_64 lz4-1.7.5-3.el7.x86_64 lzo-2.06-8.el7.x86_64 mesa-libgbm-18.3.4-5.el7.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 numactl-libs-2.0.12-5.el7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 opus-1.0.2-6.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 snappy-1.1.0-3.el7.x86_64 spice-server-0.14.0-8.el7.x86_64 systemd-libs-219-73.el7.x86_64 usbredir-0.7.1-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py

5.extract the guest vmcore with the new extension
(gdb) set height 0
(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fd81c600000 added       1
00000000000c0000 00000000000ca000 00007fd81c6c0000 added       2
00000000000ca000 00000000000cd000 00007fd81c6ca000 joined      2
00000000000cd000 00000000000e8000 00007fd81c6cd000 joined      2
00000000000e8000 00000000000f0000 00007fd81c6e8000 joined      2
00000000000f0000 0000000000100000 00007fd81c6f0000 joined      2
0000000000100000 0000000080000000 00007fd81c700000 joined      2
00000000f4000000 00000000f8000000 00007fd818000000 added       3
00000000f8000000 00000000fc000000 00007fd813c00000 added       4
00000000fc614000 00000000fc616000 00007fd817e00000 added       5
00000000fffc0000 0000000100000000 00007fd81c400000 added       6
0000000100000000 00000001d5100000 00007fd89c600000 added       7
dumping range at 00007fd81c600000 for length 00000000000a0000
dumping range at 00007fd81c6c0000 for length 000000007ff40000
dumping range at 00007fd818000000 for length 0000000004000000
dumping range at 00007fd813c00000 for length 0000000004000000
dumping range at 00007fd817e00000 for length 0000000000002000
dumping range at 00007fd81c400000 for length 0000000000040000
dumping range at 00007fd89c600000 for length 00000000d5100000
(gdb)
(gdb) bt
#0  0x00007fd97ddf6cff in ppoll () at /lib64/libc.so.6
#1  0x000055e9e7c381d9 in qemu_poll_ns (__ss=0x0, __timeout=0x7ffd6e9dddc0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x000055e9e7c381d9 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=451675965) at util/qemu-timer.c:334
#3  0x000055e9e7c390be in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:233
#4  0x000055e9e7c390be in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
#5  0x000055e9e78d8467 in main () at vl.c:2050
#6  0x000055e9e78d8467 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4813

6. change seabios to OVMF and boot guest with dump-guest-core=on
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1' \
    -machine q35,dump-guest-core=on \
    -nodefaults \
    -device qxl-vga,bus=pcie.0,addr=0x1 \
    -m 5457  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_6oWAIJ/monitor-qmpmonitor1-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_6oWAIJ/monitor-catch_monitor-20191126-064055-ae4I6Jkf,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idgXePLi \
    -chardev socket,path=/var/tmp/avocado_6oWAIJ/serial-serial0-20191126-064055-ae4I6Jkf,nowait,id=chardev_serial0,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20191126-064055-ae4I6Jkf,path=/var/tmp/avocado_6oWAIJ/seabios-20191126-064055-ae4I6Jkf,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20191126-064055-ae4I6Jkf,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=0x2 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:15:f1:28:fb:2f,id=idjEhtS7,netdev=id2p3VOQ,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=id2p3VOQ,vhost=on \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/linux/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=d  \
    -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.secboot.fd \
    -drive if=pflash,format=raw,file=/home/kvm_autotest_root/images/rhel78-64-virtio-scsi.qcow2.fd \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device qemu-xhci,id=xhci,bus=pcie_extra_root_port_0 \
    -spice port=5931,disable-ticketing \
    -drive id=drive_stg,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/usbdevice.qcow2 \
    -device usb-storage,id=stg,drive=drive_stg,serial=TARGET_DISK0,bus=xhci.0,port=1 \
    -qmp tcp:0:4444,server,nowait \

7.Kill the qemu-kvm process with SIGSEGV, as root:
# pkill -SEGV -x qemu-kvm
(qemu) Segmentation fault (core dumped)

8.Open the "coredump" file with gdb, and load the extension.
# gdb /var/core.7051-0-0-11-1574848207-hp-dl385pg8-03.lab.eng.pek2.redhat.com-qemu-kvm
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-117.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 7051]
[New LWP 7206]
......
[New LWP 7073]
[New LWP 7053]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -machine q35,dump-guest-core=on -'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fed990c6cff in ppoll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install boost-iostreams-1.53.0-28.el7.x86_64 boost-random-1.53.0-28.el7.x86_64 boost-system-1.53.0-28.el7.x86_64 boost-thread-1.53.0-28.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 cyrus-sasl-scram-2.1.26-23.el7.x86_64 elfutils-libelf-0.176-4.el7.x86_64 elfutils-libs-0.176-4.el7.x86_64 expat-2.1.0-11.el7.x86_64 glib2-2.56.1-5.el7.x86_64 glibc-2.17-307.el7.x86_64 glusterfs-api-6.0-20.el7.x86_64 glusterfs-libs-6.0-20.el7.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.29-9.el7_6.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-46.el7.x86_64 libacl-2.2.51-15.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-63.el7.x86_64 libcacard-2.7.0-1.el7.x86_64 libcap-2.22-11.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libcurl-7.29.0-56.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libdrm-2.4.97-2.el7.x86_64 libepoxy-1.5.2-1.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibumad-22.1-3.el7.x86_64 libibverbs-22.1-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libiscsi-1.9.0-7.el7.x86_64 libjpeg-turbo-1.2.90-8.el7.x86_64 libmount-2.23.2-63.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 librados2-10.2.5-4.el7.x86_64 librbd1-10.2.5-4.el7.x86_64 librdmacm-22.1-3.el7.x86_64 libseccomp-2.3.1-3.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-3.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libusbx-1.0.21-1.el7.x86_64 libuuid-2.23.2-63.el7.x86_64 libwayland-server-1.15.0-1.el7.x86_64 lz4-1.7.5-3.el7.x86_64 lzo-2.06-8.el7.x86_64 mesa-libgbm-18.3.4-5.el7.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 numactl-libs-2.0.12-5.el7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 opus-1.0.2-6.el7.x86_64 p11-kit-0.23.5-3.el7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 snappy-1.1.0-3.el7.x86_64 spice-server-0.14.0-8.el7.x86_64 systemd-libs-219-73.el7.x86_64 usbredir-0.7.1-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py

9.extract the guest vmcore with the new extension
(gdb) set height 0
(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fec37800000 added       1
00000000000a0000 00000000000b0000 00007fec32c00000 added       2
00000000000c0000 00000000000c4000 00007fec378c0000 added       3
00000000000c4000 00000000000e0000 00007fec36e04000 added       4
00000000000e0000 0000000000100000 00007fec37200000 added       5
0000000000100000 000000007f000000 00007fec37900000 added       6
0000000090000000 0000000094000000 00007fec2e800000 added       7
0000000094000000 0000000098000000 00007fec32c00000 added       8
0000000098600000 0000000098602000 00007fec32a00000 added       9
0000000100000000 00000001d5100000 00007fecb7800000 added      10
dumping range at 00007fec37800000 for length 00000000000a0000
dumping range at 00007fec32c00000 for length 0000000000010000
dumping range at 00007fec378c0000 for length 0000000000004000
dumping range at 00007fec36e04000 for length 000000000001c000
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fec36e04000:
Error occurred in Python command: Cannot access memory at address 0x7fec36e04000
(gdb)
(gdb) bt
#0  0x00007fed990c6cff in ppoll () at /lib64/libc.so.6
#1  0x000055d82aa6a1d9 in qemu_poll_ns (__ss=0x0, __timeout=0x7ffed73caca0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
#2  0x000055d82aa6a1d9 in qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=2999255243) at util/qemu-timer.c:334
#3  0x000055d82aa6b0be in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:233
#4  0x000055d82aa6b0be in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:497
#5  0x000055d82a70a467 in main () at vl.c:2050
#6  0x000055d82a70a467 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4813



Actual results:

After step 5, not hit this issue.
After step 9, hit it. 
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fec36e04000:
Error occurred in Python command: Cannot access memory at address 0x7fec36e04000


Expected results:
after step 5 and step 9, all work well.


Additional info:

--- Additional comment from RHEL Program Management on 2019-11-28 09:09:11 UTC ---

Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Xueqiang Wei on 2019-11-28 09:14:43 UTC ---

Hi Andre,

I paste your feedback. Do you have anything to add? Thank you.


I have done some research, and it seems to me it could be either a gdb
or linux change.

It's always "pc.rom" ramblock that fails (and perhaps other after)

(gdb) p *address_space_memory.current_map.ranges[4].mr.ram_block
$1 = {
  rcu = {
    next = 0x0,
    func = 0x0
  },
  mr = 0x5622cfe00f80,
  host = 0x7fa597200000 <error: Cannot access memory at address 0x7fa597200000>,
  colo_cache = 0x0,
  offset = 2152202240,
  used_length = 131072,
  max_length = 131072,
  resized = 0x0,
  flags = 16,
  idstr = "pc.rom", '\000' <repeats 249 times>,
  next = {
    le_next = 0x5622d0962c80,
    le_prev = 0x5622cff846e0
  },
  ramblock_notifiers = {
    lh_first = 0x0
  },
  fd = -1,
  page_size = 4096,
  bmap = 0x0,
  receivedmap = 0x0,
  clear_bmap = 0x0,
  clear_bmap_shift = 0 '\000'
}

(gdb) maintenance info sections
...
 [214]     0x7fa597200000->0x7fa597200000 at 0x8b188000: load171 ALLOC

So the section is there, and it is supposed to be allocated. But since
there is no content, gdb might be avoiding allocation? We will need
gdb developpers help.

--- Additional comment from Marc-Andre Lureau on 2019-11-28 09:37:42 UTC ---

Reassigning to gdb, please reply on comment 2 (the ALLOC section not being readable). Thanks

--- Additional comment from David Smith on 2019-12-03 20:47:22 UTC ---

Moving out to 7.9.

--- Additional comment from Keith Seitz on 2019-12-06 17:39:02 UTC ---

What is dump-guest-memory? Is this included in some package that I haven't installed?

I've grabbed the script off github, but that does not appear to work with the ancient version of GDB on rhel7.
At least I think it doesn't. I cannot get the reproducer working. [I'm no virtualization expert obviously.
So I've simply installed rhel-7.8 beta into a VM and hacked at the qemu-kvm commands above until something ran.
Then I sent the qemu-kvm process a segv signal and used the resulting core file.]

From my session:

(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
Python Exception <class 'gdb.error'> There is no member named nonvolatile.: 
Error occurred in Python command: There is no member named nonvolatile.

That could be anything, though. I've never used any of these programs/scripts.

Have you tried DTS8 or DTS9 gdb? That might be faster to resolve.

--- Additional comment from Keith Seitz on 2019-12-06 19:39:59 UTC ---

After digging a little further...

I've managed to reproduce something similar to the original complaint:

(gdb) dump-guest-memory vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007fb059c00000 added       1
00000000000c0000 00000000000e0000 00007fb059400000 added       2
00000000000e0000 0000000000100000 00007fb059600000 added       3
0000000000100000 0000000080000000 00007fb059d00000 added       4
dumping range at 00007fb059c00000 for length 00000000000a0000
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fb059c00000:
Error occurred in Python command: Cannot access memory at address 0x7fb059c00000

This happens regardless of seabios or ovmf, but probably does not matter. To make get here,
I've removed line 423 of the original script from github (the part that does memory_region["nonvolatile"]).

From "maint info sections" for the above address,

0x7fb059c00000->0x7fb059c00000 at 0x08571000: load30 ALLOC

This is all computed from BFD:

      addr = bfd_section_vma (asect);
      endaddr = addr + bfd_section_size (asect);

These are the addresses shown in the output. Since both are the same (0x7fb059c00000),
bfd_section_size for the section is 0. BFD simply does not know anything more.

I will coordinate with BFD maintainers and see if there is anything we can/should do.

--- Additional comment from Marc-Andre Lureau on 2019-12-06 20:33:33 UTC ---

thanks!

--- Additional comment from Xueqiang Wei on 2019-12-19 08:45:18 UTC ---

Versions:
Host:
kernel-4.18.0-167.el8.x86_64
kernel-debuginfo-common-x86_64-4.18.0-167.el8.x86_64
kernel-debuginfo-4.18.0-167.el8.x86_64
kernel-debug-4.18.0-167.el8.x86_64
gdb-headless-8.2-8.el8.x86_64
gdbm-1.18-1.el8.x86_64
gdbm-libs-1.18-1.el8.x86_64
gdb-8.2-8.el8.x86_64
gcc-gdb-plugin-8.3.1-5.el8.x86_64
qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc
spice-server-0.14.2-1.el8.x86_64
seavgabios-bin-1.12.0-5.module+el8.2.0+4793+b09dd2fb.noarch
seabios-1.12.0-5.module+el8.2.0+4793+b09dd2fb.x86_64
seabios-bin-1.12.0-5.module+el8.2.0+4793+b09dd2fb.noarch
edk2-ovmf-20190829git37eef91017ad-4.el8.noarch

Guest:
kernel-4.18.0-160.el8.x86_64


Not hit it with seabois.   
Hit it with edk2.

--- Additional comment from Xueqiang Wei on 2020-01-15 06:38:28 UTC ---


With the same environment mentioned in Comment 1, tested with dump-guest-core=off.

If set dump-guest-core=off: 1. hit this issue with seabios  2. hit this issue with edk2

(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py 
(gdb) set height 0
(gdb) dump-guest-memory /tmp/vmcore X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007f9613e00000 added       1
00000000000c0000 00000000000ca000 00007f9613ec0000 added       2
00000000000ca000 00000000000cd000 00007f9613eca000 joined      2
00000000000cd000 00000000000e8000 00007f9613ecd000 joined      2
00000000000e8000 00000000000f0000 00007f9613ee8000 joined      2
00000000000f0000 0000000000100000 00007f9613ef0000 joined      2
0000000000100000 0000000080000000 00007f9613f00000 joined      2
00000000f4000000 00000000f8000000 00007f960fc00000 added       3
00000000f8000000 00000000fc000000 00007f960ba00000 added       4
00000000fd010000 00000000fd012000 00007f971ba00000 added       5
00000000fffc0000 0000000100000000 00007f971be00000 added       6
0000000100000000 0000000180000000 00007f9693e00000 added       7
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7f970f87c000: 
Error occurred in Python command: Cannot access memory at address 0x7f970f87c000
(gdb) bt
#0  0x00007f9732829016 in ppoll () at /lib64/libc.so.6
#1  0x0000560d590351c5 in ppoll
    (__ss=0x0, __timeout=0x7ffd99034da0, __nfds=<optimized out>, __fds=<optimized out>)
    at /usr/include/bits/poll2.h:77
#2  0x0000560d590351c5 in qemu_poll_ns
    (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=1675788861)
    at util/qemu-timer.c:334
#3  0x0000560d590360d5 in os_host_main_loop_wait (timeout=1675788861) at util/main-loop.c:233
#4  0x0000560d590360d5 in main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:497
#5  0x0000560d58cfda57 in main_loop () at vl.c:1981
#6  0x0000560d58cfda57 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at vl.c:4820
(gdb)

--- Additional comment from Kevin Buettner on 2020-02-11 01:00:26 UTC ---



--- Additional comment from Kevin Buettner on 2020-02-11 01:05:41 UTC ---

I was unable to create an environment for reproducing the bug on either RHEL 8.1 or RHEL 8.2. That said, I believe the problem exists in both 8.1 and 8.2; I just wasn't able to use virt-manager to create suitable VMs for testing.

I WAS able to reproduce the problem using Fedora 31.  Using virt-manager on Fedora 31, I created two VMs, one running RHEL 8.1 with firmware set to BIOS and the other with firmware set to OVMF (UEFI x86_64: /usr/share/edk2/ovmf/OVMF_CODE.fd). I then made these VMs run stand-alone, without the libvirt framework.  With that in place, I followed the instructions in the "Steps to reproduce" section of the bug report. I observed that the dump-guest-memory.py script was able to create a guest core dump from within GDB using a core dump from running the RHEL 8.1 BIOS VM.  When I tried the same thing using a core dump obtained from running the RHEL 8.1 OVMF VM, I observed the python exception noted in the bug report.

However, when I attempt to dump the guest memory from GDB that's attached to the running QEMU process (which is running the RHEL 8.1 OVMF VM), the dump-guest-memory script works as expected. After poking around some more, I confirmed that the memory region in question is absent from the core file, but was originally accessible when the process was running.

Again using GDB, I debugged the linux kernel on the virtualization host. This kernel was responsible for making the core dump of QEMU running the RHEL 8.1 OVMF VM.  Specifically, I looked at vma_dump_size() in fs/binfmt_elf.c to see why the memory region in question was not being dumped. When I stepped through this chunk of code for the problematic memory region...

	/* Dump segments that have been written to.  */
	if (vma->anon_vma && FILTER(ANON_PRIVATE))
		goto whole;
	if (vma->vm_file == NULL)
		return 0;

...I found that the "return 0" statement was being taken, which in turn causes the region in question to not be dumped.

I took another look at the memory region (from within GDB attached to the running QEMU process running RHEL 8.1 OVMF) and found that it was all 0. It seems likely that the region in question had been allocated, but never written to, so I think it makes sense for the kernel to not dump this region.

That being the case, the memory region in question is not available to GDB or the dump-guest-memory.py script, running within GDB. Thus, the behavior of GDB, throwing the exception for inaccessible memory, is correct.

I've made a minor modification to dump-guest-memory.py which causes it to write out zeroes for inaccessible memory.  As noted above, in this case, the memory is inaccessible due to the kernel choosing to not include it in the QEMU core dump.  I've added this patch as an attachment.

In conclusion, this problem is not a GDB bug. It might be argued that it's a kernel bug, but as described earlier I think the kernel's behavior in the case makes sense. Therefore, I think the bug is actually in the dump-guest-memory.py script which doesn't handle the case of the allocated (zeroed), but never-written-to region not being placed in the core dump.

--- Additional comment from Kevin Buettner on 2020-02-11 04:10:06 UTC ---

(In reply to Xueqiang Wei from comment #2)
> With the same environment mentioned in Comment 1, tested with
> dump-guest-core=off.
> 
> If set dump-guest-core=off: 1. hit this issue with seabios  2. hit this
> issue with edk2

FWIW, dump-guest-memory.py won't work correctly with "dump-guest-core=off".  When the "off" setting is used, QEMU calls madvise(...,...,MADV_DONT_DUMP) for the guest memory regions.  So, when you cause a core dump to occur, those pages won't be dumped, leading all of the guest's memory being inaccessible.

--- Additional comment from Laszlo Ersek on 2020-02-11 08:56:18 UTC ---

Per Kevin Buettner's analysis in <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c4>, the host kernel's process coredump logic is what decides to omit the never-written-to region from the qemu process coredump. The python script should expect and handle this possibility.

Kevin proposed a patch in <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c3>; we should upstream it and backport it.

Moving this BZ back to qemu-kvm.

--- Additional comment from Marc-Andre Lureau on 2020-02-14 17:47:24 UTC ---

Kevin, can you send the patch to qemu-devel? thanks

--- Additional comment from Kevin Buettner on 2020-02-14 19:51:46 UTC ---

(In reply to Marc-Andre Lureau from comment #7)
> Kevin, can you send the patch to qemu-devel? thanks

Yes, I can (and will) do that.

--- Additional comment from Kevin Buettner on 2020-02-14 23:32:02 UTC ---

(In reply to Marc-Andre Lureau from comment #7)
> Kevin, can you send the patch to qemu-devel? thanks

Done.  See:

https://lists.nongnu.org/archive/html/qemu-devel/2020-02/msg03875.html

--- Additional comment from John Ferlan on 2020-02-15 13:13:05 UTC ---

This probably needs to move to RHEL AV 8.2.0 and then decide if we should clone for RHEL 8.2.0 - we can decide that later though

--- Additional comment from Marc-Andre Lureau on 2020-02-27 17:35:43 UTC ---

Keith, could you comment on https://lists.nongnu.org/archive/html/qemu-devel/2020-02/msg03903.html ?
thanks

--- Additional comment from Kevin Buettner on 2020-03-05 00:55:46 UTC ---

I've done further investigations and have found that there is a bug in BFD and GDB.

I've filed an upstream bug which may be found here:

https://sourceware.org/bugzilla/show_bug.cgi?id=25631

A patch series fixing this bug can be found starting here:

https://sourceware.org/ml/gdb-patches/2020-03/msg00106.html

(Though it's likely that several iterations will be necessary for the series to be approved.)

I've tested a GDB build using my patches against a QEMU core file and found that the original dump-guest-memory script works as expected.

--- Additional comment from Laszlo Ersek on 2020-03-05 13:56:55 UTC ---

Thank you, Kevin -- does that mean we should move this BZ (and bug 1777751) to a different component? Thanks.

--- Additional comment from Kevin Buettner on 2020-03-05 14:04:01 UTC ---

(In reply to Laszlo Ersek from comment #13)
> Thank you, Kevin -- does that mean we should move this BZ (and bug 1777751)
> to a different component? Thanks.

Yes, they should both be moved to gdb.

--- Additional comment from Laszlo Ersek on 2020-03-05 15:44:01 UTC ---

Moving to gdb per <https://bugzilla.redhat.com/show_bug.cgi?id=1785126#c14>. Thanks!

--- Additional comment from Keith Seitz on 2020-03-09 20:52:02 UTC ---

(In reply to Marc-Andre Lureau from comment #11)
> Keith, could you comment on
> https://lists.nongnu.org/archive/html/qemu-devel/2020-02/msg03903.html ?
> thanks

I apologize for my disappearance -- I've been out of the office for several
weeks, and I am just making may way through my email and bug backlog...

I think Kevin has everything under control, even having found the bits in BFD
about which I was curious and investigating. [Thanks, Kevin!]

--- Additional comment from Keith Seitz on 2020-06-15 19:37:14 UTC ---

Moving to 8.4.