Info: ABRT is used in RHEL6+ to collect coredumps, backtraces and python exception. Currently there are not abrt daemon in the RHEV-H image and there is not way to collect dumps without custom monitoring tools. Solution: 1) Ship RHEV-H with abrt daemon (enabled by default or configured from the RHEV-H configuration dialog). 2) Create an abrt hook to send reports (?? using vdsm (add record abrts: X to vdsCapabilities?)) This step will give admin way to monitor crashes and avoid log re-writing. 3) (Ideal) Add ovirt UI integration to display abrt event on the event tab and integration with oVirt Monitoring UI-Plugin for case opening. Additional info: going to create upstream wikipage
libvirt and qemu troubleshooting would greatly benefit from the ability to provide stacktraces on crash without necessarily storing the core dump
*** Bug 1220860 has been marked as a duplicate of this bug. ***
especially interesting is the ability to get a stack trace without storing the actual core dump as we do not want to store the huge qemu's dumps
From my POV I only care about getting stack traces and perhaps configuring storing the relevant log files on a libvirt/qemu/vdsm crash
Oved / Yaniv B. - where's the design for the feature? It's not even clear it's for ovirt-engine or VDSM.
(In reply to Yaniv Kaul from comment #11) > Oved / Yaniv B. - where's the design for the feature? It's not even clear > it's for ovirt-engine or VDSM. The details will be specified in the bugzilla and patch. The work was started by Yeela in sync with Dan in the past. It is in VDSM.
(In reply to Michal Skrivanek from comment #6) > especially interesting is the ability to get a stack trace without storing > the actual core dump as we do not want to store the huge qemu's dumps This has been fixed on platform side already, so we're good to go bug 1210601
Is there a VDSM patch for this in progress?
(In reply to Eyal Edri from comment #14) > Is there a VDSM patch for this in progress? There was work Yeela started, but nothing stable. Why?
Michal asked Pavel's help with this, so I just wanted to get an idea on timelines and effort needed.
Hello, is there anything else in ABRT that is preventing you from pushing this RFE forward? We would love to see ABRT integrated into RHEV and I believe virtualization folks will be delighted too. How could we help you to make this happen?
(In reply to Jakub Filak from comment #17) > Hello, > is there anything else in ABRT that is preventing you from pushing this RFE > forward? We would love to see ABRT integrated into RHEV and I believe > virtualization folks will be delighted too. How could we help you to make > this happen? Yes, it would be definitely great if we can get crash reports for libvirt/qemu parts.
Please provide any info about the feature. Description in #0 is 3 years old and the rest seems quite vague. Thanks!
Seems like this will be left out of 4.1.
*** Bug 1421589 has been marked as a duplicate of this bug. ***
The patch is ready https://gerrit.ovirt.org/#/c/44287/. Vdsm now installs abrt plugins to collect crashes on SIGSEGV for binaries and python processes. This should replace codedumps The admin can still unlimit coredump ulimit file size and get also coredumps under /var/log/core. For any additional requests regarding the reports and abrt configuration please reply soon. The sos output will look like that: id 9931b9c7465904c050211c75dcc20bd5dedcd419 reason: python2.7 killed by SIGSEGV time: Tue 21 Mar 2017 04:22:07 PM IST cmdline: /usr/bin/python2 /usr/share/vdsm/vdsmd package: vdsm-4.20.0-522.gitc7659ab.el7.centos uid: 36 (vdsm) count: 1 Directory: /var/tmp/abrt/ccpp-2017-03-21-16:22:07-14260 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/fb0e8e1ebb657e4f8529c17a4b192d41a1399395 id a2e805102d864442327b250421076a9a77034bc9 reason: qemu-kvm killed by SIGSEGV time: Mon 20 Mar 2017 05:00:08 PM IST cmdline: /usr/libexec/qemu-kvm -name guest=asdfsadf,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-asdfsadf/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Penryn -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 9f7ac0fd-f8ce-48bd-9c51-654f1b78c111 -smbios 'type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=44454C4C-3200-104E-804A-CAC04F58344A,uuid=9f7ac0fd-f8ce-48bd-9c51-654f1b78c111' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-asdfsadf/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-20T14:55:52,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000001-0001-0001-0001-000000000311/8994ef29-9488-4de6-8c8a-d5d83aa5e8fb/images/50cbcf14-b3a6-4bde-aa01-cb48d36267c7/3cad8872-fe64-4519-8064-60bbcbfa7f58,format=raw,if=none,id=drive-scsi0-0-0-0,serial=50cbcf14-b3a6-4bde-aa01-cb48d36267c7,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/rhev/data-center/00000001-0001-0001-0001-000000000311/8994ef29-9488-4de6-8c8a-d5d83aa5e8fb/images/32868824-c26a-45d0-b706-4228843afaab/d6b62897-a66a-4426-893f-7b4c01d48dea,format=raw,if=none,id=drive-scsi0-0-0-2,serial=32868824-c26a-45d0-b706-4228843afaab,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-0-2,id=scsi0-0-0-2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/9f7ac0fd-f8ce-48bd-9c51-654f1b78c111.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/9f7ac0fd-f8ce-48bd-9c51-654f1b78c111.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5902,addr=10.35.0.135,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on package: qemu-kvm-ev-2.6.0-28.el7_3.6.1 uid: 107 (qemu) count: 1 Directory: /var/tmp/abrt/ccpp-2017-03-20-17:00:08-14682 id 58383d1eea730397f0d6e16044159b04ee8196bd reason: sleep killed by SIGSEGV time: Mon 20 Mar 2017 02:07:36 PM IST cmdline: sleep 10m package: coreutils-8.22-18.el7 uid: 0 (root) count: 1 Directory: /var/tmp/abrt/ccpp-2017-03-20-14:07:36-5119 under /var/tmp/abrt/[report] you can find info about the environment that can assist for investigation: abrt_version cgroup core_backtrace environ global_pid last_occurrence maps os_release pkg_arch pkg_release proc_pid_status reported_to type username analyzer cmdline count event_log hostname limits open_fds package pkg_epoch pkg_vendor pwd runlevel uid uuid architecture component dso_list executable kernel machineid os_info pid pkg_name pkg_version reason time ureports_counter var_log_messages
@Yaniv, are those abbrt files included within SOS collect archive? @Ryan, will we need some changes on NGN side to support abrt instead of core dumps by default?
https://gerrit.ovirt.org/#/c/74416/ - this is for the sos report part
I'll see what's needed to add abrt. As far as I know, we should be most of the way there already (correct BUGZILLA_URL, etc), but there's probably some pieces missing
(In reply to Ryan Barry from comment #28) > I'll see what's needed to add abrt. > > As far as I know, we should be most of the way there already (correct > BUGZILLA_URL, etc), but there's probably some pieces missing Ryan, could you please confirm that everything is fine from NGN point of view? We would like to backport this RFE into 4.1.3
We're ready to go from the NGN side
there's neither abrt-cli installed by default on RHV nor EL-based host. - el 7.3 vdsm-4.19.19-1.el7ev.x86_64 - rhvh-4.1-0.20170609.0+1 imo if we want this to work out-of-the-box then either abrt-cli should be pushed in as vdsm dep or there should be a workaround for non-existence of abrt-cli. # sosreport ... # tar xOJf /var/tmp/sosreport-dell-r210ii-03.example.com-20170621142503.tar.xz sosreport-dell-r210ii-03.example.com-20170621142503/sos_logs/sos.log | grep abrt 2017-06-21 14:25:03,363 INFO: [plugin:abrt] added cmd output 'abrt-cli -lf' 2017-06-21 14:25:04,066 INFO: [plugin:vdsm] added cmd output '/usr/bin/abrt-cli list' 2017-06-21 14:25:05,132 INFO: [plugin:abrt] collecting output of 'abrt-cli -lf' 2017-06-21 14:25:05,135 INFO: [plugin:abrt] command 'abrt-cli' not found in / - re-trying in host root 2017-06-21 14:25:52,793 INFO: [plugin:vdsm] collecting output of '/usr/bin/abrt-cli list' 2017-06-21 14:25:52,796 INFO: [plugin:vdsm] command '/usr/bin/abrt-cli' not found in / - re-trying in host root # ls /var/tmp/abrt/ ccpp-2017-06-21-14:22:44-27308 last-ccpp # rpm -qa redhat-virtualization\* redhat-virtualization-host-image-update-placeholder-4.1-3.2.el7.noarch redhat-virtualization-host-image-update-4.1-20170609.2.el7_3.noarch
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
ohhh in all my tests, requiring abrt-addon-* pulls also abrt-cli. probably for rhvh we need to explicitly require abrt-cli. posting patch to require abrt-cli specifically
(In reply to Yaniv Bronhaim from comment #37) > ohhh in all my tests, requiring abrt-addon-* pulls also abrt-cli. probably > for rhvh we need to explicitly require abrt-cli. posting patch to require > abrt-cli specifically just a note - this is not RHVH/node-ng specific. EL host doesn't have abrt-cli as well.
This relates to new sos output - it doesn't effect anything, only require the admin or the sos user to install abrt-cli if wants to retrieve abrt output as well in the report. other than that everything still works well with that integration.
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Tag 'v4.19.21' doesn't contain patch 'https://gerrit.ovirt.org/75659'] gitweb: https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=shortlog;h=refs/tags/v4.19.21 For more info please contact: infra
Yaniv, please check whether it is in the build, and if so move to ON_QA.
This is probably a bug. The patch was merged quite long time ago and I verified that its there. btw, its not related to the last fix (the requirement for abrt-cli which is also part of 4.19.21)
fail. it is not 'abrt-cli -lf' but 'abrt-cli list' or 'abrt-cli ls'. it seems 'abrt-cli -lf' could be syntax from ancient fedora package[1]: # tar xOJf /var/tmp/sosreport-localhost.localdomain-20170714120813.tar.xz sosreport-localhost.localdomain-20170714120813/sos_logs/sos.log | grep abrt 2017-07-14 12:08:13,017 INFO: [plugin:abrt] added cmd output 'abrt-cli -lf' 2017-07-14 12:08:13,278 INFO: [plugin:abrt] collecting output of 'abrt-cli -lf' # tar xOJf /var/tmp/sosreport-localhost.localdomain-20170714120813.tar.xz sosreport-localhost.localdomain-20170714120813/sos_commands/abrt/abrt-log | cat - Usage: abrt-cli [--authenticate] [--version] COMMAND [DIR]... # cat /etc/redhat-release ; abrt-cli -lf ; abrt-cli --help Red Hat Enterprise Linux Server release 7.3 (Maipo) Usage: abrt-cli [--authenticate] [--version] COMMAND [DIR]... Usage: abrt-cli [--authenticate] [--version] COMMAND [DIR]... list, ls List problems [in DIRs] remove, rm Remove problem directory DIR report, e Analyze and report problem data in DIR info, i Print information about DIR status, st Print the count of the recent crashes process, p Process multiple problems See 'abrt-cli COMMAND --help' for more information [1] https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/System_Administrators_Guide/sect-abrt-cli.html
What are you referring to? We're using only "abrt-cli list" in lib/sos/vdsm.py.in: self.collectExtOutput("/usr/bin/abrt-cli list") I think you're mistaken - please ping me before reopening, it might require new bug but the RFE implementation was verified and closed already, so I don't understand why you reopen it instead of opening new issue if you find something.. especially if its sos issue. Changing to ON_QA back - if the integration is fine please move to VERIFIED and open new bug on vdsm sos reports, although I think you added some manual code to your env that causes your issue.
(In reply to Yaniv Bronhaim from comment #46) > What are you referring to? ok that 'abrt-cli -lf' is from sos's abrt plugin. > We're using only "abrt-cli list" in > lib/sos/vdsm.py.in: self.collectExtOutput("/usr/bin/abrt-cli list") > I think you're mistaken - please ping me before reopening Yes, stated above. > if the > integration is fine please move to VERIFIED and open new bug on vdsm sos > reports, although I think you added some manual code to your env that causes > your issue. Unfortunately mistake discovered an issue in vdsm sos plugin: There's exception which is visible in sosreport -v: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1247, in setup plug.setup() File "/usr/lib/python2.7/site-packages/sos/plugins/vdsm.py", line 92, in setup self.addCopySpec("/var/log/vdsm/*", logsize) TypeError: add_copy_spec() takes exactly 2 arguments (3 given) # sed -n '92p' /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py self.addCopySpec("/var/log/vdsm/*", logsize) This causes failure of continuation of vdsm plugin run: # find /tmp/sosreport-dell-r210ii-04.rhev.lab.eng.brq.redhat.com-20170717093653/ -type f | xargs grep 'added cmd output' /tmp/sosreport-dell-r210ii-04.rhev.lab.eng.brq.redhat.com-20170717093653/sos_logs/sos.log:2017-07-17 09:36:53,370 INFO: [plugin:vdsm] added cmd output 'service vdsmd status' ^^ last cmd output # egrep -n "(addCopySpec.*logsize|collectExtOutput)" /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py 69: collectExtOutput = Plugin.add_cmd_output 87: self.collectExtOutput("service vdsmd status") ^^ last cmd definition 92: self.addCopySpec("/var/log/vdsm/*", logsize) ^^ failure 106: self.collectExtOutput("/bin/ls -l /var/log/core") 107: self.collectExtOutput("/bin/ls -ldZ /etc/vdsm") 108: self.collectExtOutput( 110: self.collectExtOutput( 112: self.collectExtOutput("/sbin/lvm vgs -v -o +tags") 113: self.collectExtOutput("/sbin/lvm lvs -v -o +tags") 114: self.collectExtOutput("/sbin/lvm pvs -v -o +all") 115: self.collectExtOutput("/sbin/fdisk -l") 116: self.collectExtOutput("/usr/bin/iostat") 117: self.collectExtOutput("/sbin/iscsiadm -m node") 118: self.collectExtOutput("/sbin/iscsiadm -m session") 119: self.collectExtOutput("/usr/sbin/nodectl info") 120: self.collectExtOutput("/usr/bin/abrt-cli list") 154: self.collectExtOutput("vdsm-tool dump-volume-chains %s" % sd_uuid) ^^ all these are not executed Discovered while being curious why no 'abrt-cli list' output exists from vdsm plugin: # abrt-cli list -n | grep ^reason reason: qemu-kvm killed by SIGSEGV
How is this related to this bugzilla Jiri? Please open discussion in separate bug
(In reply to Yaniv Bronhaim from comment #48) > How is this related to this bugzilla Jiri? Please open discussion in > separate bug OK, a BZ for repairing vdsm plugin BZ1471663 which blocks sos part of this RFE.
ok, vdsm-4.19.23-1.el7ev.x86_64 * el 7.3 Jul 19 10:47:10 dell-r210ii-04 abrt-hook-ccpp: Process 6603 (qemu-kvm) of user 107 killed by SIGSEGV - dumping core # abrt-cli list id f75b54341dd12028b3d32a93d2b17d15c81615f0 reason: qemu-kvm killed by SIGSEGV time: Wed 19 Jul 2017 10:47:10 AM CEST cmdline: /usr/libexec/qemu-kvm -name guest=jbelka_test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-jbelka_test1/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu SandyBridge -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb -smbios 'type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=7.3-7.el7,serial=4C4C4544-0058-3410-8058-C3C04F38354A,uuid=3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-jbelka_test1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T08:46:22,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/311e2347-b337-4907-8099-0438d7305d8a/bb1b6832-614a-4edc-95fa-b95e7c01d052/images/c72fd8a8-4ca8-42f8-9afc-a6bb4e392118/a8c75cf4-cacc-4a3c-9e02-6aa12c8f41b7,format=raw,if=none,id=drive-scsi0-0-0-0,serial=c72fd8a8-4ca8-42f8-9afc-a6bb4e392118,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:51,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.34.63.223,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on package: qemu-kvm-rhev-2.6.0-28.el7_3.12 uid: 107 (qemu) count: 1 Directory: /var/tmp/abrt/ccpp-2017-07-19-10:47:10-6603 Run 'abrt-cli report /var/tmp/abrt/ccpp-2017-07-19-10:47:10-6603' for creating a case in Red Hat Customer Portal # find /var/log/core/ /var/log/core/ * el 7.4 Jul 19 11:00:12 dell-r210ii-13 abrt-hook-ccpp: Process 30597 (qemu-kvm) of user 107 killed by SIGSEGV - dumping core # abrt-cli list id d47e496fee53bdb95bbfde35a8e935670f3ac548 reason: qemu-kvm killed by SIGSEGV time: Wed 19 Jul 2017 11:00:12 AM CEST cmdline: /usr/libexec/qemu-kvm -name guest=jbelka_test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-jbelka_test1/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,dump-guest-core=off -cpu SandyBridge -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb -smbios 'type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=7.3-7.el7,serial=4C4C4544-0058-3410-8058-C3C04F38354A,uuid=3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-jbelka_test1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T08:58:36,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/311e2347-b337-4907-8099-0438d7305d8a/bb1b6832-614a-4edc-95fa-b95e7c01d052/images/c72fd8a8-4ca8-42f8-9afc-a6bb4e392118/a8c75cf4-cacc-4a3c-9e02-6aa12c8f41b7,format=raw,if=none,id=drive-scsi0-0-0-0,serial=c72fd8a8-4ca8-42f8-9afc-a6bb4e392118,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:51,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.34.62.205,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on package: qemu-kvm-rhev-2.9.0-16.el7_4.2 uid: 107 (qemu) count: 1 Directory: /var/tmp/abrt/ccpp-2017-07-19-11:00:12-30597 Run 'abrt-cli report /var/tmp/abrt/ccpp-2017-07-19-11:00:12-30597' for creating a case in Red Hat Customer Portal # find /var/log/core/ /var/log/core/ sosreport -o vdsm works find as well (BZ1471663).
* rhvh (WIP) for 4.1.4 # cat /proc/sys/kernel/core_pattern |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i # systemctl list-dependencies vdsmd | head -n2 vdsmd.service ● ├─abrtd.service # pkill -11 qemu-kvm # abrt-cli list id 499357d67f32bfdf5d85306fcc114a6e0584f36c reason: qemu-kvm killed by SIGSEGV time: Wed 19 Jul 2017 12:16:52 PM CEST cmdline: /usr/libexec/qemu-kvm -name guest=jbelka_test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-jbelka_test1/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,dump-guest-core=off -cpu SandyBridge -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb -smbios 'type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=7.3-7.el7,serial=4C4C4544-0058-3410-8058-C3C04F38354A,uuid=3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-jbelka_test1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T10:15:55,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/311e2347-b337-4907-8099-0438d7305d8a/bb1b6832-614a-4edc-95fa-b95e7c01d052/images/c72fd8a8-4ca8-42f8-9afc-a6bb4e392118/a8c75cf4-cacc-4a3c-9e02-6aa12c8f41b7,format=raw,if=none,id=drive-scsi0-0-0-0,serial=c72fd8a8-4ca8-42f8-9afc-a6bb4e392118,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:51,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.34.62.205,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on package: qemu-kvm-rhev-2.9.0-12.el7 uid: 107 (qemu) Directory: /var/tmp/abrt/ccpp-2017-07-19-12:16:52-27219 Run 'abrt-cli report /var/tmp/abrt/ccpp-2017-07-19-12:16:52-27219' for creating a case in Red Hat Customer Portal # imgbase w 2017-07-19 12:17:06,676 [INFO] You are on rhvh-4.1-0.20170714.0+1 # tar xOJf /var/tmp/sosreport-localhost.localdomain-20170719122006.tar.xz sosreport-localhost.localdomain-20170719122006/sos_commands/vdsm/abrt-cli_list ... id 499357d67f32bfdf5d85306fcc114a6e0584f36c reason: qemu-kvm killed by SIGSEGV time: Wed Jul 19 12:16:52 2017 cmdline: /usr/libexec/qemu-kvm -name guest=jbelka_test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-jbelka_test1/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,dump-guest-core=off -cpu SandyBridge -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb -smbios 'type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=7.3-7.el7,serial=4C4C4544-0058-3410-8058-C3C04F38354A,uuid=3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-jbelka_test1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-07-19T10:15:55,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/311e2347-b337-4907-8099-0438d7305d8a/bb1b6832-614a-4edc-95fa-b95e7c01d052/images/c72fd8a8-4ca8-42f8-9afc-a6bb4e392118/a8c75cf4-cacc-4a3c-9e02-6aa12c8f41b7,format=raw,if=none,id=drive-scsi0-0-0-0,serial=c72fd8a8-4ca8-42f8-9afc-a6bb4e392118,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:51,bus=pci.0,addr=0x3,bootindex=2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/3dd7f023-9ed2-4ea2-8434-7c0d6ff07feb.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.34.62.205,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on package: qemu-kvm-rhev-2.9.0-12.el7 uid: 107 (qemu) count: 1 Directory: /var/tmp/abrt/ccpp-2017-07-19-12:16:52-27219 Run 'abrt-cli report /var/tmp/abrt/ccpp-2017-07-19-12:16:52-27219' for creating a case in Red Hat Customer Portal
for each vm crash report we should see under /var/tmp/abrt folder with all crash related output. one of the file is core_backtrace : # cat core_backtrace { "signal": 11 , "executable": "/usr/libexec/qemu-kvm" , "stacktrace": [ { "crash_thread": true , "frames": [ { "address": 140274758388479 , "build_id": "c3f28802314af4ee866bf8d2e1b506b7bbf34cf6" , "build_id_offset": 973567 , "function_name": "ppoll" , "file_name": "/usr/lib64/libc-2.17.so" } , { "address": 94478079266713 , "build_id": "250731da86353c7dda890a58874c5eb9137f95fc" , "build_id_offset": 6033305 , "function_name": "qemu_poll_ns" , "file_name": "/usr/libexec/qemu-kvm" } , { "address": 94478079270312 , "build_id": "250731da86353c7dda890a58874c5eb9137f95fc" , "build_id_offset": 6036904 , "function_name": "main_loop_wait" , "file_name": "/usr/libexec/qemu-kvm" } , { "address": 94478076064140 , "build_id": "250731da86353c7dda890a58874c5eb9137f95fc" , "build_id_offset": 2830732 , "function_name": "main" , "file_name": "/usr/libexec/qemu-kvm" } ] } ] } we disabled MakeCompatCore to make the output small as possible - if we need that information of the compact core we need to configure that variable to yes