Bug 865767

Just to be clear, is the order of events to reproduce:
1. boot RHEL 6.3
2. launch X with qxl driver
3. reboot

?

Hi Gerd,

 Assigning this to you as it seems to be in code you know best,

Thanks,
Alon

*** Bug 865718 has been marked as a duplicate of this bug. ***

Looks like use-after-free or buffer overflow killed malloc data structures.
Does it reproduce outside autotest?

Hmm, didn't reproduce locally so far, even with a screendump loop like autotest does (stack traces look like this could be involved).

Any chance you can run the autotest job with the electricfence malloc debugger?

(1) install ElectricFence
(2) make sure EF_ALLOW_MALLOC_0 environment variable is set to 1
    dunno how to do that with autotest
(3) use ef wrapper script to start qemu
    qemu_binary = /usr/bin/ef /usr/libexec/qemu-kvm

This way we should get a stacktrace showing the place where the memory corruption actually happens rather than the place where malloc is tripped up by the corruption.

(In reply to comment #8)
> Looks like use-after-free or buffer overflow killed malloc data structures.
> Does it reproduce outside autotest?

Hmmm, I booted a vm manually,
tried 100 times of reboot, vm still alive and didn't core dump.
on qemu-kvm-rhev-0.12.1.2-2.323.el6.x86_64
cmd:

qemu-kvm -name RHEL.6.3.64.REBOOT -nodefaults -monitor stdio -chardev socket,id=serial_id_20120913-134744-NKxy,path=/tmp/serial-20120913-134744-NKxy,server,nowait -device isa-serial,chardev=serial_id_20120913-134744-NKxy -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 -drive file=/root/staf-kvm/autotest/client/tests/kvm/images/RHEL-Server-6.3-64-virtio.qcow2,if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idUbqACX,mac=9a:ef:9d:77:de:06,id=ndev00idUbqACX,bus=pci.0,addr=0x3 -netdev tap,id=idUbqACX,vhost=on -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu SandyBridge -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -spice port=3000,password=123456,addr=0,image-compression=auto_glz,jpeg-wan-compression=auto,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 -vga qxl -global qxl-vga.vram_size=33554432 -rtc base=utc,clock=host,driftfix=slew -M rhel6.4.0 -boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -bios /usr/share/seabios/bios-pm.bin -enable-kvm

(In reply to comment #9)
> Hmm, didn't reproduce locally so far, even with a screendump loop like
> autotest does (stack traces look like this could be involved).
> 
> Any chance you can run the autotest job with the electricfence malloc
> debugger?
> 
> (1) install ElectricFence
> (2) make sure EF_ALLOW_MALLOC_0 environment variable is set to 1
>     dunno how to do that with autotest
> (3) use ef wrapper script to start qemu
>     qemu_binary = /usr/bin/ef /usr/libexec/qemu-kvm
> 
> This way we should get a stacktrace showing the place where the memory
> corruption actually happens rather than the place where malloc is tripped up
> by the corruption.

Hi Gerd,

no knowing if this meets your requirement:
I append this line to tests.cfg
qemu_binary = ";EF_ALLOW_MALLOC_0=1 /usr/bin/ef `which qemu-kvm`"

then autotest will set that variable to 1 before launching the vm:
EF_ALLOW_MALLOC_0=1 /usr/bin/ef `which qemu-kvm` -name 'vm1' -nodefaults -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121017-173104-PFVf,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20121017-173104-PFVf,path=/tmp/serial-20121017-173104-PFVf,server,nowait -device isa-serial,chardev=serial_id_20121017-173104-PFVf -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 -drive file='/root/staf-kvm/autotest/client/tests/kvm/images/RHEL-Server-6.3-64-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idLUd7lP,mac=9a:a2:ad:6e:91:ca,id=ndev00idLUd7lP,bus=pci.0,addr=0x3 -netdev tap,id=idLUd7lP,vhost=on,fd=19 -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu 'Penryn' -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -spice port=3001,password=123456,addr=0,image-compression=auto_glz,jpeg-wan-compression=auto,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 -vga qxl -global qxl-vga.vram_size=33554432 -rtc base=utc,clock=host,driftfix=slew -M rhel6.4.0 -boot order=cdn,once=c,menu=off    -no-kvm-pit-reinjection -bios /usr/share/seabios/bios.bin -enable-kvm

and when the core dump happens, it prints:

17:43:13 INFO | [qemu output] /usr/bin/ef: line 20:  3835 Segmentation fault      (core dumped) ( export LD_PRELOAD=libefence.so.0.0; exec "$@" )
17:43:13 INFO | [qemu output] (Process terminated with status 139)

Looks good, try "tread apply all bt" on the core dump produced (autotest collects it, right?).

Created attachment 629143 [details]
thread apply all bt full

Created attachment 634904 [details]
backtrace info

Meet the same issue with the package qemu-kvm-0.12.1.2-2.331.el6.x86_64 in run autotest job;

Created attachment 635335 [details]
qemu-kvm backtrace

Hi, I guess I hit this issue also, qemu-kvm crashed when I boot a RHEL5.8 guest.

qemu version: qemu-kvm-0.12.1.2-2.331.el6.x86_64

Created attachment 635336 [details]
glibc backtrace report

Attach the glibc backtrace log in case someone like it.

(In reply to comment #14)
> Created attachment 634904 [details]
> backtrace info
> 
> Meet the same issue with the package qemu-kvm-0.12.1.2-2.331.el6.x86_64 in
> run autotest job;

Looks helpful.  Do you still have the core dump?  Can you upload it somewhere?

Given this happens in autotest I guess you don't need a spice client connected to trigger it, correct?

(In reply to comment #17)
> (In reply to comment #14)
> > Created attachment 634904 [details]
> > backtrace info
> > 
> > Meet the same issue with the package qemu-kvm-0.12.1.2-2.331.el6.x86_64 in
> > run autotest job;
> 
> Looks helpful.  Do you still have the core dump?  Can you upload it
> somewhere?

you can download it from http://fileshare.englab.nay.redhat.com/pub/section2/kvm/xu/bz865767/core

http://patchwork.ozlabs.org/patch/196184/

Patch posted.

*** Bug 873214 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0527.html