Bug 552474
| Summary: | Migrated guest is dead and segfault is found in host dmesg | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Yolkfull Chow <yzhou> | ||||
| Component: | kvm | Assignee: | Juan Quintela <quintela> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 5.5 | CC: | llim, michen, qzhang, tburke, virt-maint, ykaul | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2010-01-21 13:04:02 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 556823 | ||||||
| Attachments: |
|
||||||
Please provide the backtrace from the core dump (gdb) dir /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/ Source directories searched: /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu:$cdir:$cwd (gdb) thread apply all bt full Thread 5 (Thread 0x42803940 (LWP 3709)): #0 0x00002aaaae646744 in do_sigwaitinfo () from /lib64/libc.so.6 No symbol table info available. #1 0x00002aaaae6467fd in sigwaitinfo () from /lib64/libc.so.6 No symbol table info available. #2 0x000000000041a291 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #3 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x41401940 (LWP 3707)): #0 0x00002aaaad45f2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00002aaaad45ac3a in _L_lock_1034 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00002aaaad45aafc in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000000004fe454 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #4 0x00000000004feab4 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #5 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x40a00940 (LWP 3706)): #0 0x00002aaaae646744 in do_sigwaitinfo () from /lib64/libc.so.6 No symbol table info available. #1 0x00002aaaae6467fd in sigwaitinfo () from /lib64/libc.so.6 No symbol table info available. #2 0x000000000041a291 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #3 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x2aaab2375080 (LWP 3697)): #0 0x0000000000429075 in snd_pcm_hw_params_set_channels_near () No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #1 0x000000000042d304 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #2 0x0000000000472460 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #3 0x000000000046b70c in snd_pcm_hw_params_set_channels_near () No symbol table info available. #4 0x000000000046bb08 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #5 0x0000000000409502 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #6 0x00000000004fe6fa in snd_pcm_hw_params_set_channels_near () No symbol table info available. #7 0x000000000040e445 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #8 0x00002aaaae632994 in __libc_start_main () from /lib64/libc.so.6 No symbol table info available. #9 0x0000000000406cc9 in snd_pcm_hw_params_set_channels_near () No symbol table info available. #10 0x00007fffffffe748 in ?? () No symbol table info available. #11 0x0000000000000000 in ?? () No symbol table info available. # grep -r "snd_pcm_hw_params_set_channels_near" ./* ./audio/alsaaudio.c: err = snd_pcm_hw_params_set_channels_near ( (In reply to comment #2) > (gdb) dir /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/ > Source directories searched: > /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu:$cdir:$cwd > (gdb) thread apply all bt full > > Thread 5 (Thread 0x42803940 (LWP 3709)): > #0 0x00002aaaae646744 in do_sigwaitinfo () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00002aaaae6467fd in sigwaitinfo () from /lib64/libc.so.6 > No symbol table info available. > #2 0x000000000041a291 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #3 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #4 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 3 (Thread 0x41401940 (LWP 3707)): > #0 0x00002aaaad45f2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > No symbol table info available. > #1 0x00002aaaad45ac3a in _L_lock_1034 () from /lib64/libpthread.so.0 > No symbol table info available. > #2 0x00002aaaad45aafc in pthread_mutex_lock () from /lib64/libpthread.so.0 > No symbol table info available. > #3 0x00000000004fe454 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #4 0x00000000004feab4 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #5 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #6 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 2 (Thread 0x40a00940 (LWP 3706)): > #0 0x00002aaaae646744 in do_sigwaitinfo () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00002aaaae6467fd in sigwaitinfo () from /lib64/libc.so.6 > No symbol table info available. > #2 0x000000000041a291 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #3 0x00002aaaad458617 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #4 0x00002aaaae6e8d1d in clone () from /lib64/libc.so.6 > No symbol table info available. > > Thread 1 (Thread 0x2aaab2375080 (LWP 3697)): > #0 0x0000000000429075 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > ---Type <return> to continue, or q <return> to quit--- > #1 0x000000000042d304 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #2 0x0000000000472460 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #3 0x000000000046b70c in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #4 0x000000000046bb08 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #5 0x0000000000409502 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #6 0x00000000004fe6fa in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #7 0x000000000040e445 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #8 0x00002aaaae632994 in __libc_start_main () from /lib64/libc.so.6 > No symbol table info available. > #9 0x0000000000406cc9 in snd_pcm_hw_params_set_channels_near () > No symbol table info available. > #10 0x00007fffffffe748 in ?? () > No symbol table info available. > #11 0x0000000000000000 in ?? () > No symbol table info available. > > > # grep -r "snd_pcm_hw_params_set_channels_near" ./* > ./audio/alsaaudio.c: err = snd_pcm_hw_params_set_channels_near ( Seems sound related problem. 0. I'm not sure why you don't have symbols. You should have them. Did you install debuginfo? 1. What's your audio device? You do not seem to be specifying it in the command line (and I'm not sure what the default is) 2. It should be AC97. Please make sure desktop OSes (XP, Win7) all have this option. 3. If it is AC97, note that Win7 (at least the 32bit I test) does NOT have a inbox driver for the device - it downloads it via Windows Update! Do you happen to know if the VM had a chance to install the driver or not? (In reply to comment #4) > 0. I'm not sure why you don't have symbols. You should have them. Did you > install debuginfo? Sure. We can get to know it from first line of comment #2. > > 1. What's your audio device? You do not seem to be specifying it in the command > line (and I'm not sure what the default is) I'm not sure. The job was running on a remote machine. Haven't got a chance to check it. > 2. It should be AC97. Please make sure desktop OSes (XP, Win7) all have this > option. > 3. If it is AC97, note that Win7 (at least the 32bit I test) does NOT have a > inbox driver for the device - it downloads it via Windows Update! Do you happen > to know if the VM had a chance to install the driver or not? I didn't know whether the VM has installed the driver but I guess not. Anything further testing I can do? Thanks for quick response. :) I tested this with guest Win2008r2, can reproduce as well. Would delete 'win7' from bug summary. (In reply to comment #6) > I tested this with guest Win2008r2, can reproduce as well. Would delete 'win7' > from bug summary. BTW, the kvm version is: kvm-83-143.el5 (In reply to comment #4) > 0. I'm not sure why you don't have symbols. You should have them. Did you > install debuginfo? > > 1. What's your audio device? You do not seem to be specifying it in the command > line (and I'm not sure what the default is) > 2. It should be AC97. Please make sure desktop OSes (XP, Win7) all have this > option. Tested that adding option '-soundhw ac97' doesn't help, problem still exists and segfault could be found. And what I'm interested in is how the internal guest sound driver affects external qemu process? In my opinion, if the sound driver is not installed in guest OS, it will not be loaded into guest memory therefore will not affect migration at all. Please correct me if I'm wrong. > 3. If it is AC97, note that Win7 (at least the 32bit I test) does NOT have a > inbox driver for the device - it downloads it via Windows Update! Do you happen > to know if the VM had a chance to install the driver or not? Tested it on kvm-83-139, both with option '-soundhw ac97' and without that option cannot reproduce this problem. Apparently it's a REGRESSION. I met the different segfault when implementing migration. Command line I used: /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet -rtc-td-hack -smp 2 -m 2G -drive file=/root/win2k3-32-sp2.raw,media=disk,if=ide -net nic,vlan=0,macaddr=10:1a:4a:20:80:5f,model=rtl8139 -net tap,vlan=0,script=/etc/qemu-ifup -boot c -cpu qemu64,+sse2 -uuid `uuidgen` -monitor stdio -vnc :10 -name win2k3-32 No Segfault found in the source host. It only happens in the destination host. (qemu) Program received signal SIGSEGV, Segmentation fault. 0x0000000000429075 in ide_load (f=0x1e3d9c0, s=0x1e26550, version_id=3) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/hw/ide.c:2907 2907 s->bs->enable_write_cache = 0; (gdb) bt #0 0x0000000000429075 in ide_load (f=0x1e3d9c0, s=0x1e26550, version_id=3) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/hw/ide.c:2907 #1 0x000000000042d304 in pci_ide_load (f=0x1e3d9c0, opaque=0x1e26010, version_id=3) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/hw/ide.c:3285 #2 0x0000000000472460 in qemu_loadvm_state (f=0x1e3d9c0) at savevm.c:981 #3 0x000000000046b70c in tcp_incomming_run_hook () at migration-tcp.c:472 #4 0x000000000046bb08 in tcp_accept_incoming_migration ( opaque=<value optimized out>) at migration-tcp.c:561 #5 0x0000000000409502 in main_loop_wait (timeout=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:3981 #6 0x00000000004fe6fa in kvm_main_loop () at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/qemu-kvm.c:596 #7 0x000000000040e445 in main (argc=29, argv=0x7fff9c8693b8, envp=<value optimized out>) at /usr/src/debug/kvm-83-maint-snapshot-20090205/qemu/vl.c:4038 It looks like the segfault in Bug 550269 The kvm version I used is also kvm-83-142.el5. Retest migration with kvm-83-147.el5, this problem does not exist, migration works fine. Juan, what's the patch that made the change? I still don't know. Winxp works for me. Installed win7 and it also works for me (tm). I am using the latest 5.5 tree. [root@ninfa ~]# rpm -qa kvm kmod-kvm; uname -a kmod-kvm-83-147.el5 kvm-83-147.el5 Linux ninfa.mitica 2.6.18-185.el5 #1 SMP Thu Jan 14 16:44:40 EST 2010 x86_64 x86_64 x86_64 GNU/Linux [root@ninfa ~]# I am looking now at the differences between my command line and theirs. /me re-reads the whole thread. Ok. I don't know what patch made the change, I just read now Chen comment that -147 fixes the problem (that is the reason why I was not able to reproduce it). Searching. I ended the bisect. the backtrace show in previous comments is wrong. problem is not related with sound at all. This is a duplicate of bz549938. *** This bug has been marked as a duplicate of bug 549938 *** |
Created attachment 381721 [details] Full result log of autotest test case 'migrate' Description of problem: After migration of win7 guest, migrated guest is dead(could not even move mouse although can see login screen). And can found qemu segfault in dmesg: --- qemu[6802]: segfault at 0000000000000894 rip 0000000000429075 rsp 00007fff56b3fe80 error 6 --- And following is log of autotest test case 'migrate': ... 01/05 03:29:59 DEBUG| kvm_vm:0521| Sending monitor command: info migrate 01/05 03:30:18 DEBUG| kvm_utils:0829| Waiting for migration to finish... (193.434382 secs) 01/05 03:30:18 DEBUG| kvm_vm:0521| Sending monitor command: info migrate 01/05 03:30:30 DEBUG|kvm_subpro:0686| (qemu) migration_notify_done 01/05 03:30:30 DEBUG| kvm_vm:0521| Sending monitor command: info migrate 01/05 03:30:30 INFO |kvm_test_u:0160| Migration finished successfully 01/05 03:30:32 DEBUG|kvm_subpro:0686| (qemu) /bin/sh: line 1: 669 Segmentation fault /usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6002,server,nowait -drive file=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/images/RHEL-Server-5.4-32.raw,if=ide,cache=writethrough,boot=on -net nic,vlan=0,model=e1000,macaddr=00:19:30:09:8A:00 -net tap,vlan=0,ifname=e1000_0_6002,script=/usr/local/staf/test/RHEV/kvm-new/kvm-test/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 2048 -smp 1 -usbdevice tablet -rtc-td-hack -no-hpet -cpu qemu64,+sse2 -no-kvm-pit-reinjection -vnc :1 -incoming tcp:0:5200 01/05 03:30:32 DEBUG|kvm_subpro:0686| (qemu) (Process terminated with status 139) 01/05 03:30:34 DEBUG| kvm_utils:0139| freeing mac addr 00:19:30:09:8A:00 01/05 03:30:34 DEBUG| kvm_vm:0583| Destroying VM with PID 417... ... Version-Release number of selected component (if applicable): # rpm -qa |grep kvm kvm-83-142.el5 etherboot-zroms-kvm-5.4.4-12.el5 kvm-tools-83-142.el5 kvm-qemu-img-83-142.el5 kmod-kvm-83-142.el5 kvm-debuginfo-83-142.el5 host kernel: 2.6.18-183.el5 How reproducible: Everytime Steps to Reproduce: 1.Start guest: #/root/autotest/client/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6001,server,nowait -drive file=/root/autotest/client/tests/kvm/images/win7-32-virtio.qcow2,if=virtio,boot=on -net nic,vlan=0,model=virtio,macaddr=00:AE:9D:90:9C:00 -net tap,vlan=0,ifname=virtio_0_6001,script=/root/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 512 -smp 1 -cdrom /root/autotest/client/tests/kvm/isos/windows/winutils.iso -usbdevice tablet -rtc-td-hack -no-hpet -cpu qemu64,+sse2 -no-kvm-pit-reinjection -rtc-td-hack -vnc :0 & 2. Start guest in listen mode: # /root/autotest/client/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6002,server,nowait -drive file=/root/autotest/client/tests/kvm/images/win7-32-virtio.qcow2,if=virtio,boot=on -net nic,vlan=0,model=virtio,macaddr=00:AE:9D:90:9C:00 -net tap,vlan=0,ifname=virtio_0_6002,script=/root/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 512 -smp 1 -cdrom /root/autotest/client/tests/kvm/isos/windows/winutils.iso -usbdevice tablet -rtc-td-hack -no-hpet -cpu qemu64,+sse2 -no-kvm-pit-reinjection -rtc-td-hack -vnc :1 -incoming tcp:0:5200 & 3. Start migration 4. Check migrated guest, and issue 'dmesg' on host Actual results: Migrated guest is dead and found segfault in host dmesg output Expected results: Migrate successfully Additional info: