Bug 1596013
Summary: | sometime L1 qemu process core dump when rebooting L2 RHEL7.6 guest | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | FuXiangChun <xfu> | ||||||
Component: | qemu-kvm-rhev | Assignee: | Bandan Das <bdas> | ||||||
Status: | CLOSED DUPLICATE | QA Contact: | FuXiangChun <xfu> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 7.6 | CC: | chayang, choma, juzhang, michen, virt-maint, xfu | ||||||
Target Milestone: | rc | ||||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | |||||||||
: | 1599260 (view as bug list) | Environment: | |||||||
Last Closed: | 2018-07-10 17:22:34 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1599260 | ||||||||
Attachments: |
|
Description
FuXiangChun
2018-06-28 03:49:19 UTC
Is the backtrace from the qemu process inside L1 ? Because none of the threads in the backtrace show signs of a fault! Also, can you please post dmesg from host when this happens ? (In reply to Bandan Das from comment #2) > Is the backtrace from the qemu process inside L1 ? Because none of the > threads in the backtrace show signs of a fault! Also, can you please post > dmesg from host when this happens ? qemu process is from inside L1. sometimes system_power command also can trigger this bug. I will upload L1 host's dmesg and /var/log/message log to attachment. Created attachment 1455602 [details]
L1 dmesg log
Created attachment 1455603 [details]
part of message log
Thank you for the logs. Sorry I wasn't clear, can you please post the dmesg from L0 when the qemu in L1 aborts ? If the syscall in the first thread leads to abort, I am assuming whatever's happening in the kernel should print a message as well. I reproduced it again. This is L0 dmesg log as below. L0 kernel didn't print useful message when qemu process core dump in L1. # dmesg [ 1333.832860] switch: port 2(tap0) entered disabled state [ 1333.839437] device tap0 left promiscuous mode [ 1333.844315] switch: port 2(tap0) entered disabled state [ 1369.796789] switch: port 2(tap0) entered blocking state [ 1369.802637] switch: port 2(tap0) entered disabled state [ 1369.808503] device tap0 entered promiscuous mode [ 1369.813697] switch: port 2(tap0) entered blocking state [ 1369.819531] switch: port 2(tap0) entered forwarding state [10667.222600] switch: port 2(tap0) entered disabled state [10667.229289] device tap0 left promiscuous mode [10667.234166] switch: port 2(tap0) entered disabled state [10877.594898] switch: port 2(tap0) entered blocking state [10877.600735] switch: port 2(tap0) entered disabled state [10877.606625] device tap0 entered promiscuous mode [10877.611824] switch: port 2(tap0) entered blocking state [10877.617672] switch: port 2(tap0) entered forwarding state I tested another 2 scenarios in RHEL8.0 host. Action: reboot L2 guest inside guest. s1) L0 and L1 and L2 are RHEL8->works S2) L0 and L1 are RHEL8,But L2 is RHEL7.6-> L1 qemu process Aborted (core dumped) and (L0 and L1)dmesg log doesn't print any useful message. (process:7776): Spice-CRITICAL **: 05:03:52.062: display-channel.c:2035:display_channel_update: condition `display_channel_validate_surface(display, surface_id)' failed Thread 23 (Thread 0x7fdf07fff700 (LWP 7851)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fdf208f2700 (LWP 7850)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fdf211f4700 (LWP 7849)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fdf219f5700 (LWP 7848)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fdf221f6700 (LWP 7847)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fdf229f7700 (LWP 7846)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fdf231f8700 (LWP 7845)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fdf239f9700 (LWP 7844)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fdf388e7700 (LWP 7843)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fdf390e8700 (LWP 7842)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fdf398e9700 (LWP 7841)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fdf3a0ea700 (LWP 7840)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fdf3a9ec700 (LWP 7839)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdf437ff700 (LWP 7833)): #0 0x00007fe081f1658c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb635ce18d in qemu_cond_wait_impl () #2 0x000055fb634eab9b in vnc_worker_thread_loop () #3 0x000055fb634eb490 in vnc_worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fe068bff700 (LWP 7818)): #0 0x00007fe081f19ad4 in read () from /lib64/libpthread.so.0 #1 0x00007fe0830dcfb9 in ?? () from /lib64/libspice-server.so.1 #2 0x00007fe0830e4580 in ?? () from /lib64/libspice-server.so.1 #3 0x00007fe083098bf8 in ?? () from /lib64/libspice-server.so.1 #4 0x00007fe0830c5e7e in ?? () from /lib64/libspice-server.so.1 #5 0x00007fe083092aa8 in ?? () from /lib64/libspice-server.so.1 #6 0x00007fe0830994ef in ?? () from /lib64/libspice-server.so.1 #7 0x00007fe0879058ad in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #8 0x00007fe087905c78 in ?? () from /lib64/libglib-2.0.so.0 #9 0x00007fe087905fa2 in g_main_loop_run () from /lib64/libglib-2.0.so.0 #10 0x00007fe0830c623e in ?? () from /lib64/libspice-server.so.1 #11 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fe069926700 (LWP 7794)): #0 0x00007fe081c3ae47 in ioctl () from /lib64/libc.so.6 #1 0x000055fb632d8d09 in kvm_vcpu_ioctl () #2 0x000055fb632d8dc2 in kvm_cpu_exec () #3 0x000055fb632b610e in qemu_kvm_cpu_thread_fn () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fe06a127700 (LWP 7793)): #0 0x00007fe081c3ae47 in ioctl () from /lib64/libc.so.6 #1 0x000055fb632d8d09 in kvm_vcpu_ioctl () #2 0x000055fb632d8dc2 in kvm_cpu_exec () #3 0x000055fb632b610e in qemu_kvm_cpu_thread_fn () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fe06a928700 (LWP 7792)): #0 0x00007fe081c3ae47 in ioctl () from /lib64/libc.so.6 #1 0x000055fb632d8d09 in kvm_vcpu_ioctl () #2 0x000055fb632d8dc2 in kvm_cpu_exec () #3 0x000055fb632b610e in qemu_kvm_cpu_thread_fn () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fe06b129700 (LWP 7790)): #0 0x00007fe081c3ae47 in ioctl () from /lib64/libc.so.6 #1 0x000055fb632d8d09 in kvm_vcpu_ioctl () #2 0x000055fb632d8dc2 in kvm_cpu_exec () #3 0x000055fb632b610e in qemu_kvm_cpu_thread_fn () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fe06ba1b700 (LWP 7789)): #0 0x00007fe081c396d6 in ppoll () from /lib64/libc.so.6 #1 0x000055fb635ca459 in qemu_poll_ns () #2 0x000055fb635cc5cc in aio_poll () #3 0x000055fb6339dc4e in iothread_run () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fe06c31e700 (LWP 7788)): #0 0x00007fe081f19032 in do_futex_wait () from /lib64/libpthread.so.0 #1 0x00007fe081f19143 in __new_sem_wait_slow () from /lib64/libpthread.so.0 #2 0x000055fb635ce39f in qemu_sem_timedwait () #3 0x000055fb635c9a84 in worker_thread () #4 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fe06da02700 (LWP 7777)): #0 0x00007fe081c3ea69 in syscall () from /lib64/libc.so.6 #1 0x000055fb635ce5ff in qemu_event_wait () #2 0x000055fb635df342 in call_rcu_thread () #3 0x00007fe081f105f4 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe081c4405f in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fe088206200 (LWP 7776)): #0 0x00007fe081c396d6 in ppoll () from /lib64/libc.so.6 #1 0x000055fb635ca415 in qemu_poll_ns () #2 0x000055fb635cb348 in main_loop_wait () #3 0x000055fb63273c75 in main () Aborted (core dumped) Thank you for trying out the other scenario. Can you please give me access to the system or maybe, just copy the guest image to a place from where I can download it ? I am running into install issues when using the nightly iso or even the qcow2 image. I finally got my system setup... I am using a slightly different Skylake host (E3-1270 v5) and the same cmd line as yours. I am scripted the reboot and let it run for 10 minutes and still didn't hit the crash. I think it would be much for easier if I can jump on to your system. Please let me know. Thank you, I was able to reproduce it on your system. I believe this should be fixed by a more recent qemu and is probably related to the vga-qxl on your command line. Can you please try qemu-kvm-rhev-2.12.0-7.el7 both on L0 and L1 and check if you can still reproduce the problem ? (In reply to Bandan Das from comment #13) > Thank you, I was able to reproduce it on your system. I believe this should > be fixed by a more recent qemu and is probably related to the vga-qxl on > your command line. Can you please try qemu-kvm-rhev-2.12.0-7.el7 both on L0 > and L1 and check if you can still reproduce the problem ? I tested the latest qemu-kvm-rhev-2.12.0-7.el7. can not reproduce the problem. This problem is gone. Thank you for confirming, I am marking this a duplicate of bug 1567733. *** This bug has been marked as a duplicate of bug 1567733 *** |