Bug 1350688
| Summary: | libvirtd crashes after qemu-attach in qemuDomainPerfRestart() | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jingjing Shao <jishao> |
| Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.2 | CC: | dyuan, dzheng, mzhan, pkrempa, rbalakri, yafu, yalzhang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-2.0.0-1.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-03 18:47:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Perf event state is not initialized in qemuAttach Fixed upstream:
commit d7c40d50d721f5e34522efc57c4c4537721602c2
Author: Peter Krempa <pkrempa>
Date: Tue Jun 28 14:37:29 2016 +0200
conf: def: Avoid unnecessary allocation of 'perf' events definition
Some code paths already assume that it is allocated since it was always
allocated by virDomainPerfDefParseXML. Make it member of virDomainDef
directly so that we don't have to allocate it all the time.
This fixes crash when attempting to connect to an existing process via
virDomainQemuAttach since we would not allocate it in that code path.
test on libvirt-2.0.0-1.el7.x86_64, the result is as expected, move the bug to verified. # /usr/libexec/qemu-kvm -m 1024 -drive file=/var/lib/libvirt/images/rhel7.2.qcow2,index=0 -monitor unix:/tmp/ff,server,nowait -name r7 --uuid 1fdf7c78-866a-4dcf-b017-5a9299682e1f & [3] 11634 # warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5] warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6] VNC server running on '::1;5900' # ps -ef | grep r7 | grep -v grep root 11634 6721 99 14:59 pts/1 00:00:17 /usr/libexec/qemu-kvm -m 1024 -drive file=/var/lib/libvirt/images/rhel7.2.qcow2,index=0 -monitor unix:/tmp/ff,server,nowait -name r7 --uuid 1fdf7c78-866a-4dcf-b017-5a9299682e1f # virsh qemu-attach 11634 Domain r7 attached to pid 11634 # virsh list --all Id Name State ---------------------------------------------------- 3 r7 running # ps -ef | grep r7 | grep -v grep root 11634 6721 83 14:59 pts/1 00:00:49 /usr/libexec/qemu-kvm -m 1024 -drive file=/var/lib/libvirt/images/rhel7.2.qcow2,index=0 -monitor unix:/tmp/ff,server,nowait -name r7 --uuid 1fdf7c78-866a-4dcf-b017-5a9299682e1f Hi peter, Even this 'qemu-attach' command is discouraged to use (stated in the man page), there is one change make me confused. Would you please help to confirm? Test on libvirt-2.0.0-10.el7.x86_64 Steps to Reproduce: 1. # qemu-img create /var/lib/libvirt/images/foo.img 1G 2. # /usr/libexec/qemu-kvm -cdrom /var/lib/libvirt/images/foo.img -monitor unix:/tmp/demo,server,nowait -name foo -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ea & [1] 7044 # VNC server running on '::1;5900' 3.qemu-attach failed # virsh qemu-attach 7044 error: Failed to attach to pid 7044 error: Operation not supported: JSON monitor is required # virsh list --all Id Name State ---------------------------------------------------- ===> not connected 4. Run the qemu-attach command the 2nd time, it will hang. On another terminal, you will found the vm is already connected with "shut off" state # virsh qemu-attach 7044 ===> it will hang # virsh list Id Name State ---------------------------------------------------- 4 foo shut off # ps -aux | grep qemu-kvm root 7044 18.4 0.5 591372 39536 pts/1 Sl 12:34 0:12 /usr/libexec/qemu-kvm -cdrom /var/lib/libvirt/images/foo.img -monitor unix:/tmp/demo,server,nowait -name foo -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ea 5. Run the qemu-attach the 3rd time, it says the domain is already active # virsh qemu-attach 7044 error: Failed to attach to pid 7044 error: Requested operation is not valid: domain 'foo' is already active Actual results: qemu-attach will fail with 'Json monitor' required. Expected results: The qemu-attach command should work well. Additional info: Downgrade to libvirt-2.0.0-6.el7.x86_64 on the same host, it works well. # virsh qemu-attach 6636 Domain foo attached to pid 6636 I have no idea about 'JSON monitor'. If I should install some packages like "Json monitor"? Thank you in advance. Please file a new bug with the findings. I'm not sure what caused us to require the JSON monitor in this case, but the VM should not stay there in case of the second attach attempt. Hi peter, Thank you, file a bug 1378401 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html |
Description of problem: The libvirtd service will be crash after qemu-attach domain PID Version-Release number of selected component (if applicable): libvirt-1.3.5-1.el7.x86_64 qemu-kvm-rhev-2.6.0-9.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. make sure the libvirtd service is running [root@hp-dl385g8-02 images]# service libvirtd status Redirecting to /bin/systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2016-06-28 02:00:29 EDT; 16s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3335 (libvirtd) 2.# ll /var/lib/libvirt/images/test.img -rw-r--r--. 1 root root 1647771648 Jun 28 01:51 /var/lib/libvirt/images/test.img 3. [root@hp-dl385g8-02 tmp]# /usr/libexec/qemu-kvm -m 1500 -drive file=/var/lib/libvirt/images/test.img,index=0 -monitor unix:/tmp/ss,server,nowait -name test -uuid 1fdf7c78-866a-4dcf-b017-5a9299682e1f & [1] 2960 [root@hp-dl385g8-02 tmp]# VNC server running on '::1;5900' 4.[root@hp-dl385g8-02 images]# ps -ef | grep qemu root 2960 11242 99 01:44 pts/0 00:00:12 /usr/libexec/qemu-kvm -m 1500 -drive file=/var/lib/libvirt/images/test.img,index=0 -monitor unix:/tmp/ss,server,nowait -name test -uuid 1fdf7c78-866a-4dcf-b017-5a9299682e1f 5.[root@hp-dl385g8-02 images]# virsh qemu-attach 2960 error: Disconnected from qemu:///system due to I/O error error: Failed to attach to pid 2960 error: End of file while reading data: Input/output error 6.[root@hp-dl385g8-02 images]# service libvirtd status Redirecting to /bin/systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2016-06-28 02:01:03 EDT; 17s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3433 (libvirtd) Actual results: As subject Expected results: The domain can be attached successfully, and the libvirtd service should not crash Additional info: Thread 17 (Thread 0x7f56e06d6700 (LWP 3684)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f56dfed5700 (LWP 3685)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f56df6d4700 (LWP 3686)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f56deed3700 (LWP 3687)): #0 0x00007f56d40c9042 in qemuDomainPerfRestart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #1 0x00007f56d40ce132 in qemuProcessAttach () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #2 0x00007f56d41119f4 in qemuDomainQemuAttach () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007f56f00501ac in virDomainQemuAttach () from /lib64/libvirt-qemu.so.0 #4 0x00007f56f08db7b9 in qemuDispatchDomainAttachHelper () #5 0x00007f56efccdef2 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #6 0x00007f56f08eae0d in virNetServerHandleJob () #7 0x00007f56efbbc581 in virThreadPoolWorker () from /lib64/libvirt.so.0 #8 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #9 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f56de6d2700 (LWP 3688)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f56dded1700 (LWP 3689)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 ---Type <return> to continue, or q <return> to quit--- #2 0x00007f56efbbc5cb in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f56dd6d0700 (LWP 3690)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc5cb in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f56dcecf700 (LWP 3691)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc5cb in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f56d7fff700 (LWP 3692)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc5cb in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f56d77fe700 (LWP 3693)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc5cb in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f56cd58e700 (LWP 3694)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f56ccd8d700 (LWP 3695)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 ---Type <return> to continue, or q <return> to quit--- #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f56ada7e700 (LWP 3696)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f56ad27d700 (LWP 3697)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f56aca7c700 (LWP 3698)): #0 0x00007f56ed1e26d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f56efbbbb76 in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f56efbbc633 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f56efbbb908 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f56ed1dedc5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f56ecf051cd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f56f082a8c0 (LWP 3683)): #0 0x00007f56ecefab7d in poll () from /lib64/libc.so.6 #1 0x00007f56efb75bb6 in virEventPollRunOnce () from /lib64/libvirt.so.0 #2 0x00007f56efb74692 in virEventRunDefaultImpl () from /lib64/libvirt.so.0 #3 0x00007f56efcc7fdd in virNetDaemonRun () from /lib64/libvirt.so.0 #4 0x00007f56f08b1d02 in main ()