Bug 1295164

Description of problem:

libvirtd is always in R states, and its CPU usage always exceed 100%.
It did not respond to gdb and pstack. So I could not provide the call stacks.

I could not kill libvirtd with -9 and could stop it with service stop. 
Related virsh cli could not get response.
We met this issue in the version 1.2.14,1.2.15 and 0.10.2. 
 now we revert to version 0.10.2.
In order to keep service normal, we restart another libvitd with port 16510.
Now new libvirtd is normal. 

[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19705  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19709  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19711  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19713  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19715  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19717  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19719  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19723  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19725  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19727  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19733  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19737  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# kill -9 11607
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:23 libvirtd --daemon --listen
root     19743  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# pstack 11607
[root@computer2 ~]# 
[root@computer2 ~]# pstack 11607



[root@computer2 ~]# libvirtd --version
libvirtd (libvirt) 0.10.2 



Version-Release number of selected component (if applicable):
[root@computer2 ~]# uname -a
Linux computer2 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@computer2 ~]# libvirtd --version
libvirtd (libvirt) 0.10.2 

How reproducible:

did not know
Steps to Reproduce:
1.
2.
3.

Actual results:
libvirtd hung

Expected results:
libvirtd work right.
Additional info:

Created attachment 1112817 [details]
deadlock detection using valgrind --tool=helgrind

it could be reproduciable in our test environment.

If it's not even responding to gdb or pstack, that suggests to me that there is something wrong with your host kernel or hardware... nothing libvirt should be able to do should prevent those from working AFAIK.

If you are using RHEL, I suggest filing a bug there for further triage

As this issue frustrated us a lot of days, we update the linux kernel and openstack VM migration algorithms.
The original VM migration will consume lots of time when VM mounting a big cloud disk.
When adopting new VM migration algorithms, VM migration only need very short time.   
So the issue disappear.