Bug 1295164 - libvirtd always hung and its states is in R state
libvirtd always hung and its states is in R state
Status: CLOSED DEFERRED
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Libvirt Maintainers
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-03 02:17 EST by wjw7869
Modified: 2016-04-10 22:00 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-10 18:54:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
deadlock detection using valgrind --tool=helgrind (198.65 KB, text/plain)
2016-01-08 04:38 EST, wjw7869
no flags Details

  None (edit)
Description wjw7869 2016-01-03 02:17:08 EST
Description of problem:
libvirtd is always in R states, and its CPU usage always exceed 100%.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 wjw7869 2016-01-03 02:33:24 EST
Description of problem:

libvirtd is always in R states, and its CPU usage always exceed 100%.
It did not respond to gdb and pstack. So I could not provide the call stacks.

I could not kill libvirtd with -9 and could stop it with service stop. 
Related virsh cli could not get response.
We met this issue in the version 1.2.14,1.2.15 and 0.10.2. 
 now we revert to version 0.10.2.
In order to keep service normal, we restart another libvitd with port 16510.
Now new libvirtd is normal. 

[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19705  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19709  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13986:57 libvirtd --daemon --listen
root     19711  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19713  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19715  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19717  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19719  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19723  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19725  0.0  0.0 103256   852 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19727  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19733  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:06 libvirtd --daemon --listen
root     19737  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# kill -9 11607
[root@computer2 ~]# ps aux | grep libvirtd
root     11607 81.9  0.0 1012292 12348 ?       R     2015 13987:23 libvirtd --daemon --listen
root     19743  0.0  0.0 103256   856 pts/4    S+   15:19   0:00 grep libvirtd
root     31334  0.6  0.0 1543000 13596 ?       Sl    2015  77:23 libvirtd --daemon --listen
[root@computer2 ~]# pstack 11607
[root@computer2 ~]# 
[root@computer2 ~]# pstack 11607



[root@computer2 ~]# libvirtd --version
libvirtd (libvirt) 0.10.2 



Version-Release number of selected component (if applicable):
[root@computer2 ~]# uname -a
Linux computer2 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@computer2 ~]# libvirtd --version
libvirtd (libvirt) 0.10.2 

How reproducible:

did not know
Steps to Reproduce:
1.
2.
3.

Actual results:
libvirtd hung

Expected results:
libvirtd work right.
Additional info:
Comment 2 wjw7869 2016-01-08 04:38 EST
Created attachment 1112817 [details]
deadlock detection using valgrind --tool=helgrind

it could be reproduciable in our test environment.
Comment 3 Cole Robinson 2016-04-10 18:54:47 EDT
If it's not even responding to gdb or pstack, that suggests to me that there is something wrong with your host kernel or hardware... nothing libvirt should be able to do should prevent those from working AFAIK.

If you are using RHEL, I suggest filing a bug there for further triage
Comment 4 wjw7869 2016-04-10 22:00:55 EDT
As this issue frustrated us a lot of days, we update the linux kernel and openstack VM migration algorithms.
The original VM migration will consume lots of time when VM mounting a big cloud disk.
When adopting new VM migration algorithms, VM migration only need very short time.   
So the issue disappear.

Note You need to log in before you can comment on or make changes to this bug.