Bug 1088864

Summary: nwfilter deadlock
Product: Red Hat Enterprise Linux 7 Reporter: Stefan Berger <stefanb>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: dyuan, honzhang, jdenemar, jiahu, jkurik, mzhan, rbalakri, tdosek
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: libvirt-1.2.7-1.el7 Doc Type: Bug Fix
Doc Text:
Cause: Trying to start a domain which does not exist anymore leaves network filters locked for read-only access. Consequence: Any attempt to gain read-write access to network filters (such as trying to edit a filter) causes a deadlock. Fix: Properly unlock network filters. Result: Starting a non-existent domain does not cause a deadlock on network filters any more.
Story Points: ---
Clone Of:
: 1112690 (view as bug list) Environment:
Last Closed: 2015-03-05 07:34:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1112690    

Description Stefan Berger 2014-04-17 10:38:51 UTC
Description of problem:

A deadlock may occur in nwfilter due to a QEMU driver function not releasing the nwfilter updated lock upon an error condition. The patch solving this has been pushed to the upstream repo:

commit a4209f53795290ff9f2173092800eb3f767fff3e
Author: Stefan Berger <stefanb.ibm.com>
Date:   Thu Apr 17 06:30:08 2014 -0400

    qemu: Unlock the NWFilter update lock by leaving via the cleanup label
    
    Fix a locking problem by leaving the function via the cleanup label.
    
    Signed-off-by: Stefan Berger <stefanb.ibm.com>

Comment 3 Hu Jianwei 2014-05-06 10:09:32 UTC
I can reproduce it.

Version:
libvirt-1.1.1-29.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7ev.x86_64
kernel-3.10.0-121.el7.x86_64

Using python to access that function, trigger the bug.
[root@localhost ~]# virsh create test.xml 
Domain test created from test.xml

[root@localhost ~]# python
Python 2.7.5 (default, Feb 11 2014, 07:46:25) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open("")
>>> dom = conn.lookupByName("test")
>>> dom.destroy()
0
>>> dom.createWithFlags(0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '9c455799-4684-4588-9dc4-312e87cfaeec' (test)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 728, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '9c455799-4684-4588-9dc4-312e87cfaeec' (test)
>>> 

Then, in another termianl, try to edit one nwfilter:
[root@localhost src]# virsh nwfilter-edit clean-traffic





^C      <==== no output, hung here
[root@localhost src]# virsh nwfilter-list


^C     <==== no output, hung here

Comment 6 Hu Jianwei 2014-11-24 11:07:25 UTC
Verified the bug as below:

[root@ibm-x3850x5-06 ~]# rpm -q libvirt
libvirt-1.2.8-7.el7.x86_64

[root@ibm-x3850x5-06 ~]# virsh create test.xml 
Domain test created from test.xml

[root@ibm-x3850x5-06 ~]# python
Python 2.7.5 (default, Feb 11 2014, 07:46:25) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open("")
>>> dom = conn.lookupByName("test")
>>> dom.destroy()
0
>>> dom.createWithFlags(0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '633638d6-7077-42f8-afd9-d2cbcfba8897' (test)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 993, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '633638d6-7077-42f8-afd9-d2cbcfba8897' (test)
>>> 
[root@ibm-x3850x5-06 ~]# 

Then, in another termianl, try to edit one nwfilter:
[root@ibm-x3850x5-06 ~]# virsh nwfilter-edit clean-traffic
Network filter clean-traffic XML configuration edited.

[root@ibm-x3850x5-06 ~]# virsh nwfilter-list 
 UUID                                  Name                 
------------------------------------------------------------------
 a1d1710b-32d8-4ca7-9a9a-ce16a3789cac  allow-arp           
 c6a96bf9-077d-49a4-835e-10564d010c80  allow-dhcp          
 379c24c3-bac9-49ba-994c-f51f930f9a33  allow-dhcp-server   
 34d3e568-f4d7-4b3e-926b-f82db56801ad  allow-incoming-ipv4 
 8da203c9-8c48-48cd-9601-0a523eee8dea  allow-ipv4          
 f3d9b618-9097-4b37-86a7-e804066e7fbe  clean-traffic       
 f280dd7d-32f6-4c5c-a64f-be28ebbb8a53  my                  
 71ef822b-4220-4752-b1aa-3653d40254a1  mynwfilter          
 dfdbb624-dc47-418b-87b3-c002723ae380  no-arp-ip-spoofing  
 4bb11c16-e23e-4935-b9ab-85820a45dd8f  no-arp-mac-spoofing 
 17bdfea7-1afc-4280-a776-ff343dee1f3c  no-arp-spoofing     
 8b7ef3ca-3689-49e3-b6cf-3df9dcb5f21b  no-ip-multicast     
 1fdef6f6-2d97-424c-b96c-b91a6913cb8d  no-ip-spoofing      
 d19a6e02-bf09-4d2e-b749-e5ad60d51dd2  no-mac-broadcast    
 3977fc1c-518f-42e2-9f20-c1a2a1d9de68  no-mac-spoofing     
 16e7180b-cb39-490b-ad9c-08f861b711d9  no-other-l2-traffic 
 41920bb3-82ce-4836-9354-5e5a965c31ff  no-other-rarp-traffic
 7fa7206f-8a2b-4dc9-8738-a114ba683a8f  qemu-announce-self  
 d624d4e9-13f7-4cc4-a7d4-ad402609ee33  qemu-announce-self-rarp

No deadlock occurs, changed to Verified.

Comment 7 Stefan Berger 2015-01-08 16:43:30 UTC
Verified also for libvirt-1.2.8-10.el7.x86_64.

Comment 9 errata-xmlrpc 2015-03-05 07:34:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html