Bug 984267 - superVdsm fails to connect to libvirt socket after linvirtd service restart
superVdsm fails to connect to libvirt socket after linvirtd service restart
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.3.0
Unspecified Linux
unspecified Severity urgent
: ---
: 3.3.0
Assigned To: Yaniv Bronhaim
Tareq Alayan
infra
:
Depends On:
Blocks: 1019461
  Show dependency treegraph
 
Reported: 2013-07-14 05:35 EDT by Meni Yakove
Modified: 2016-02-10 14:05 EST (History)
11 users (show)

See Also:
Fixed In Version: is8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 11:28:22 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm and supervdsm logs (654.65 KB, application/zip)
2013-07-14 05:35 EDT, Meni Yakove
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 17066 None None None Never

  None (edit)
Description Meni Yakove 2013-07-14 05:35:16 EDT
Description of problem:
When restarting libvirt service superVdsm fails to connect to libvirt socket.
Operation like add/del network fails although libvirt is up and virsh is working.


Version-Release number of selected component (if applicable):
vdsm-4.11.0-121.git082925a.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.vdsm, supervdsm and libvirt is up.
2.restart libvirt service - initctl restart libvirtd
3.try to create network - vdsClient -s 0 addNetwork bridge=sw1 nics=eth1

Actual results:
Action success

Expected results:
Action failed

Additional info:
MainProcess|Thread-14::ERROR::2013-07-14 12:03:37,599::supervdsmServer::77::SuperVdsm.ServerCallback::(wrapper) Error in addNetwork
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 162, in addNetwork
    return configNetwork.addNetwork(bridge, **options)
  File "/usr/share/vdsm/configNetwork.py", line 156, in addNetwork
    _netinfo = netinfo.NetInfo()
  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 503, in __init__
    _netinfo = get()
  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 422, in get
    for net, netAttr in networks().iteritems():
  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 106, in networks
    allNets = ((net, net.name()) for net in conn.listAllNetworks(0))
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3743, in listAllNetworks
    raise libvirtError("virConnectListAllNetworks() failed", conn=self)
libvirtError: internal error client socket is closed
Comment 1 Meni Yakove 2013-07-14 05:35:49 EDT
Created attachment 773257 [details]
vdsm and supervdsm logs
Comment 2 Yaniv Bronhaim 2013-07-18 06:05:38 EDT
Catching RemoteError and reconnect solves the issue, but I still investigate why this is not occurred over fedora. IMHO, It has something with multiprocessing package.
Dan, any rejects to merge that fix?
Comment 3 Yaniv Bronhaim 2013-07-23 08:00:30 EDT
After restart of libvirt, calls with old connection will lead to sigterm, that will restart the caller's service. after establishing new connection all will work properly. 
Although, if supervdsm did the call and restarted (as happens while you call to setupNetworks), the next call of vdsm to supervdsm (whatever call it is) will fail and lead to reconnect. Just keep in mind that after such scenario we raise exception to see the error. Restart of libvirt or supervdsm is something that needs to be handled, we could restart and make the same call automatically (as we did in the past) but that hides the crash. We preferred to report the error first, then reconnect and then let the caller decide if to make the call again or handle the error specifically.
Comment 4 Meni Yakove 2013-08-11 10:04:25 EDT
vdsm-4.12.0-52.gitce029ba.el6ev.x86_64
Comment 5 Charlie 2013-11-27 19:35:09 EST
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.
Comment 6 errata-xmlrpc 2014-01-21 11:28:22 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html

Note You need to log in before you can comment on or make changes to this bug.