Bug 948216

Created attachment 731532 [details]
## Logs vdsm, rhevm, libvirt

Description of problem:
Get ERROR: libvirt error, internal error client socket is closed, via activate host action

Version-Release number of selected component (if applicable):
RHEVM 3.2 - SF11 environment: 

RHEVM: rhevm-3.2.0-10.14.beta1.el6ev.noarch	 
VDSM: vdsm-4.10.2-11.0.el6ev.x86_64 
LIBVIRT: libvirt-0.10.2-18.el6.x86_64 
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64 
SANLOCK: sanlock-2.6-2.el6.x86_64

Run scenarios:
1. Activate host (see actions below)

How reproducible:
unknown

Actual results:
Failed Activate host

Expected results:
Succeed Activate host

Workaround:
1. Restart VDSM
2. Enter host to Maintenance
3. Activate host

Additional info:

/var/log/ovirt-engine/engine.log

2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Failed in GetCapabilitiesVDS method
2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Error code unexpected and error message VDSGenericExcept
ion: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception
2013-04-04 12:06:10,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCap
abilitiesVDSCommand return value 
 org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@df421c9
2013-04-04 12:06:10,974 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) HostName = tigris04.scl.lab.tlv.redhat.com
2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) Command GetCapabilitiesVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception
2013-04-04 12:06:10,974 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) VdsManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 4f67b59d-6fd7-41b3-ae59-4a1b9537dc0f : tigris04.scl.lab.tlv.redhat.com, error = VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception, continuing.
2013-04-04 12:06:10,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) ResourceManager::refreshVdsRunTimeInfo: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:65) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:200) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.ProceedProxyReturnValue(BrokerCommandBase.java:186) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.ExecuteVdsBrokerCommand(GetCapabilitiesVDSCommand.java:21) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.ExecuteVDSCommand(VdsBrokerCommand.java:92) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.ExecuteCommand(VDSCommandBase.java:63) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.Execute(VdcCommandBase.java:28) [engine-dal.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:522) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:530) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:381) [engine-vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:225) [engine-vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:]

2013-04-04 12:06:14,014 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Failed in GetCapabilitiesVDS method
2013-04-04 12:06:14,014 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception

/var/log/vdsm/vdsm.log

Thread-215556::DEBUG::2013-04-04 10:06:09,163::BindingXMLRPC::913::vds::(wrapper) client [10.35.102.12]::call getCapabilities with () {} flowID [77b8fab0]
Thread-215556::ERROR::2013-04-04 10:06:09,173::BindingXMLRPC::923::vds::(wrapper) libvirt error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 287, in getCapabilities
    ret = api.getCapabilities()
  File "/usr/share/vdsm/API.py", line 1116, in getCapabilities
    c = caps.get()
  File "/usr/share/vdsm/caps.py", line 278, in get
    caps.update(netinfo.get())
  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 292, in get
    nets = networks()
  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 93, in networks
    for name in conn.listNetworks():
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3433, in listNetworks
    if ret is None: raise libvirtError ('virConnectListNetworks() failed', conn=self)
libvirtError: internal error client socket is closed

Steps to Reproduce:
1. Create DC with 3 hosts connected to one SD, and in beginning they in following state
Host-A – Maintenance 
Host-B – Maintenance 
Host-C – Active (SPM)
2. Second step
Host-A – Maintenance 
Host-B – Active (HSM) 
Host-C – Active (SPM)
3. Step 3
Host-A – Maintenance 
Host-B – Active (SPM) 
Host-C – Maintenance
4. Step 4
Host-A – Active (HSM) 
Host-B – Active (SPM) 
Host-C – Maintenance
5. Step 5
Host-A – Active (SPM) 
Host-B – Maintenance
Host-C – Maintenance
6. Step 6, activate Host-B & Host-C
Host-A – Active (SPM) 
Host-B – Active (HSM) 
Host-C – Unassigned - Failed activate