Created attachment 731532 [details] ## Logs vdsm, rhevm, libvirt Description of problem: Get ERROR: libvirt error, internal error client socket is closed, via activate host action Version-Release number of selected component (if applicable): RHEVM 3.2 - SF11 environment: RHEVM: rhevm-3.2.0-10.14.beta1.el6ev.noarch VDSM: vdsm-4.10.2-11.0.el6ev.x86_64 LIBVIRT: libvirt-0.10.2-18.el6.x86_64 QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64 SANLOCK: sanlock-2.6-2.el6.x86_64 Run scenarios: 1. Activate host (see actions below) How reproducible: unknown Actual results: Failed Activate host Expected results: Succeed Activate host Workaround: 1. Restart VDSM 2. Enter host to Maintenance 3. Activate host Additional info: /var/log/ovirt-engine/engine.log 2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Failed in GetCapabilitiesVDS method 2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Error code unexpected and error message VDSGenericExcept ion: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception 2013-04-04 12:06:10,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCap abilitiesVDSCommand return value org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@df421c9 2013-04-04 12:06:10,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) HostName = tigris04.scl.lab.tlv.redhat.com 2013-04-04 12:06:10,974 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) Command GetCapabilitiesVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception 2013-04-04 12:06:10,974 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) VdsManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 4f67b59d-6fd7-41b3-ae59-4a1b9537dc0f : tigris04.scl.lab.tlv.redhat.com, error = VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception, continuing. 2013-04-04 12:06:10,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) ResourceManager::refreshVdsRunTimeInfo: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:65) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:200) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.ProceedProxyReturnValue(BrokerCommandBase.java:186) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.ExecuteVdsBrokerCommand(GetCapabilitiesVDSCommand.java:21) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.ExecuteVDSCommand(VdsBrokerCommand.java:92) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.ExecuteCommand(VDSCommandBase.java:63) [engine-vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.Execute(VdcCommandBase.java:28) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:522) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:530) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:381) [engine-vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:225) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] 2013-04-04 12:06:14,014 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Failed in GetCapabilitiesVDS method 2013-04-04 12:06:14,014 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to GetCapabilitiesVDS, error = Unexpected exception /var/log/vdsm/vdsm.log Thread-215556::DEBUG::2013-04-04 10:06:09,163::BindingXMLRPC::913::vds::(wrapper) client [10.35.102.12]::call getCapabilities with () {} flowID [77b8fab0] Thread-215556::ERROR::2013-04-04 10:06:09,173::BindingXMLRPC::923::vds::(wrapper) libvirt error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 287, in getCapabilities ret = api.getCapabilities() File "/usr/share/vdsm/API.py", line 1116, in getCapabilities c = caps.get() File "/usr/share/vdsm/caps.py", line 278, in get caps.update(netinfo.get()) File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 292, in get nets = networks() File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 93, in networks for name in conn.listNetworks(): File "/usr/lib64/python2.6/site-packages/libvirt.py", line 3433, in listNetworks if ret is None: raise libvirtError ('virConnectListNetworks() failed', conn=self) libvirtError: internal error client socket is closed Steps to Reproduce: 1. Create DC with 3 hosts connected to one SD, and in beginning they in following state Host-A – Maintenance Host-B – Maintenance Host-C – Active (SPM) 2. Second step Host-A – Maintenance Host-B – Active (HSM) Host-C – Active (SPM) 3. Step 3 Host-A – Maintenance Host-B – Active (SPM) Host-C – Maintenance 4. Step 4 Host-A – Active (HSM) Host-B – Active (SPM) Host-C – Maintenance 5. Step 5 Host-A – Active (SPM) Host-B – Maintenance Host-C – Maintenance 6. Step 6, activate Host-B & Host-C Host-A – Active (SPM) Host-B – Active (HSM) Host-C – Unassigned - Failed activate
May be a duplicate of Bug 949192
This should be libvirt bug and might be duplicate with Bug 924756 (libvirtError: internal error client socket is closed - shouldn't have happened.. libvirt stopped to communicate with vdsm) The bug in VDSM is that vdsm does not response rightly when the connection brakes, and this treated in Bug 949192 as barak mentioned. *** This bug has been marked as a duplicate of bug 949192 ***