Bug 1134434
| Summary: | Snapshot disks in locked state after failed delete | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Retired] oVirt | Reporter: | Kevin Alon Goldblatt <kgoldbla> | ||||||
| Component: | ovirt-engine-core | Assignee: | Daniel Erez <derez> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Kevin Alon Goldblatt <kgoldbla> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.5 | CC: | acanan, amureini, ecohen, gklein, iheim, rbalakri, tnisan, yeylon | ||||||
| Target Milestone: | --- | Keywords: | Triaged | ||||||
| Target Release: | 3.5.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | storage | ||||||||
| Fixed In Version: | ovirt-3.5.0_rc2 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-10-06 06:04:43 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 944069 [details]
engine vdsm and server logs
Moving to Reopen. Ran the same scenation and the problems Still happens in V3.5 vt4. Added additional logs According the the logs [1], the issue now is a failure in marshaling response when using vdsm with json-rpc (bug 1148498). The locked disks issue is already addressed by bug 1134866 - should be resolved in a following build. [1] 2014-10-01 11:52:02,957 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-7-thread-41) [74f40c41] ERROR, CloneImageGroupStructureVDSCommand( storagePoolId = 90779371-001b-4e94-875d-1c1a502b8409, ignoreFailoverLimit = false, storageDomainId = 740dff84-7986-42a5-88fd-6c227b0c35e8, imageGroupId = f4ef4ba1-2188-4c1e-9868-9001517d2e35, dstDomainId = e80c2433-fe4e-4def-a062-c83edd238b6f), exception: ClassCastException: java.lang.Boolean cannot be cast to java.lang.String, log id: 32823cd9: java.lang.ClassCastException: java.lang.Boolean cannot be cast to java.lang.String at org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc.<init>(OneUuidReturnForXmlRpc.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcIIrsServer.cloneImageStructure(JsonRpcIIrsServer.java:452) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.CloneImageGroupStructureVDSCommand.executeIrsBrokerCommand(CloneImageGroupStructureVDSCommand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:156) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:418) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] *** This bug has been marked as a duplicate of bug 1134866 *** |
Created attachment 931487 [details] engine vdsm and server logs Description of problem: When deleting the snapshot disks and blocking access to the server from the host during this operation the operation fails BUT all host remain permanently in a Locked state after the network connection is restored and the host becomes operational again Version-Release number of selected component (if applicable): ovirt-engine-3.5.0-0.0.master.20140821064931.gitb794d66.el6.noarch vdsm-4.16.1-6.gita4a4614.el6.x86_64 How reproducible: All the time Steps to Reproduce: 1.create vm +4 disks (create the none bootable disks as thin) install os and run it 2.write total of 3 giga into all disks using dd 3.take a snapshot 4.multiple remove all volumes 5.add ip block from vdsm to engine during removal (Used iptables) 6. A failed fence is attempted by the engine and the host is non responsive 6. Remove the IP block about 5 minutes later >>>> the host is now operational again BUT The snapshot Disks that were not deleted are now in locked state and remain Actual results: The snapshot disks are in a locked state Expected results: The snapshot disks should return to the normal state Additional info: FROM ENGINE LOG>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>... STARTING THE COUNTDOWN TO FENCING THE HOST WHEN THE IPTABLE BLOCKS ACCESS 2014-08-24 19:20:16,451 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) [79d010d5] Host nott-vds1 is not responding. It will stay in Connecting state for a grace period of $160 seconds and after that an attempt to fence the host will be issued. 2014-08-24 19:20:16,457 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) [79d010d5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:567) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:628) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:485) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) [:1.7.0_55] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_55] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_55] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_55] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_55] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_55] at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_55] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) [jsse.jar:1.7.0_55] at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) [utils.jar:] at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) [commons-httpclient.jar:] at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3]