Bug 1726758
| Summary: | [Cinderlib] - Error connecting to ceph cluster when starting VM with Ceph MBD disk | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Shir Fishbain <sfishbai> | ||||
| Component: | BLL.Storage | Assignee: | Benny Zlotnik <bzlotnik> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Shir Fishbain <sfishbai> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.3.5.2 | CC: | bugs, bzlotnik, frolland, tnisan | ||||
| Target Milestone: | ovirt-4.3.6 | Flags: | pm-rhel:
ovirt-4.3+
|
||||
| Target Release: | 4.3.6.3 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | ovirt-engine-4.3.6.3 | Doc Type: | No Doc Update | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-09-26 19:43:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1587112 [details]
logs_ceph
There is a configuration mismatch between the host and the ceph server: 'RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".\\\\nIn some cases useful info is found in syslog - try "dmesg | tail" Verified with DEV ENV ovirt-engine-4.3.6.2-0.1.el7.noarch vdsm-4.30.24-2.el7ev.x86_64 This bugzilla is included in oVirt 4.3.6 release, published on September 26th 2019. Since the problem described in this bug report should be resolved in oVirt 4.3.6 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: Start VM with Ceph managed storage domain disk fails with the error: "VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster" Name: rados Version: 0 Summary: Python libraries for the Ceph librados library Home-page: UNKNOWN Author: UNKNOWN Author-email: UNKNOWN License: UNKNOWN Location: /usr/lib64/python2.7/site-packages Requires: Required-by: Version-Release number of selected component (if applicable): ovirt-engine-4.3.5.2-0.1.el7.noarch vdsm-4.30.22-1.el7ev.x86_64 Cinderlib version : 0.9.0 How reproducible: 100% Steps to Reproduce: 1.Create a managed block storage domain from Ceph driver via REST-API: 2.Create VM 3.Create a disk from the SD in step 1 and activate and add it to VM 4.Start VM Actual results: From engine log: 2019-07-03 19:09:38,636+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedThreadFactory-engine-Thread-7570) [5eabaa50] cinderlib execution failed: Traceback (most recent call last): File "./cinderlib-client.py", line 179, in main args.command(args) File "./cinderlib-client.py", line 232, in connect_volume backend = load_backend(args) File "./cinderlib-client.py", line 210, in load_backend return cl.Backend(**json.loads(args.driver)) File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 88, in __init__ self.driver.check_for_setup_error() File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 288, in check_for_setup_error with RADOSClient(self): File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 170, in __init__ self.cluster, self.ioctx = driver._connect_to_rados(pool) File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 346, in _connect_to_rados return _do_conn(pool, remote, timeout) File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper return r.call(f, *args, **kwargs) File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call raise attempt.get() File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 344, in _do_conn raise exception.VolumeBackendAPIException(data=msg) VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. 2019-07-03 19:10:13,699+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] cinderlib execution failed: Traceback (most recent call last): File "./cinderlib-client.py", line 179, in main args.command(args) File "./cinderlib-client.py", line 232, in connect_volume backend = load_backend(args) File "./cinderlib-client.py", line 210, in load_backend return cl.Backend(**json.loads(args.driver)) File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 88, in __init__ self.driver.check_for_setup_error() File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 288, in check_for_setup_error with RADOSClient(self): File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 170, in __init__ self.cluster, self.ioctx = driver._connect_to_rados(pool) File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 346, in _connect_to_rados return _do_conn(pool, remote, timeout) File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper return r.call(f, *args, **kwargs) File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call raise attempt.get() File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 344, in _do_conn raise exception.VolumeBackendAPIException(data=msg) VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster. 2019-07-03 19:10:13,816+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM my_new_vm_ceph on Host host_mixed_1. 2019-07-03 19:10:13,822+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM my_new_vm_ceph (User: admin@internal-authz). 2019-07-03 19:10:13,915+03 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [2fdee278] Running command: ProcessDownVmCommand internal: true. 2019-07-03 19:10:14,047+03 INFO [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Running command: DisconnectManagedBlockStorageDeviceCommand internal: true. 2019-07-03 19:10:14,050+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Command 'org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand' failed: null 2019-07-03 19:10:14,050+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Exception: java.lang.NullPointerException at org.ovirt.engine.core.common.vdscommands.VdsIdAndVdsVDSCommandParametersBase.<init>(VdsIdAndVdsVDSCommandParametersBase.java:18) [common.jar:] at org.ovirt.engine.core.common.vdscommands.AttachManagedBlockStorageVolumeVDSCommandParameters.<init>(AttachManagedBlockStorageVolumeVDSCommandParameters.java:17) [common.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand.detachVolume(DisconnectManagedBlockStorageDeviceCommand.java:76) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand.executeCommand(DisconnectManagedBlockStorageDeviceCommand.java:47) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:378) [bll.jar:] at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) [:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) [:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:201) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.lambda$disconnectManagedBlockStorageDisks$2(ManagedBlockStorageCommandUtil.java:170) [bll.jar:] at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) [rt.jar:1.8.0_222-ea] at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) [rt.jar:1.8.0_222-ea] at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) [rt.jar:1.8.0_222-ea] at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) [rt.jar:1.8.0_222-ea] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) [rt.jar:1.8.0_222-ea] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) [rt.jar:1.8.0_222-ea] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) [rt.jar:1.8.0_222-ea] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) [rt.jar:1.8.0_222-ea] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_222-ea] at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) [rt.jar:1.8.0_222-ea] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:170) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:160) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_222-ea] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:237) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view8.disconnectManagedBlockStorageDisks(Unknown Source) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_222-ea] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.disconnectManagedBlockStorageDisks(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.ProcessDownVmCommand.executeCommand(ProcessDownVmCommand.java:144) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:639) [bll.jar:] at sun.reflect.GeneratedMethodAccessor188.invoke(Unknown Source) [:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source) [:1.8.0_222-ea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2354) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2379) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2374) [bll.jar:] at org.ovirt.engine.core.bll.RunVmCommandBase.lambda$processVmOnDown$0(RunVmCommandBase.java:210) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_222-ea] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_222-ea] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_222-ea] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_222-ea] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222-ea] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1] Expected results: Start VM with Ceph MBD disk should succeed Additional info: