1726758 – [Cinderlib] - Error connecting to ceph cluster when starting VM with Ceph MBD disk

Bug 1726758 - [Cinderlib] - Error connecting to ceph cluster when starting VM with Ceph MBD disk

Summary: [Cinderlib] - Error connecting to ceph cluster when starting VM with Ceph MBD...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	ovirt-engine
Classification:	oVirt
Component:	BLL.Storage
Sub Component:
Version:	4.3.5.2
Hardware:	x86_64
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	ovirt-4.3.6
Target Release:	4.3.6.3
Assignee:	Benny Zlotnik
QA Contact:	Shir Fishbain
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-07-03 16:28 UTC by Shir Fishbain
Modified:	2019-09-26 19:43 UTC (History)
CC List:	4 users (show)
Fixed In Version:	ovirt-engine-4.3.6.3
Clone Of:
Environment:
Last Closed:	2019-09-26 19:43:30 UTC
oVirt Team:	Storage
Embargoed:
Dependent Products:
Flags:	pm-rhel: ovirt-4.3+

Attachments	(Terms of Use)
logs_ceph (1.53 MB, application/zip) 2019-07-03 16:29 UTC, Shir Fishbain	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
oVirt gerrit	101532	master	MERGED	core: remove attached_vds_id when disconnecting a disk	2020-08-05 12:24:27 UTC
oVirt gerrit	101533	master	MERGED	core: change access modifier	2020-08-05 12:24:27 UTC
oVirt gerrit	101534	master	MERGED	core: do not disconnect MBS if attach failed	2020-08-05 12:24:27 UTC
oVirt gerrit	102448	ovirt-engine-4.3	MERGED	core: do not disconnect MBS if attach failed	2020-08-05 12:24:26 UTC

Description Shir Fishbain 2019-07-03 16:28:17 UTC

Description of problem:
Start VM with Ceph managed storage domain disk fails with the error:
"VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster"

Name: rados
Version: 0
Summary: Python libraries for the Ceph librados library
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /usr/lib64/python2.7/site-packages
Requires: 
Required-by: 

Version-Release number of selected component (if applicable):
ovirt-engine-4.3.5.2-0.1.el7.noarch
vdsm-4.30.22-1.el7ev.x86_64
Cinderlib version : 0.9.0

How reproducible:
100%

Steps to Reproduce:
1.Create a managed block storage domain from Ceph driver via REST-API:
2.Create VM
3.Create a disk from the SD in step 1 and activate and add it to VM 
4.Start VM


Actual results:

From engine log:
2019-07-03 19:09:38,636+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedThreadFactory-engine-Thread-7570) [5eabaa50] cinderlib execution failed: Traceback (most recent call last):
  File "./cinderlib-client.py", line 179, in main
    args.command(args)
  File "./cinderlib-client.py", line 232, in connect_volume
    backend = load_backend(args)
  File "./cinderlib-client.py", line 210, in load_backend
    return cl.Backend(**json.loads(args.driver))
  File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 88, in __init__
    self.driver.check_for_setup_error()
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 288, in check_for_setup_error
    with RADOSClient(self):
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 170, in __init__
    self.cluster, self.ioctx = driver._connect_to_rados(pool)
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 346, in _connect_to_rados
    return _do_conn(pool, remote, timeout)
  File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper
    return r.call(f, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
    raise attempt.get()
  File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 344, in _do_conn
    raise exception.VolumeBackendAPIException(data=msg)
VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.

2019-07-03 19:10:13,699+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] cinderlib execution failed: Traceback (most recent call last):
  File "./cinderlib-client.py", line 179, in main
    args.command(args)
  File "./cinderlib-client.py", line 232, in connect_volume
    backend = load_backend(args)
  File "./cinderlib-client.py", line 210, in load_backend
    return cl.Backend(**json.loads(args.driver))
  File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 88, in __init__
    self.driver.check_for_setup_error()
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 288, in check_for_setup_error
    with RADOSClient(self):
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 170, in __init__
    self.cluster, self.ioctx = driver._connect_to_rados(pool)
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 346, in _connect_to_rados
    return _do_conn(pool, remote, timeout)
  File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 799, in _wrapper
    return r.call(f, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
    raise attempt.get()
  File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 344, in _do_conn
    raise exception.VolumeBackendAPIException(data=msg)
VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.

2019-07-03 19:10:13,816+03 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM my_new_vm_ceph on Host host_mixed_1.
2019-07-03 19:10:13,822+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7570) [7b7feeae] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM my_new_vm_ceph  (User: admin@internal-authz).
2019-07-03 19:10:13,915+03 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [2fdee278] Running command: ProcessDownVmCommand internal: true.
2019-07-03 19:10:14,047+03 INFO  [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Running command: DisconnectManagedBlockStorageDeviceCommand internal: true.
2019-07-03 19:10:14,050+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Command 'org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand' failed: null
2019-07-03 19:10:14,050+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand] (EE-ManagedThreadFactory-engine-Thread-7613) [25a5c13d] Exception: java.lang.NullPointerException
	at org.ovirt.engine.core.common.vdscommands.VdsIdAndVdsVDSCommandParametersBase.<init>(VdsIdAndVdsVDSCommandParametersBase.java:18) [common.jar:]
	at org.ovirt.engine.core.common.vdscommands.AttachManagedBlockStorageVolumeVDSCommandParameters.<init>(AttachManagedBlockStorageVolumeVDSCommandParameters.java:17) [common.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand.detachVolume(DisconnectManagedBlockStorageDeviceCommand.java:76) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.DisconnectManagedBlockStorageDeviceCommand.executeCommand(DisconnectManagedBlockStorageDeviceCommand.java:47) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:]
	at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:378) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) [:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) [:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:201) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.lambda$disconnectManagedBlockStorageDisks$2(ManagedBlockStorageCommandUtil.java:170) [bll.jar:]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) [rt.jar:1.8.0_222-ea]
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) [rt.jar:1.8.0_222-ea]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) [rt.jar:1.8.0_222-ea]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) [rt.jar:1.8.0_222-ea]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) [rt.jar:1.8.0_222-ea]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) [rt.jar:1.8.0_222-ea]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) [rt.jar:1.8.0_222-ea]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) [rt.jar:1.8.0_222-ea]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_222-ea]
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) [rt.jar:1.8.0_222-ea]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:170) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.disconnectManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:160) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_222-ea]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:237) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view8.disconnectManagedBlockStorageDisks(Unknown Source) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_222-ea]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.disconnectManagedBlockStorageDisks(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.ProcessDownVmCommand.executeCommand(ProcessDownVmCommand.java:144) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:]
	at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:639) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor188.invoke(Unknown Source) [:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.2.GA-redhat-00001.jar:7.2.2.GA-redhat-00001]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source) [:1.8.0_222-ea]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222-ea]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222-ea]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.6.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.6.Final-redhat-00001]
	at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2354) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2379) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInternalActionWithTasksContext(CommandBase.java:2374) [bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommandBase.lambda$processVmOnDown$0(RunVmCommandBase.java:210) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_222-ea]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_222-ea]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_222-ea]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_222-ea]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222-ea]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]

Expected results:
Start VM with Ceph MBD disk should succeed

Additional info:

Comment 1 Shir Fishbain 2019-07-03 16:29:23 UTC

Created attachment 1587112 [details]
logs_ceph

Comment 2 Fred Rolland 2019-07-04 07:25:55 UTC

There is a configuration mismatch between the host and the ceph server:

'RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".\\\\nIn some cases useful info is found in syslog - try "dmesg | tail"

Comment 3 Shir Fishbain 2019-08-21 14:45:50 UTC

Verified with DEV ENV

ovirt-engine-4.3.6.2-0.1.el7.noarch
vdsm-4.30.24-2.el7ev.x86_64

Comment 4 Sandro Bonazzola 2019-09-26 19:43:30 UTC

This bugzilla is included in oVirt 4.3.6 release, published on September 26th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.6 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Note You need to log in before you can comment on or make changes to this bug.