Bug 1684889

Summary: [cinderLib] ISCSI MBS SD's VM's can not be used when Data ISCSI SD's VM's are used - StartVM and HOT-PLUG with an HP3PAR-ISCSI MBS disk failes when an existing ISCSI Data SD is logged in to the same target
Product: [oVirt] ovirt-engine Reporter: Avihai <aefrat>
Component: BLL.StorageAssignee: Benny Zlotnik <bzlotnik>
Status: CLOSED WONTFIX QA Contact: Avihai <aefrat>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: bugs, eshenitz, sfishbai, tnisan
Target Milestone: ---Flags: pm-rhel: ovirt-4.4+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-05 08:50:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1539837, 1673035    
Attachments:
Description Flags
engine and vdsm
none
2nd Env(storage-ge-8) reproduction logs
none
engine and vdsm logs H3PAR hot plug issue none

Description Avihai 2019-03-03 16:03:46 UTC 
Created attachment 1540321 [details]
engine and vdsm

Description of problem:
StartVM with an HP3PAR block storage domain disk fails 

VDSM log:
2019-03-03 17:55:38,907+0200 WARN  (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'targ
et_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iq
n.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'9b25f5b1-c32c-4961-a24c-629
12a18b9f0'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'c57cfb4a-ba61-448e-b78a-b404c25132f3'} at 0x7fac66de1d50> timeout=60, duration=2940.17 at 0x7fac66de1a90> task#=5 at 0x7fac7a7953d
0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
  self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
  self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run
  ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
  self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task
  task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
  self._callable()
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__
  self._handler(self._ctx, self._req)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest
  response = self._handle_request(req, ctx)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request
  res = method(**params)
File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod
  result = fn(*methodArgs)
File: "<string>", line 2, in attach_volume
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
  ret = func(*args, **kwargs)
File: "<string>", line 2, in attach_volume
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
  ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume
  return managedvolume.attach_volume(vol_id, connection_info)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper
  return func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume
  attachment = run_helper("attach", connection_info)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper
  sub_cmd, cmd_input=cmd_input)
File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__
  return callMethod()
File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>
  **kwargs)
File: "<string>", line 2, in managedvolume_run_helper
File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod
  kind, result = conn.recv() (executor:363)
2019-03-03 17:55:39,059+0200 DEBUG (mailbox-spm) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 dd if=/rhev/data-center/c13e66e9-5640-4b49-97f3-704c2a36bbf4/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) (commands:198)


Engine log:
2019-03-03 17:51:14,070+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] START, AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='24d3e2d4-feab-482f-8f47-52aa8d79c14c', vds='Host[host_mixed_2,24d3e2d4-feab-482f-8f47-52aa8d79c14c]'}), log id: 3d3f6fa8
2019-03-03 17:52:05,527+02 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [] EVENT_ID: VDS_HIGH_CPU_USE(534), Used CPU of host host_mixed_2 [96%] exceeded defined threshold [95%].
2019-03-03 17:54:14,137+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Failed in 'AttachManagedBlockStorageVolumeVDS' method, for vds: 'host_mixed_2'; host: 'storage-ge4-vdsm2.scl.lab.tlv.redhat.com': null
2019-03-03 17:54:14,139+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Command 'AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='24d3e2d4-feab-482f-8f47-52aa8d79c14c', vds='Host[host_mixed_2,24d3e2d4-feab-482f-8f47-52aa8d79c14c]'})' execution failed: null
2019-03-03 17:54:14,139+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] FINISH, AttachManagedBlockStorageVolumeVDSCommand, return: , log id: 3d3f6fa8
2019-03-03 17:54:14,175+02 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Lock freed to object 'EngineLock:{exclusiveLocks='[6cc984d7-4357-4105-b03a-5a2c28b61053=VM]', sharedLocks=''}'
2019-03-03 17:54:14,178+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Command 'org.ovirt.engine.core.bll.RunVmCommand' failed: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
2019-03-03 17:54:14,178+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Exception: javax.ejb.EJBException: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:246) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view7.attachManagedBlockStorageDisks(Unknown Source) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor1746.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.attachManagedBlockStorageDisks(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommand.runVm(RunVmCommand.java:263) [bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommand.perform(RunVmCommand.java:432) [bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommand.executeVmCommand(RunVmCommand.java:357) [bll.jar:]
	at org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:159) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1147) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1305) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1954) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1365) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:413) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:]
	at org.ovirt.engine.core.bll.SortedMultipleActionsRunnerBase.runCommands(SortedMultipleActionsRunnerBase.java:20) [bll.jar:]
	at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_201]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]
	at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
Caused by: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:121) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:71) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.lambda$attachManagedBlockStorageDisks$0(ManagedBlockStorageCommandUtil.java:66) [bll.jar:]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) [rt.jar:1.8.0_201]
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) [rt.jar:1.8.0_201]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) [rt.jar:1.8.0_201]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) [rt.jar:1.8.0_201]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) [rt.jar:1.8.0_201]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) [rt.jar:1.8.0_201]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) [rt.jar:1.8.0_201]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) [rt.jar:1.8.0_201]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_201]
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) [rt.jar:1.8.0_201]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:64) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor1746.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:237) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	... 67 more
Caused by: java.lang.NullPointerException
	at java.util.HashMap.putMapEntries(HashMap.java:501) [rt.jar:1.8.0_201]
	at java.util.HashMap.<init>(HashMap.java:490) [rt.jar:1.8.0_201]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.DeviceInfoReturn.<init>(DeviceInfoReturn.java:19) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.attachManagedBlockStorageVolume(JsonRpcVdsServer.java:2525) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand.executeVdsBrokerCommand(AttachManagedBlockStorageVolumeVDSCommand.java:14) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:]
	... 104 more

2019-03-03 17:54:14,232+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm1_hpiscsi  (User: admin@internal-authz).


Version-Release number of selected component (if applicable):
ovirt-engine-4.3.1.1-0.1.el7.noarch
vdsm-4.30.9-1.el7ev.x86_64
CinderLib Version: 0.3.9


How reproducible:
100%

Steps to Reproduce:
1.Create managed block storage SD with HP3PAR backend (see in additional info the RESTAPI)

* From here done via webadmin:
2.Create VM
3.Create a disk from the SD in step 1 and activate and add it to VM as bootable
4. Start VM

Actual results:
VM start fails

Expected results:
VM start should succeed

Additional info:
REST API :

URL used:
https://storage-ge-04.scl.lab.tlv.redhat.com/ovirt-engine/api/storagedomains

Method: POST .

body:
<storage_domain>
   <name>cinder-hp3par2</name>
   <type>managed_block_storage</type>
   <storage>
      <type>managed_block_storage</type>
      <driver_options>
      	<property>
            <name>hpe3par_api_url</name>
            <value>https://3par-cli.mgmt.lab3.tlv.redhat.com:8080/api/v1</value>
         </property>
      	<property>
            <name>san_ip</name>
            <value>10.35.84.14</value>
         </property>
      	<property>
            <name>san_login</name>
            <value>admin</value>
         </property>
         <property>
            <name>san_password</name>
            <value>Qum!0net</value>
         </property>
         <property>
            <name>hpe3par_username</name>
            <value>3paredit</value>
         </property>
         <property>
            <name>hpe3par_password</name>
            <value>123456</value>
         </property>
         <property>
            <name>hpe3par_cpg</name>
            <value>SSD_r1</value>
         </property>
         <property>
            <name>volume_driver</name>
            <value>cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver</value>
         </property>
         <property>
            <name>hpe3par_debug</name>
            <value>False</value>
         </property>
         <property>
            <name>hpe3par_iscsi_chap_enabled</name>
            <value>False</value>
         </property>
         <property>
            <name>hpe3par_snapshot_retention</name>
            <value>48</value>
         </property>
         <property>
            <name>hpe3par_snapshot_expiration</name>
            <value>72</value>
         </property>
         <property>
            <name>max_over_subscription_ratio</name>
            <value>20.0</value>
         </property>
         <property>
            <name>reserved_percentage</name>
            <value>15</value>
         </property>
         <property>
            <name>hpe3par_iscsi_ips</name>
            <value>10.35.146.1,10.35.146.2,10.35.146.3,10.35.146.4</value>
         </property>
      </driver_options>
   </storage>
   <host>
      <name>host_mixed_1</name>
   </host>
</storage_domain>
Comment 1 Avihai 2019-03-04 12:03:53 UTC 
This bug reproduces also in other Env's and causes the CPU of the host to be stuck on at least 90% CPU.

Raising this to urgent.
Comment 2 Avihai 2019-03-04 12:34:28 UTC 
See logs from the 2nd env(storage-ge-08) attached.

From message log of the host:

Mar  4 13:57:20 storage-ge8-vdsm2 vdsm[21677]: WARN Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'target_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iqn.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'cdbb6e3e-83f9-4370-aae4-c7977484dc8f'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'd9290dc6-c75e-4e14-8c85-fedbd9135f9e'} at 0x7f5fd067d090> timeout=60, duration=240.00 at 0x7f5fd067ded0> task#=20 at 0x7f60000293d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012  self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012  self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012  self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012  ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012  self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012  task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012  self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012  self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012  response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012  res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012  result = fn(*methodArgs)#012File: "<string>", line 2, in attach_volume#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012  ret = func(*args, **kwargs)#012File: "<string>", line 2, in attach_volume#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method#012  ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume#012  return managedvolume.attach_volume(vol_id, connection_info)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper#012  return func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume#012  attachment = run_helper("attach", connection_info)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper#012  sub_cmd, cmd_input=cmd_input)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012  return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012  **kwargs)#012File: "<string>", line 2, in managedvolume_run_helper#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012  kind, result = conn.recv()
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19179): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19180): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19181): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19182): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19183): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19184): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
Mar  4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19185): avc:  denied  { read } for  pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0


VDSM:
2019-03-04 13:57:20,539+0200 WARN  (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'target_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iqn.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'cdbb6e3e-83f9-4370-aae4-c7977484dc8f'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'd9290dc6-c75e-4e14-8c85-fedbd9135f9e'} at 0x7f5fd067d090> timeout=60, duration=240.01 at 0x7f5fd067ded0> task#=20 at 0x7f60000293d0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
  self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
  self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
  self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run
  ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
  self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task
  task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
  self._callable()
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__
  self._handler(self._ctx, self._req)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest
  response = self._handle_request(req, ctx)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request
  res = method(**params)
File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod
  result = fn(*methodArgs)
File: "<string>", line 2, in attach_volume
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
  ret = func(*args, **kwargs)
File: "<string>", line 2, in attach_volume
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
  ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume
  return managedvolume.attach_volume(vol_id, connection_info)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper
  return func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume
  attachment = run_helper("attach", connection_info)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper
  sub_cmd, cmd_input=cmd_input)
File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__
  return callMethod()
File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>
  **kwargs)
File: "<string>", line 2, in managedvolume_run_helper
File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod
  kind, result = conn.recv() (executor:363)
2019-03-04 13:57:20,891+0200 DEBUG (check/loop) [storage.check] START check '/dev/70d44c6d-958c-4444-aa6d-8f752004f0fa/metadata' (delay=0.00) (check:289)
2019-03-04 13:57:20,946+0200 DEBUG (check/loop) [storage.asyncevent] Process <subprocess32.Popen object at 0x7f5fd068da10> terminated (count=1) (asyncevent:659)


Host 'top' :
top - 14:34:01 up 3 days, 20:16,  1 user,  load average: 2.54, 2.70, 2.78
Tasks: 219 total,   3 running, 216 sleeping,   0 stopped,   0 zombie
%Cpu(s): 30.8 us, 65.2 sy,  0.0 ni,  3.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.3 st
KiB Mem :  3878232 total,   284112 free,  2664384 used,   929736 buff/cache
KiB Swap:  2097148 total,  2075388 free,    21760 used.   804388 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                         
24423 root      15  -5  201744  29636   1940 S 15.5  0.8   6:02.59 privsep-helper                                                                                                                                  
21677 vdsm       0 -20 1391308  72268  13920 S  3.6  1.9   1:52.88 vdsmd                                                                                                                                           
12456 root      20   0       0      0      0 S  2.6  0.0   0:02.68 kworker/0:2                                                                                                                                     
24384 root      15  -5  997508  43768   8160 S  2.6  1.1   0:57.67 managedvolume-h                                                                                                                                 
14824 root      20   0  162144   2692   1864 R  0.6  0.1   0:00.49 top                                                                                                                                             
20915 root      20   0  911032  28224   8648 S  0.6  0.7  12:44.16 python                                                                                                                                          
 7091 root      20   0       0      0      0 S  0.3  0.0   0:00.59 kworker/u32:12                                                                                                                                  
16843 root      15  -5   29892   1520    768 R  0.3  0.0   0:00.01 iscsiadm                                                                                                                                        
21679 vdsm      20   0  601820  24972   5368 S  0.3  0.6   0:26.10 python                                                                                                                                          
22073 vdsm       0 -20  401336    900    688 S  0.3  0.0   0:00.37 ioprocess                                                                                                                                       
22549 sanlock   rt   0  692340  35044   2700 S  0.3  0.9  14:08.54 sanlock                                                                                                                                         
22590 root      rt   0  589624   8820   6352 S  0.3  0.2   2:50.94 multipathd
Comment 3 Avihai 2019-03-04 12:35:21 UTC 
Created attachment 1540635 [details]
2nd Env(storage-ge-8) reproduction logs
Comment 4 Avihai 2019-03-06 19:23:14 UTC 
Hot plug an H3PAR managed block disk to a running VM cause the same issue.

Hot plug (HP3PAR SD) fails with NPE.
Engine and VDSM logs are attached.

Engine:
2019-03-06 20:15:51,887+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] Failed in 'AttachManagedBlockStorageVolumeVDS' method, for vds: 'host_mixed_2'; host: 'storage-ge8-vdsm2.scl.lab.tlv.redhat.com': null
2019-03-06 20:15:51,889+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] Command 'AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='9d79dc93-59e1-4e91-b935-5cf80923f839', vds='Host[host_mixed_2,9d79dc93-59e1-4e91-b935-5cf80923f839]'})' execution failed: null
2019-03-06 20:15:51,889+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] FINISH, AttachManagedBlockStorageVolumeVDSCommand, return: , log id: 8f88ecc
2019-03-06 20:15:51,961+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Command 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand' failed: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
2019-03-06 20:15:51,961+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Exception: javax.ejb.EJBTransactionRolledbackException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:364) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view8.saveDevices(Unknown Source) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.saveDevices(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.AbstractDiskVmCommand.performPlugCommand(AbstractDiskVmCommand.java:121) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand.executeVmCommand(AttachDiskToVmCommand.java:243) [bll.jar:]
	at org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:172) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:]
	at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor406.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.5.Final-redhat-00001]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runAction(Unknown Source) [common.jar:]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:176)
	at sun.reflect.GeneratedMethodAccessor824.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:706) [jboss-servlet-api_4.0_spec.jar:1.0.0.Final-redhat-1]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec.jar:1.0.0.Final-redhat-1]
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201]
Caused by: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:]
	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:145) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:88) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:80) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
	at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005]
	... 202 more
Caused by: java.lang.NullPointerException
	at java.util.HashMap.putMapEntries(HashMap.java:501) [rt.jar:1.8.0_201]
	at java.util.HashMap.<init>(HashMap.java:490) [rt.jar:1.8.0_201]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.DeviceInfoReturn.<init>(DeviceInfoReturn.java:19) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.attachManagedBlockStorageVolume(JsonRpcVdsServer.java:2525) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand.executeVdsBrokerCommand(AttachManagedBlockStorageVolumeVDSCommand.java:14) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) [:1.8.0_201]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
	at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl.jar:3.0.5.Final-redhat-00001]
	at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:]
	... 229 more

2019-03-06 20:15:52,041+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand'.
2019-03-06 20:15:52,133+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-19) [4ab2a2b5] EVENT_ID: USER_FAILED_ATTACH_DISK_TO_VM(2,017), Failed to attach Disk hpar_disk to VM vm2 (User: admin@internal-authz).
Comment 5 Avihai 2019-03-06 19:24:07 UTC 
Created attachment 1541585 [details]
engine and vdsm logs H3PAR hot plug issue
Comment 6 Avihai 2019-07-21 09:00:07 UTC 
ISCSI MBS SD's VM's can not be used when Data ISCSI SD's VM's are used.
Meaning ISCSI MBS SD's and ISCSI Data domains can not really work together at all.
  
In order for one to work you need to ISCSI disconnect from all hosts(or unmap all hosts at the storage) the existing SD(Data/MBS) using the current ISCSI connections in order for the brickos/VDSM to succeed connecting to the storage when you VM up.
Comment 7 Tal Nisan 2019-07-22 14:17:05 UTC 
*** Bug 1728255 has been marked as a duplicate of this bug. ***
Comment 9 Benny Zlotnik 2021-05-05 08:50:38 UTC 
Currently no plans to support iSCSI backends, additionally a setup of where iSCSI targets are used for both Vdsm managed SDs and MBS SDs cannot really work, closing