Created attachment 1540321 [details] engine and vdsm Description of problem: StartVM with an HP3PAR block storage domain disk fails VDSM log: 2019-03-03 17:55:38,907+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'targ et_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iq n.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'9b25f5b1-c32c-4961-a24c-629 12a18b9f0'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'c57cfb4a-ba61-448e-b78a-b404c25132f3'} at 0x7fac66de1d50> timeout=60, duration=2940.17 at 0x7fac66de1a90> task#=5 at 0x7fac7a7953d 0>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod result = fn(*methodArgs) File: "<string>", line 2, in attach_volume File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File: "<string>", line 2, in attach_volume File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume return managedvolume.attach_volume(vol_id, connection_info) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper return func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume attachment = run_helper("attach", connection_info) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper sub_cmd, cmd_input=cmd_input) File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File: "<string>", line 2, in managedvolume_run_helper File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() (executor:363) 2019-03-03 17:55:39,059+0200 DEBUG (mailbox-spm) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-0 dd if=/rhev/data-center/c13e66e9-5640-4b49-97f3-704c2a36bbf4/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) (commands:198) Engine log: 2019-03-03 17:51:14,070+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] START, AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='24d3e2d4-feab-482f-8f47-52aa8d79c14c', vds='Host[host_mixed_2,24d3e2d4-feab-482f-8f47-52aa8d79c14c]'}), log id: 3d3f6fa8 2019-03-03 17:52:05,527+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [] EVENT_ID: VDS_HIGH_CPU_USE(534), Used CPU of host host_mixed_2 [96%] exceeded defined threshold [95%]. 2019-03-03 17:54:14,137+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Failed in 'AttachManagedBlockStorageVolumeVDS' method, for vds: 'host_mixed_2'; host: 'storage-ge4-vdsm2.scl.lab.tlv.redhat.com': null 2019-03-03 17:54:14,139+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Command 'AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='24d3e2d4-feab-482f-8f47-52aa8d79c14c', vds='Host[host_mixed_2,24d3e2d4-feab-482f-8f47-52aa8d79c14c]'})' execution failed: null 2019-03-03 17:54:14,139+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] FINISH, AttachManagedBlockStorageVolumeVDSCommand, return: , log id: 3d3f6fa8 2019-03-03 17:54:14,175+02 INFO [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Lock freed to object 'EngineLock:{exclusiveLocks='[6cc984d7-4357-4105-b03a-5a2c28b61053=VM]', sharedLocks=''}' 2019-03-03 17:54:14,178+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Command 'org.ovirt.engine.core.bll.RunVmCommand' failed: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) 2019-03-03 17:54:14,178+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] Exception: javax.ejb.EJBException: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:246) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view7.attachManagedBlockStorageDisks(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor1746.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.attachManagedBlockStorageDisks(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.RunVmCommand.runVm(RunVmCommand.java:263) [bll.jar:] at org.ovirt.engine.core.bll.RunVmCommand.perform(RunVmCommand.java:432) [bll.jar:] at org.ovirt.engine.core.bll.RunVmCommand.executeVmCommand(RunVmCommand.java:357) [bll.jar:] at org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:159) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1147) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1954) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1365) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:413) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:] at org.ovirt.engine.core.bll.SortedMultipleActionsRunnerBase.runCommands(SortedMultipleActionsRunnerBase.java:20) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_201] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) Caused by: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:121) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:71) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.lambda$attachManagedBlockStorageDisks$0(ManagedBlockStorageCommandUtil.java:66) [bll.jar:] at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) [rt.jar:1.8.0_201] at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) [rt.jar:1.8.0_201] at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) [rt.jar:1.8.0_201] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) [rt.jar:1.8.0_201] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_201] at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) [rt.jar:1.8.0_201] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisks(ManagedBlockStorageCommandUtil.java:64) [bll.jar:] at sun.reflect.GeneratedMethodAccessor1746.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:237) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] ... 67 more Caused by: java.lang.NullPointerException at java.util.HashMap.putMapEntries(HashMap.java:501) [rt.jar:1.8.0_201] at java.util.HashMap.<init>(HashMap.java:490) [rt.jar:1.8.0_201] at org.ovirt.engine.core.vdsbroker.vdsbroker.DeviceInfoReturn.<init>(DeviceInfoReturn.java:19) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.attachManagedBlockStorageVolume(JsonRpcVdsServer.java:2525) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand.executeVdsBrokerCommand(AttachManagedBlockStorageVolumeVDSCommand.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] ... 104 more 2019-03-03 17:54:14,232+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-98113) [a323159] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm1_hpiscsi (User: admin@internal-authz). Version-Release number of selected component (if applicable): ovirt-engine-4.3.1.1-0.1.el7.noarch vdsm-4.30.9-1.el7ev.x86_64 CinderLib Version: 0.3.9 How reproducible: 100% Steps to Reproduce: 1.Create managed block storage SD with HP3PAR backend (see in additional info the RESTAPI) * From here done via webadmin: 2.Create VM 3.Create a disk from the SD in step 1 and activate and add it to VM as bootable 4. Start VM Actual results: VM start fails Expected results: VM start should succeed Additional info: REST API : URL used: https://storage-ge-04.scl.lab.tlv.redhat.com/ovirt-engine/api/storagedomains Method: POST . body: <storage_domain> <name>cinder-hp3par2</name> <type>managed_block_storage</type> <storage> <type>managed_block_storage</type> <driver_options> <property> <name>hpe3par_api_url</name> <value>https://3par-cli.mgmt.lab3.tlv.redhat.com:8080/api/v1</value> </property> <property> <name>san_ip</name> <value>10.35.84.14</value> </property> <property> <name>san_login</name> <value>admin</value> </property> <property> <name>san_password</name> <value>Qum!0net</value> </property> <property> <name>hpe3par_username</name> <value>3paredit</value> </property> <property> <name>hpe3par_password</name> <value>123456</value> </property> <property> <name>hpe3par_cpg</name> <value>SSD_r1</value> </property> <property> <name>volume_driver</name> <value>cinder.volume.drivers.hpe.hpe_3par_iscsi.HPE3PARISCSIDriver</value> </property> <property> <name>hpe3par_debug</name> <value>False</value> </property> <property> <name>hpe3par_iscsi_chap_enabled</name> <value>False</value> </property> <property> <name>hpe3par_snapshot_retention</name> <value>48</value> </property> <property> <name>hpe3par_snapshot_expiration</name> <value>72</value> </property> <property> <name>max_over_subscription_ratio</name> <value>20.0</value> </property> <property> <name>reserved_percentage</name> <value>15</value> </property> <property> <name>hpe3par_iscsi_ips</name> <value>10.35.146.1,10.35.146.2,10.35.146.3,10.35.146.4</value> </property> </driver_options> </storage> <host> <name>host_mixed_1</name> </host> </storage_domain>
This bug reproduces also in other Env's and causes the CPU of the host to be stuck on at least 90% CPU. Raising this to urgent.
See logs from the 2nd env(storage-ge-08) attached. From message log of the host: Mar 4 13:57:20 storage-ge8-vdsm2 vdsm[21677]: WARN Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'target_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iqn.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'cdbb6e3e-83f9-4370-aae4-c7977484dc8f'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'd9290dc6-c75e-4e14-8c85-fedbd9135f9e'} at 0x7f5fd067d090> timeout=60, duration=240.00 at 0x7f5fd067ded0> task#=20 at 0x7f60000293d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "<string>", line 2, in attach_volume#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "<string>", line 2, in attach_volume#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume#012 return managedvolume.attach_volume(vol_id, connection_info)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper#012 return func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume#012 attachment = run_helper("attach", connection_info)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper#012 sub_cmd, cmd_input=cmd_input)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in managedvolume_run_helper#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19179): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19180): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19181): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19182): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19183): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19184): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 Mar 4 13:57:40 storage-ge8-vdsm2 kernel: type=1400 audit(1551700660.476:19185): avc: denied { read } for pid=2917 comm="qemu-ga" name="dev" dev="proc" ino=4026531976 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0 VDSM: 2019-03-04 13:57:20,539+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connection_info': {u'driver_volume_type': u'iscsi', u'data': {u'target_discovered': True, u'target_luns': [6, 6, 6, 6], u'target_iqns': [u'iqn.2000-05.com.3pardata:20210002ac021f6b', u'iqn.2000-05.com.3pardata:20220002ac021f6b', u'iqn.2000-05.com.3pardata:21210002ac021f6b', u'iqn.2000-05.com.3pardata:21220002ac021f6b'], u'target_portals': [u'10.35.146.1:3260', u'10.35.146.3:3260', u'10.35.146.2:3260', u'10.35.146.4:3260'], u'encrypted': False}}, u'vol_id': u'cdbb6e3e-83f9-4370-aae4-c7977484dc8f'}, 'jsonrpc': '2.0', 'method': u'ManagedVolume.attach_volume', 'id': u'd9290dc6-c75e-4e14-8c85-fedbd9135f9e'} at 0x7f5fd067d090> timeout=60, duration=240.01 at 0x7f5fd067ded0> task#=20 at 0x7f60000293d0>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod result = fn(*methodArgs) File: "<string>", line 2, in attach_volume File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File: "<string>", line 2, in attach_volume File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1763, in attach_volume return managedvolume.attach_volume(vol_id, connection_info) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 67, in wrapper return func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 98, in attach_volume attachment = run_helper("attach", connection_info) File: "/usr/lib/python2.7/site-packages/vdsm/storage/managedvolume.py", line 179, in run_helper sub_cmd, cmd_input=cmd_input) File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File: "<string>", line 2, in managedvolume_run_helper File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod kind, result = conn.recv() (executor:363) 2019-03-04 13:57:20,891+0200 DEBUG (check/loop) [storage.check] START check '/dev/70d44c6d-958c-4444-aa6d-8f752004f0fa/metadata' (delay=0.00) (check:289) 2019-03-04 13:57:20,946+0200 DEBUG (check/loop) [storage.asyncevent] Process <subprocess32.Popen object at 0x7f5fd068da10> terminated (count=1) (asyncevent:659) Host 'top' : top - 14:34:01 up 3 days, 20:16, 1 user, load average: 2.54, 2.70, 2.78 Tasks: 219 total, 3 running, 216 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.8 us, 65.2 sy, 0.0 ni, 3.0 id, 0.0 wa, 0.0 hi, 0.7 si, 0.3 st KiB Mem : 3878232 total, 284112 free, 2664384 used, 929736 buff/cache KiB Swap: 2097148 total, 2075388 free, 21760 used. 804388 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24423 root 15 -5 201744 29636 1940 S 15.5 0.8 6:02.59 privsep-helper 21677 vdsm 0 -20 1391308 72268 13920 S 3.6 1.9 1:52.88 vdsmd 12456 root 20 0 0 0 0 S 2.6 0.0 0:02.68 kworker/0:2 24384 root 15 -5 997508 43768 8160 S 2.6 1.1 0:57.67 managedvolume-h 14824 root 20 0 162144 2692 1864 R 0.6 0.1 0:00.49 top 20915 root 20 0 911032 28224 8648 S 0.6 0.7 12:44.16 python 7091 root 20 0 0 0 0 S 0.3 0.0 0:00.59 kworker/u32:12 16843 root 15 -5 29892 1520 768 R 0.3 0.0 0:00.01 iscsiadm 21679 vdsm 20 0 601820 24972 5368 S 0.3 0.6 0:26.10 python 22073 vdsm 0 -20 401336 900 688 S 0.3 0.0 0:00.37 ioprocess 22549 sanlock rt 0 692340 35044 2700 S 0.3 0.9 14:08.54 sanlock 22590 root rt 0 589624 8820 6352 S 0.3 0.2 2:50.94 multipathd
Created attachment 1540635 [details] 2nd Env(storage-ge-8) reproduction logs
Hot plug an H3PAR managed block disk to a running VM cause the same issue. Hot plug (HP3PAR SD) fails with NPE. Engine and VDSM logs are attached. Engine: 2019-03-06 20:15:51,887+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] Failed in 'AttachManagedBlockStorageVolumeVDS' method, for vds: 'host_mixed_2'; host: 'storage-ge8-vdsm2.scl.lab.tlv.redhat.com': null 2019-03-06 20:15:51,889+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] Command 'AttachManagedBlockStorageVolumeVDSCommand(HostName = host_mixed_2, AttachManagedBlockStorageVolumeVDSCommandParameters:{hostId='9d79dc93-59e1-4e91-b935-5cf80923f839', vds='Host[host_mixed_2,9d79dc93-59e1-4e91-b935-5cf80923f839]'})' execution failed: null 2019-03-06 20:15:51,889+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-19) [4ab2a2b5] FINISH, AttachManagedBlockStorageVolumeVDSCommand, return: , log id: 8f88ecc 2019-03-06 20:15:51,961+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Command 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand' failed: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) 2019-03-06 20:15:51,961+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Exception: javax.ejb.EJBTransactionRolledbackException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:364) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.ejb3.component.singleton.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:106) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$$$view8.saveDevices(Unknown Source) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil$Proxy$_$$_Weld$EnterpriseProxy$.saveDevices(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.AbstractDiskVmCommand.performPlugCommand(AbstractDiskVmCommand.java:121) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand.executeVmCommand(AttachDiskToVmCommand.java:243) [bll.jar:] at org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:172) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) [bll.jar:] at sun.reflect.GeneratedMethodAccessor406.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb.jar:3.0.5.Final-redhat-00001] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:176) at sun.reflect.GeneratedMethodAccessor824.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:706) [jboss-servlet-api_4.0_spec.jar:1.0.0.Final-redhat-1] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78) at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec.jar:1.0.0.Final-redhat-1] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] Caused by: org.ovirt.engine.core.common.errors.EngineException: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.attachManagedBlockStorageDisk(ManagedBlockStorageCommandUtil.java:145) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:88) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.managedblock.ManagedBlockStorageCommandUtil.saveDevices(ManagedBlockStorageCommandUtil.java:80) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:185) [wildfly-ejb3-7.2.0.GA-redhat-00005.jar:7.2.0.GA-redhat-00005] ... 202 more Caused by: java.lang.NullPointerException at java.util.HashMap.putMapEntries(HashMap.java:501) [rt.jar:1.8.0_201] at java.util.HashMap.<init>(HashMap.java:490) [rt.jar:1.8.0_201] at org.ovirt.engine.core.vdsbroker.vdsbroker.DeviceInfoReturn.<init>(DeviceInfoReturn.java:19) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.attachManagedBlockStorageVolume(JsonRpcVdsServer.java:2525) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand.executeVdsBrokerCommand(AttachManagedBlockStorageVolumeVDSCommand.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl.jar:3.0.5.Final-redhat-00001] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] ... 229 more 2019-03-06 20:15:52,041+02 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-19) [4ab2a2b5] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand'. 2019-03-06 20:15:52,133+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-19) [4ab2a2b5] EVENT_ID: USER_FAILED_ATTACH_DISK_TO_VM(2,017), Failed to attach Disk hpar_disk to VM vm2 (User: admin@internal-authz).
Created attachment 1541585 [details] engine and vdsm logs H3PAR hot plug issue
ISCSI MBS SD's VM's can not be used when Data ISCSI SD's VM's are used. Meaning ISCSI MBS SD's and ISCSI Data domains can not really work together at all. In order for one to work you need to ISCSI disconnect from all hosts(or unmap all hosts at the storage) the existing SD(Data/MBS) using the current ISCSI connections in order for the brickos/VDSM to succeed connecting to the storage when you VM up.
*** Bug 1728255 has been marked as a duplicate of this bug. ***
Currently no plans to support iSCSI backends, additionally a setup of where iSCSI targets are used for both Vdsm managed SDs and MBS SDs cannot really work, closing