I've just updated to 4.1.1.8 async release However this error exists in 4.1.0 release as well: 2017-04-06 14:59:57,141+03 ERROR [org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-6-thread-19) [f584873] Command 'org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand' failed: Unable to determine the correct call signature - no procedure/function/signature for 'deleteluns' 2017-04-06 14:59:57,142+03 ERROR [org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-6-thread-19) [f584873] Exception: org.springframework.dao.InvalidDataAccessApiUsageException: Unable to determine the correct call signature - no procedure/function/signature for 'deleteluns' at org.springframework.jdbc.core.metadata.GenericCallMetaDataProvider.processProcedureColumns(GenericCallMetaDataProvider.java:347) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.metadata.GenericCallMetaDataProvider.initializeWithProcedureColumnMetaData(GenericCallMetaDataProvider.java:112) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.metadata.CallMetaDataProviderFactory$1.processMetaData(CallMetaDataProviderFactory.java:133) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:299) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.metadata.CallMetaDataProviderFactory.createMetaDataProvider(CallMetaDataProviderFactory.java:73) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.metadata.CallMetaDataContext.initializeMetaData(CallMetaDataContext.java:286) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.compileInternal(AbstractJdbcCall.java:303) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.compile(AbstractJdbcCall.java:288) [spring-jdbc.jar:4.2.4.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.getCall(SimpleJdbcCallsHandler.java:157) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.getCall(SimpleJdbcCallsHandler.java:150) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback.fillProcMetaData(BatchProcedureExecutionConnectionCallback.java:92) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback.getStoredProcedureMetaData(BatchProcedureExecutionConnectionCallback.java:84) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback.doInConnection(BatchProcedureExecutionConnectionCallback.java:51) [dal.jar:] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:349) [spring-jdbc.jar:4.2.4.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeStoredProcAsBatch(SimpleJdbcCallsHandler.java:58) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeStoredProcAsBatch(SimpleJdbcCallsHandler.java:72) [dal.jar:] at org.ovirt.engine.core.dao.MassOperationsGenericDao.removeAllInBatch(MassOperationsGenericDao.java:75) [dal.jar:] at org.ovirt.engine.core.dao.MassOperationsGenericDao.removeAllInBatch(MassOperationsGenericDao.java:86) [dal.jar:] at org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand.cleanupLunsFromDb(SyncLunsInfoForBlockStorageDomainCommand.java:124) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand.lambda$executeCommand$3(SyncLunsInfoForBlockStorageDomainCommand.java:78) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.bll.storage.domain.SyncLunsInfoForBlockStorageDomainCommand.executeCommand(SyncLunsInfoForBlockStorageDomainCommand.java:75) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1251) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1391) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2055) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1451) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:397) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:437) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:128) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.VdsEventListener.lambda$syncLunsInfoForBlockStorageDomain$2(VdsEventListener.java:275) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
Looking through the code history, frankly, I'm not sure how (if?) this ever worked. Should be a reasonably easy fix to deliver in 4.1.2 though.
Is it related to bug 1403839 ?
(In reply to Kapetanakis Giannis from comment #2) > Is it related to bug 1403839 ? Not as far as I can tell. But perhaps Liron (who implemented that RFE) would have a different insight. Liron?
(In reply to Allon Mureinik from comment #3) > (In reply to Kapetanakis Giannis from comment #2) > > Is it related to bug 1403839 ? > > Not as far as I can tell. But perhaps Liron (who implemented that RFE) would > have a different insight. > Liron? Hi Giannis/Aloon This RFE isn't related to that issue - the issue is in a different flow. I see that Allon already sent a fix, so it should be merged soon. Regards, Liron
Allon/Liron, can you please add steps to reproduce?
To be completely honest, I worked back from the stack trace, created a DAO test and developed the fix based on that. From reading the code, this faulty call should occur when you activate a block domain and the database has an old lun that isn't returned by getVGInfo. I think the easiest way to simulate this is to move a block domain to maintenance and then manually insert a row to the "luns" table. Unless Kapetanakis has a better suggestion?
no suggestion from me :)
Set block domain to maintenance and inserted a line to 'luns' table: engine=# insert INTO luns values ('lun','lun','lun','lun','3','lun','lun','10','233','t'); INSERT 0 1 physical_volume_id | lun_id ----------------------------------------+------------------- XJRsDH-13bf-29bB-6c9G-6vMP-j3pL-CXr3XI | 3514f0c5a516004ac 2Vlo30-ciJY-d6xf-zEVz-YYY8-sbKl-IVJWEB | 3514f0c5a516004ab DSbwKT-rGJu-aOma-G8lf-vg9q-Yk52-ds4TZf | 3514f0c5a516004aa 4fpBIy-Hpxg-P0Ej-t9db-ZZt7-N4Ge-vloyaS | 3514f0c5a51600215 L7XnOI-A5r1-B8uq-gaOH-Sgbd-ySoB-mWDbdG | 3514f0c5a51600213 LJmZ5x-0avg-rS32-fyOw-1SdW-nLHY-m0fkQh | 3514f0c5a516004ad lun | lun T3SH1U-sgQq-ItqR-wl4h-1ZpY-NJiC-VQEm0X | 3514f0c5a51600214 (8 rows) Then, activated the domain. The LUN wasn't removed from 'luns' table. Allon, what am I missing?
Used: rhevm-4.1.2-0.1.el7.noarch
(In reply to Elad from comment #8) > Set block domain to maintenance and inserted a line to 'luns' table: > > engine=# insert INTO luns values > ('lun','lun','lun','lun','3','lun','lun','10','233','t'); > INSERT 0 1 > > physical_volume_id | lun_id > ----------------------------------------+------------------- > XJRsDH-13bf-29bB-6c9G-6vMP-j3pL-CXr3XI | 3514f0c5a516004ac > 2Vlo30-ciJY-d6xf-zEVz-YYY8-sbKl-IVJWEB | 3514f0c5a516004ab > DSbwKT-rGJu-aOma-G8lf-vg9q-Yk52-ds4TZf | 3514f0c5a516004aa > 4fpBIy-Hpxg-P0Ej-t9db-ZZt7-N4Ge-vloyaS | 3514f0c5a51600215 > L7XnOI-A5r1-B8uq-gaOH-Sgbd-ySoB-mWDbdG | 3514f0c5a51600213 > LJmZ5x-0avg-rS32-fyOw-1SdW-nLHY-m0fkQh | 3514f0c5a516004ad > lun | lun > T3SH1U-sgQq-ItqR-wl4h-1ZpY-NJiC-VQEm0X | 3514f0c5a51600214 > (8 rows) > > > > Then, activated the domain. The LUN wasn't removed from 'luns' table. > > Allon, what am I missing? This LUN is unrelated to the domain you're activating, so it won't be affected. You need to set the volume_group_id column to match the domain's.
Thanks Allon. Inserted the following row to 'luns' table while iscsi domain with VG UUID is vj7PzK-pAUn-u6tC-mgLG-0mKV-G9f9-8tGsQy was in maintenance (iscsi_0). storage_name | status ------------------------+-------- rhevm-qe-infra-glance | ovirt-image-repository | iscsi_1 | 3 fcp_1 | 3 test_gluster_1 | 3 nfs_1 | 3 fcp_2 | 3 fcp_0 | 3 nfs_0 | 3 nfs_2 | 3 export_domain | 3 test_gluster_2 | 3 test_gluster_0 | 3 iscsi_2 | 3 iscsi_0 | 6 (15 rows) engine=# insert INTO luns values ('pv_id','lun_id','vj7PzK-pAUn-u6tC-mgLG-0mKV-G9f9-8tGsQy','serial','3','vendor','product','11','233','t'); Then, activated the domain. The row didn't get removed upon domain activation. Logs will be provided.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Created attachment 1277090 [details] engine.log
The urgent part for 4.1.2 is that the activate should succeed, which it does. The remaining LUN that Elad reported in comment 11 is obviously still a bug though. Idan - can you look into this please?
Yes, sure. I will soon push a fix for it.
Inserted the following row to 'luns' table while iscsi domain with VG UUID is 8murUB-rAYZ-y63c-Oljm-bjy3-3Lwe-o7cDJ5 was in maintenance (iscsi_0). After domain activation, the inserted row got removed from luns table. Tested using: rhevm-4.1.3.1-0.1.el7.noarch postgresql-9.2.18-1.el7.x86_64 vdsm-4.19.17-1.el7ev.x86_64
* The row instereted to luns table: insert INTO luns values ('pv_id','lun_id','8murUB-rAYZ-y63c-Oljm-bjy3-3Lwe-o7cDJ5','serial','3','vendor','product','11','233','t');