Bug 1269531
| Summary: | Psql exceptions while performing multiple operations with Cinder provider cause engine timeouts | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Ori Gofen <ogofen> | ||||
| Component: | ovirt-engine | Assignee: | Daniel Erez <derez> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Ori Gofen <ogofen> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.6.0 | CC: | acanan, amureini, eedri, gklein, lsurette, rbalakri, Rhev-m-bugs, tnisan, yeylon, ykaul | ||||
| Target Milestone: | ovirt-3.6.1 | ||||||
| Target Release: | 3.6.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | 3.6.0-16 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Looks like the usual race condition in the search mechanism. Daniel, please take a look? fixed version & build id rhevm-3.6.0.1-0.1.el6.noarch RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE |
Created attachment 1080690 [details] log Description of problem: While performing multi-threaded operations of add vm+cinder disk and removing it, have noticed the engine reports very slowly, the engine log was flooded with postgresql ERROR messages such as: 2015-10-07 14:14:34,404 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8702-24) [] Query 'SearchQuery' failed: StatementCallback; bad SQL grammar [SELECT * FRO M ((SELECT distinct providers.* FROM providers ) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. 2015-10-07 14:14:34,404 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8702-24) [] Exception: org.springframework.jdbc.BadSqlGrammarException: StatementCallbac k; bad SQL grammar [SELECT * FROM ((SELECT distinct providers.* FROM providers ) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util .PSQLException: The column name disk_storage_type was not found in this ResultSet. at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:98) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:456) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:464) [spring-jdbc.jar:3.1.1.RELEASE] at org.ovirt.engine.core.dao.DiskDaoImpl.getAllWithQuery(DiskDaoImpl.java:121) [dal.jar:] at org.ovirt.engine.core.bll.SearchQuery.genericSearch(SearchQuery.java:281) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.genericSearch(SearchQuery.java:269) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.searchDisk(SearchQuery.java:315) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.executeQueryCommand(SearchQuery.java:127) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:82) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:537) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:511) [bll.jar:] at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) [:1.7.0_85] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_85] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_85] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.5.3.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.5.3.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.5.3.Final-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.5.3.Final-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.5.3.Final-redhat-2] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.5.3.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) [:1.7.0_85] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_85] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_85] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) [jboss-as-ee.jar:7.5.3 the ui seem to drift as if it's calculating something big, it seems similar to the Future.get java interface bug, because of those time outs and because of the fact that I have run the same tests a while ago on the pre-integration environment without noticing anything like it. Version-Release number of selected component (if applicable): rhevm-3.6-14 How reproducible: ? Steps to Reproduce: will go over my tests and provide a detailed reproducer, as I mentioned, have encountered this behaviour while creating and removing vms with cinder disk (different vms, same time) Actual results: engine becomes very slow, time outs, log is flooded with postgresql exceptions Expected results: like on Fedora! Additional info: