1482099 – Metrics deployment fails using the advanced ansible deployer

Bug 1482099 - Metrics deployment fails using the advanced ansible deployer

Summary: Metrics deployment fails using the advanced ansible deployer

Keywords:
Status:	CLOSED DUPLICATE of bug 1492159
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Hawkular
Sub Component:
Version:	3.5.1
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	3.5.z
Assignee:	John Sanda
QA Contact:	Junqi Zhao
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-08-16 13:01 UTC by Sandeep Dhar
Modified:	2021-06-10 12:49 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-10-05 19:24:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
web console screenshot (161.93 KB, image/png) 2017-08-16 13:01 UTC, Sandeep Dhar	no flags	Details
View All

Description Sandeep Dhar 2017-08-16 13:01:02 UTC

Created attachment 1314114 [details]
web console screenshot

Description of problem: After a successful OCP 3.5 deployment, the hawkular metrics logs show an error that cannot talk with Cassandra and the Cassandra logs show a column family id mismatch and restart.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Prepare atomic hosts
2. Prepare bastion host
3. Prepare /etc/ansible/host for hosted metrics deployment
4. Run advanced deployer config.yml to deploy
5. 

Actual results: Run successful, but metrics not working from the web console


Expected results: Metrics should be working


Additional info:
Hawkular failure log output:
 
2017-08-15 11:56:17,506 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 78) MSC000001: Failed to start service jboss.deployment.subunit."hawkular-metrics.ear"."hawkular-alerts.war".component.CassCluster.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."hawkular-metrics.ear"."hawkular-alerts.war".component.CassCluster.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
       at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
       at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
       at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
       at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
       at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
       at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
       at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
       ... 6 more
Caused by: javax.ejb.EJBException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.77.181:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/172.30.77.181:9042] Timed out waiting for server response))
       at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
       at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
       at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
       at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
       at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
       at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
       ... 11 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.77.181:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/172.30.77.181:9042] Timed out waiting for server response))
       at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
       at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
       at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
       at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
       at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
       at com.datastax.driver.core.Session$execute$0.call(Unknown Source)
       at org.cassalog.core.CassalogImpl.executeCQL(CassalogImpl.groovy:351)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
       at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
       at org.cassalog.core.CassalogImpl$_applyChangeSet_closure16.doCall(CassalogImpl.groovy:323)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
       at groovy.lang.Closure.call(Closure.java:426)
       at groovy.lang.Closure.call(Closure.java:442)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2030)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2015)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2056)
       at org.codehaus.groovy.runtime.dgm$162.invoke(Unknown Source)
       at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
       at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
       at org.cassalog.core.CassalogImpl.applyChangeSet(CassalogImpl.groovy:323)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
       at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
       at org.cassalog.core.CassalogImpl$_execute_closure3.doCall(CassalogImpl.groovy:130)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
       at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
       at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
       at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
       at groovy.lang.Closure.call(Closure.java:426)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.eachWithIndex(DefaultGroovyMethods.java:1946)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.eachWithIndex(DefaultGroovyMethods.java:1926)
       at org.codehaus.groovy.runtime.DefaultGroovyMethods.eachWithIndex(DefaultGroovyMethods.java:1976)
       at org.codehaus.groovy.runtime.dgm$174.invoke(Unknown Source)
       at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
       at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
       at org.cassalog.core.CassalogImpl.execute(CassalogImpl.groovy:109)
       at org.cassalog.core.Cassalog$execute.callCurrent(Unknown Source)
       at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
       at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
       at org.cassalog.core.CassalogImpl.execute(CassalogImpl.groovy:57)
       at org.hawkular.alerts.engine.impl.CassCluster.createSchema(CassCluster.java:390)
       at org.hawkular.alerts.engine.impl.CassCluster.initScheme(CassCluster.java:287)
       at org.hawkular.alerts.engine.impl.CassCluster.initSchemeDistributed(CassCluster.java:281)
       at org.hawkular.alerts.engine.impl.CassCluster.initCassCluster(CassCluster.java:262)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
       at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
       at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
       at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ee.component.ManagedReferenceFieldInjectionInterceptorFactory$ManagedReferenceFieldInjectionInterceptor.processInvocation(ManagedReferenceFieldInjectionInterceptorFactory.java:107)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ee.component.ManagedReferenceFieldInjectionInterceptorFactory$ManagedReferenceFieldInjectionInterceptor.processInvocation(ManagedReferenceFieldInjectionInterceptorFactory.java:107)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
       at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
       ... 29 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.77.181:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [hawkular-cassandra/172.30.77.181:9042] Timed out waiting for server response))
       at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:210)
       at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:46)
       at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:274)
       at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:405)
       at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:383)
       at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:771)
       at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1374)
       at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
       at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
       at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
       at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
       at java.lang.Thread.run(Thread.java:748)      
 
Cassandra is running. However I see errors in Cassandra container log entries pertaining hawkular_alerts:
 
INFO  [MigrationStage:1] 2017-08-15 07:13:09,210 ColumnFamilyStore.java:389 - Initializing hawkular_alerts.alerts_statuses
ERROR [MigrationStage:1] 2017-08-15 07:13:15,453 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:785) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1372) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1328) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1278) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_141]
       at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
       at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
ERROR [SharedPool-Worker-2] 2017-08-15 07:13:15,456 QueryMessage.java:128 - Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:402) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:513) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:343) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:313) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:87) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_141]
       at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_141]
       at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_141]
       at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:398) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       ... 19 common frames omitted
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:785) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1372) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1328) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1278) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_141]
       at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_141]
       at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       ... 1 common frames omitted
ERROR [SharedPool-Worker-2] 2017-08-15 07:13:15,458 ErrorMessage.java:349 - Unexpected exception during request
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:402) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:513) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:343) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:313) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:87) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_141]
       at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_141]
       at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_141]
       at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:398) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       ... 19 common frames omitted
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 30a96540-8189-11e7-88d5-5b05750bd455; expected 2888b500-8189-11e7-88d5-5b05750bd455)
       at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:785) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1372) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1328) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1278) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_141]
       at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_141]
       at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) ~[apache-cassandra-3.0.12.redhat-1.jar:3.0.12.redhat-1]
       ... 1 common frames omitted
INFO  [SharedPool-Worker-1] 2017-08-15 07:13:16,589 MigrationManager.java:464 - Drop table 'hawkular_alerts/alerts_statuses'
 
I think Column family ID mismatch is leading to Casssandra to an infinite loop of drop/re-create table process, due to pod hawkular-metrics being re-created as it ends in failure each time it tries to finalize it’s configuration.
 
INFO  [MigrationStage:1] 2017-08-15 12:19:49,443 ColumnFamilyStore.java:389 - Initializing hawkular_alerts.alerts_statuses
INFO  [SharedPool-Worker-1] 2017-08-15 12:19:49,785 MigrationManager.java:464 - Drop table 'hawkular_alerts/alerts_statuses'
INFO  [SharedPool-Worker-2] 2017-08-15 12:20:02,447 MigrationManager.java:342 - Create new table: org.apache.cassandra.config.CFMetaData@b43859a[cfId=103a92e0
 
This is why we have pod hawkular-cassandra stable, online but pod hawkular-metrics re-spawning
 
[root@rsomtapae237 origin]# oc get pod
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-2v93j   1/1       Running   0          17h
hawkular-metrics-jnhr7       0/1       Running   111        17h
hawkular-metrics-k3fqq       0/1       Running   103        16h
heapster-lq047               0/1       Running   110        17h

Comment 1 John Sanda 2017-08-16 19:04:39 UTC

Are there multiple cassandra pods? I assume there are because this exception indicates a schema disagreement between cassandra instances. This could have happened as a result of two hawkular-metrics pods starting up at the same time and trying to apply schema updates concurrently. To resolve this I would do the following:

1) Scale hawkular-metrics down to zero. 

2) oc exec <cassandra_pod> -- cqlsh --ssl -e "drop keyspace hawkular_alerts"

3) Scale hawkular-metrics back up to one

4) Wailt until hawkular-metrics is in the ready state, and then scale it back to two pods

The hawkular_alerts keyspace can be dropped because it is not used by hawkular-metrics. It will get rebuilt though at step 3.

Comment 2 Sandeep Dhar 2017-10-03 13:19:23 UTC

No, multiple cassandra pods were not running. We also tried to scale down to 0 and then back to 1, but the issue persisted.

Comment 3 Matt Wringe 2017-10-05 19:24:25 UTC


*** This bug has been marked as a duplicate of bug 1492159 ***

Note You need to log in before you can comment on or make changes to this bug.