Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1610230

Summary:

Exception in hawkular-cassandra pod logs

Product:

OpenShift Container Platform

Reporter:

Junqi Zhao <juzhao>

Component:

Hawkular

Assignee:

Ruben Vargas Palma <rvargasp>

Status:

CLOSED NOTABUG

QA Contact:

Junqi Zhao <juzhao>

Severity:

low

Docs Contact:

Priority:

low

Version:

3.11.0

CC:

aos-bugs

Target Milestone:

---

Target Release:

3.11.z

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2021-08-24 05:30:12 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
metrics logs	none

Description Junqi Zhao 2018-07-31 09:18:56 UTC

Created attachment 1471743 [details]
metrics logs

Description of problem:
Deployed metrics with openshift_metrics_cassandra_replicas=2, hawkular-cassandra-2  pod restarted 1 time, and has exception in pod logs, more info see the attached logs file. It does not affect the function, metrics works well
# oc get pod -n openshift-infra
NAME                            READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-7w2sf      1/1       Running     0          30m
hawkular-cassandra-2-9mtrd      1/1       Running     1          30m
hawkular-metrics-hbll5          1/1       Running     0          30m
hawkular-metrics-schema-5qzj8   0/1       Completed   0          32m
heapster-dtjpf                  1/1       Running     0          30m

ERROR [MigrationStage:1] 2018-07-31 08:43:38,057 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
java.lang.AssertionError: null
	at org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:371) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.<init>(CFMetaData.java:324) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:428) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:1000) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:910) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1325) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1304) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
ERROR [MigrationStage:1] 2018-07-31 08:43:38,145 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
java.lang.AssertionError: null
	at org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:371) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.<init>(CFMetaData.java:324) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:428) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:1000) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:910) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1325) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1304) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
ERROR [MigrationStage:1] 2018-07-31 08:43:38,233 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.InvalidRequestException: Unknown type hawkular_metrics.trigger_def
	at org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1059) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_181]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:910) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1325) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1304) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
INFO  [GossipStage:1] 2018-07-31 08:43:38,276 Gossiper.java:1042 - Node /10.2.2.9 is now part of the cluster
INFO  [SharedPool-Worker-1] 2018-07-31 08:43:38,282 Gossiper.java:1006 - InetAddress /10.2.2.9 is now UP
INFO  [GossipStage:1] 2018-07-31 08:43:38,307 TokenMetadata.java:479 - Updating topology for /10.2.2.9
INFO  [GossipStage:1] 2018-07-31 08:43:38,308 TokenMetadata.java:479 - Updating topology for /10.2.2.9
ERROR [MigrationStage:1] 2018-07-31 08:43:38,334 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.InvalidRequestException: Unknown type hawkular_metrics.trigger_def
	at org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1059) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_181]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:910) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1325) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1304) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
ERROR [MigrationStage:1] 2018-07-31 08:43:38,445 CassandraDaemon.java:207 - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.InvalidRequestException: Unknown type hawkular_metrics.trigger_def
	at org.apache.cassandra.cql3.CQL3Type$Raw$RawUT.prepare(CQL3Type.java:745) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.cql3.CQL3Type$Raw.prepareInternal(CQL3Type.java:533) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1059) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_181]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1045) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:995) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:959) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:918) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:910) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1325) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1304) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]

Version-Release number of selected component (if applicable):
v3.11.0-0.10.0.0

How reproducible:
Sometimes

Steps to Reproduce:
1. Deploy metrics 311, parameters see [Additional info] part
2.
3.

Actual results:


Expected results:


Additional info:
openshift_metrics_install_metrics=true
openshift_metrics_cassandra_storage_type=dynamic
openshift_metrics_cassandra_limits_memory=3G
openshift_metrics_hawkular_requests_cpu=200m
openshift_metrics_cassandra_replicas=2