Bug 1612648 - [3.7]hawkular-metrics pod failed to start up due to unsuccessful version check
Summary: [3.7]hawkular-metrics pod failed to start up due to unsuccessful version check
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Ruben Vargas Palma
QA Contact: Junqi Zhao
URL:
Whiteboard:
: 1612813 (view as bug list)
Depends On: 1611941 1612813 1619497
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-06 03:42 UTC by Junqi Zhao
Modified: 2019-11-21 18:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1611941
: 1613095 (view as bug list)
Environment:
Last Closed: 2019-11-21 18:37:52 UTC
Target Upstream Version:


Attachments (Terms of Use)

Comment 1 Junqi Zhao 2018-08-06 03:47:23 UTC
images:
metrics-cassandra-v3.7.61-2
metrics-hawkular-metrics-v3.7.61-2
metrics-heapster-v3.7.61-2

# oc get pod
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-cfn4w   1/1       Running   0          30m
hawkular-metrics-kzvkc       0/1       Running   4          30m
heapster-nh9dq               0/1       Running   3          30m


logs in hawkular-metrics pod, Version check unsuccessful after 30 attempts
**************************************************************************
2018-08-06 03:21:19,616 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-06 03:21:19,617 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms

2018-08-06 03:22:39,693 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-06 03:22:39,693 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-06 03:22:49,694 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-08-06 03:23:15,486 INFO  [org.jboss.as.server] (Thread-5) WFLYSRV0220: Server shutdown has been requested via an OS signal
*** JBossAS process (316) received TERM signal ***
2018-08-06 03:23:15,584 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0022: Unregistered web context: /hawkular/metrics
2018-08-06 03:23:15,665 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0019: Host default-host stopping
2018-08-06 03:23:15,665 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 165ms
2018-08-06 03:23:15,668 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0010: Stopping weld service for deployment hawkular-metrics.war
2018-08-06 03:23:15,672 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-2) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:91)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:131)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:189)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:147)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:161)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more

Checked, unlike 3.6, 3.7 Keyspace hawkular_metrics does not exist
# oc rsh hawkular-cassandra-1-cfn4w
sh-4.2$  cqlsh --ssl -e "select table_name from system_schema.tables where keyspace_name = 'hawkular_metrics'"

 table_name
------------

(0 rows)

Comment 2 John Sanda 2018-08-06 18:33:36 UTC
*** Bug 1612813 has been marked as a duplicate of this bug. ***

Comment 3 Junqi Zhao 2018-08-07 05:35:34 UTC
Tested with, 
metrics-cassandra-v3.7.61-10
metrics-hawkular-metrics-v3.7.61-10
metrics-heapster-v3.7.61-10

it is still "Version check unsuccessful after 30 attempts"
# oc get po -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-5lbzv   1/1       Running   0          23m
hawkular-metrics-wp7wz       0/1       Running   3          23m
heapster-wkbl7               0/1       Running   2          23m

logs in hawkular-metrics pods
******************************************************************
2018-08-07 05:17:00,807 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:17:00,808 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:17:10,812 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:17:10,813 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:17:20,818 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:17:20,818 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:17:30,819 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
*** JBossAS process (340) received TERM signal ***
2018-08-07 05:17:58,938 INFO  [org.jboss.as.server] (Thread-5) WFLYSRV0220: Server shutdown has been requested via an OS signal
2018-08-07 05:17:59,011 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 62) WFLYUT0022: Unregistered web context: /hawkular/metrics
2018-08-07 05:17:59,042 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0019: Host default-host stopping
2018-08-07 05:17:59,047 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0010: Stopping weld service for deployment hawkular-metrics.war
2018-08-07 05:17:59,049 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-2) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:91)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:131)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:189)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:147)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:161)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more
2018-08-07 05:17:59,084 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 126ms
2018-08-07 05:17:59,096 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2018-08-07 05:17:59,114 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0008: Undertow HTTP listener default suspending
2018-08-07 05:17:59,114 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTPS listener https suspending
2018-08-07 05:17:59,114 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:8080
2018-08-07 05:17:59,114 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 0.0.0.0:8443
2018-08-07 05:17:59,115 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.3.28.Final-redhat-4 stopping
2018-08-07 05:17:59,110 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
2018-08-07 05:17:59,164 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment hawkular-metrics.war (runtime-name: hawkular-metrics.war) in 197ms
2018-08-07 05:17:59,168 INFO  [org.jboss.as] (MSC service thread 1-5) WFLYSRV0050: JBoss EAP 7.0.6.GA (WildFly Core 2.1.15.Final-redhat-1) stopped in 191ms
**************************************************************
# oc exec hawkular-cassandra-1-5lbzv -n openshift-infra -- cqlsh --ssl -e "select table_name from system_schema.tables where keyspace_name = 'hawkular_metrics'"


 table_name
------------

(0 rows)

Comment 4 Junqi Zhao 2018-08-08 05:59:22 UTC
tested with
metrics-cassandra-v3.7.61-11
metrics-hawkular-metrics-v3.7.61-11
metrics-heapster-v3.7.61-11
metrics-hawkular-openshift-agent-v3.7.61-11

Issue is fixed, and metrics works well

Comment 5 Junqi Zhao 2018-08-22 10:50:23 UTC
add TestBlocker back,issue is reproduced with
metrics-cassandra-v3.7.62-2
metrics-hawkular-metrics-v3.7.62-2
metrics-heapster-v3.7.62-2

# oc get pod -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-nzrmw   1/1       Running   0          9m
hawkular-metrics-k2wtz       0/1       Running   1          9m
heapster-7gxvk               0/1       Running   1          9m

# oc logs -f hawkular-metrics-k2wtz -n openshift-infra
2018-08-22 10:45:25,405 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-22 10:45:35,409 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-22 10:45:35,409 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-22 10:45:45,410 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-08-22 10:46:09,906 INFO  [org.jboss.as.server] (Thread-5) WFLYSRV0220: Server shutdown has been requested via an OS signal
*** JBossAS process (318) received TERM signal ***
2018-08-22 10:46:09,958 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 70) WFLYUT0022: Unregistered web context: /hawkular/metrics
2018-08-22 10:46:10,010 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0019: Host default-host stopping
2018-08-22 10:46:10,022 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 108ms
2018-08-22 10:46:10,024 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0010: Stopping weld service for deployment hawkular-metrics.war
2018-08-22 10:46:10,029 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-2) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:91)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:131)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:189)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:147)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:161)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more
2018-08-22 10:46:10,063 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2018-08-22 10:46:10,070 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0019: Stopped Driver service with driver-name = h2
2018-08-22 10:46:10,093 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0008: Undertow HTTPS listener https suspending
2018-08-22 10:46:10,094 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 0.0.0.0:8443
2018-08-22 10:46:10,094 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0008: Undertow HTTP listener default suspending
2018-08-22 10:46:10,094 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:8080
2018-08-22 10:46:10,095 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0004: Undertow 1.3.28.Final-redhat-4 stopping
2018-08-22 10:46:10,128 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment hawkular-metrics.war (runtime-name: hawkular-metrics.war) in 213ms
2018-08-22 10:46:10,135 INFO  [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: JBoss EAP 7.0.6.GA (WildFly Core 2.1.15.Final-redhat-1) stopped in 211ms

Comment 6 Junqi Zhao 2018-09-14 05:44:49 UTC
Issue is not fixed with images:
metrics-cassandra:v3.7.64-2
metrics-hawkular-metrics:v3.7.64-2
metrics-heapster:v3.7.64-2

Comment 7 John Sanda 2018-09-17 19:31:16 UTC
The latest 3.7 image is 3.7.64 and the problem still exist because that image uses Hawkular Metrics 0.28.5

The schema installer was introduced in OCP 3.10. We wanted to back port it to 3.7. The changes were introduced upstream in Hawkular Metrics 0.29.5. Because of the ongoing problems we have had with trying to back port, we are going to update the 3.7 image to use Hawkular Metrics 0.28.4.

Comment 9 Junqi Zhao 2018-09-20 02:11:19 UTC
fixed with metrics-hawkular-metrics-v3.7.64-5
other images:
metrics-cassandra-v3.7.64-3
metrics-heapster-v3.7.64-3

# oc get pod -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-f8mwb   1/1       Running   0          30m
hawkular-metrics-l9sl7       1/1       Running   0          30m
heapster-z4cn5               1/1       Running   0          30m

Comment 10 Junqi Zhao 2018-09-21 11:26:55 UTC
please change to ON_QA, so we can close it.

Comment 11 Junqi Zhao 2018-10-12 00:44:58 UTC
Per Comment 9, set it to VERIFIED


Note You need to log in before you can comment on or make changes to this bug.