Bug 1613095 - [3.9]hawkular-metrics pod failed to start up due to unsuccessful version check
Summary: [3.9]hawkular-metrics pod failed to start up due to unsuccessful version check
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.z
Assignee: Ruben Vargas Palma
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1612813 1619497
Blocks: 1599241 1610733
TreeView+ depends on / blocked
 
Reported: 2018-08-07 01:07 UTC by Junqi Zhao
Modified: 2019-11-20 06:23 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1612648
Environment:
Last Closed: 2018-12-13 19:27:05 UTC
Target Upstream Version:


Attachments (Terms of Use)
metrics logs - metrics-hawkular-metrics:v3.9.43-4 (19.32 KB, application/x-gzip)
2018-09-19 05:37 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3748 0 None None None 2018-12-13 19:27:21 UTC

Comment 1 Junqi Zhao 2018-08-07 01:11:52 UTC
same issue with v3.9 metrics images
metrics-cassandra-v3.9.40-2
metrics-hawkular-metrics-v3.9.40-2
metrics-heapster-v3.9.40-2

# oc get pod -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-hrwmb   1/1       Running   0          14m
hawkular-metrics-djc9v       0/1       Running   1          14m
heapster-c58zj               0/1       Running   1          14m


logs in hawkular-metrics pods
******************************************************************
2018-08-07 01:05:29,427 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 01:05:29,428 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 01:05:39,433 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 01:05:39,434 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 01:05:49,443 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 01:05:49,444 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 01:05:59,448 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 01:05:59,448 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 01:06:09,449 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2018-08-07 01:06:35,949 INFO  [org.jboss.as.server] (Thread-8) WFLYSRV0220: Server shutdown has been requested via an OS signal
*** JBossAS process (376) received TERM signal ***
2018-08-07 01:06:36,060 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 16) WFLYUT0022: Unregistered web context: '/hawkular/metrics' from server 'default-server'
2018-08-07 01:06:36,189 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0019: Host default-host stopping
2018-08-07 01:06:36,188 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-5) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:97)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:90)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:127)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:191)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:139)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:153)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more

2018-08-07 01:06:36,198 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 226ms
2018-08-07 01:06:36,254 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 69) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2018-08-07 01:06:36,331 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2018-08-07 01:06:36,352 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0019: Stopped Driver service with driver-name = h2
2018-08-07 01:06:36,369 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0008: Undertow HTTP listener default suspending
2018-08-07 01:06:36,377 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0008: Undertow HTTPS listener https suspending
2018-08-07 01:06:36,389 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:8080
2018-08-07 01:06:36,390 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 0.0.0.0:8443
2018-08-07 01:06:36,391 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0004: Undertow 1.4.18.SP5-redhat-1 stopping
2018-08-07 01:06:36,450 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment hawkular-metrics.war (runtime-name: hawkular-metrics.war) in 478ms
2018-08-07 01:06:36,460 INFO  [org.jboss.as] (MSC service thread 1-6) WFLYSRV0050: JBoss EAP 7.1.2.GA (WildFly Core 3.0.13.Final-redhat-1) stopped in 469ms
*** JBossAS process (376) received TERM signal ***
*************************************************************************

Keyspace hawkular_metrics does not exist
# oc exec hawkular-cassandra-1-hrwmb -n openshift-infra -- cqlsh --ssl -e "select table_name from system_schema.tables where keyspace_name = 'hawkular_metrics'"

 table_name
------------

(0 rows)

Comment 2 Junqi Zhao 2018-08-07 05:54:51 UTC
Tested with, 
metrics-cassandra-v3.9.40-10
metrics-hawkular-metrics-v3.9.40-10
metrics-heapster-v3.9.40-10

it is still "Version check unsuccessful after 30 attempts"
# oc get po -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-sxc22   1/1       Running   0          13m
hawkular-metrics-5cj5q       0/1       Running   4          12m
heapster-nwmsb               0/1       Running   1          12m



logs in hawkular-metrics pods
******************************************************************
2018-08-07 05:49:15,204 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:49:15,205 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:49:25,214 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:49:25,215 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:49:35,230 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:49:35,231 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:49:45,239 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:49:45,240 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:49:55,270 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:49:55,271 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:50:05,280 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: Keyspace hawkular_metrics does not exist
2018-08-07 05:50:05,280 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-07 05:50:15,282 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2018-08-07 05:50:41,434 INFO  [org.jboss.as.server] (Thread-8) WFLYSRV0220: Server shutdown has been requested via an OS signal
*** JBossAS process (386) received TERM signal ***
2018-08-07 05:50:41,664 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 62) WFLYUT0022: Unregistered web context: '/hawkular/metrics' from server 'default-server'
2018-08-07 05:50:41,858 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
2018-08-07 05:50:41,863 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-2) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:97)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:90)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:127)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:191)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:139)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:153)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more

2018-08-07 05:50:41,977 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2018-08-07 05:50:42,072 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 573ms
2018-08-07 05:50:42,101 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2018-08-07 05:50:42,147 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0019: Stopped Driver service with driver-name = h2
2018-08-07 05:50:42,203 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTPS listener https suspending
2018-08-07 05:50:42,208 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow HTTP listener default suspending
2018-08-07 05:50:42,210 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:8080
2018-08-07 05:50:42,210 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 0.0.0.0:8443
2018-08-07 05:50:42,222 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0004: Undertow 1.4.18.SP5-redhat-1 stopping
2018-08-07 05:50:42,402 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment hawkular-metrics.war (runtime-name: hawkular-metrics.war) in 923ms
2018-08-07 05:50:42,420 INFO  [org.jboss.as] (MSC service thread 1-5) WFLYSRV0050: JBoss EAP 7.1.2.GA (WildFly Core 3.0.13.Final-redhat-1) stopped in 939ms
*** JBossAS process (386) received TERM signal ***

******************************************************************

Keyspace hawkular_metrics does not exist
# oc exec hawkular-cassandra-1-sxc22 -n openshift-infra -- cqlsh --ssl -e "select table_name from system_schema.tables where keyspace_name = 'hawkular_metrics'"

 table_name
------------

(0 rows)

Comment 3 Junqi Zhao 2018-08-08 05:59:52 UTC
tested with
metrics-cassandra-v3.9.40-11
metrics-hawkular-metrics-v3.9.40-11
metrics-heapster-v3.9.40-11
metrics-hawkular-openshift-agent-v3.9.40-11

Issue is fixed, and metrics works well

Comment 4 Junqi Zhao 2018-08-08 06:06:03 UTC
New regression issue: Bug 1613656, it is not test blocker

Comment 5 Junqi Zhao 2018-08-22 08:22:35 UTC
add TestBlocker back,issue is reproduced with 
metrics-hawkular-metrics-v3.9.41-1
metrics-cassandra-v3.9.41-1
metrics-heapster-v3.9.41-1

# oc logs -f hawkular-metrics-w956g
2018-08-22 08:20:35,859 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-22 08:20:45,860 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:73)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

*** JBossAS process (375) received TERM signal ***
2018-08-22 08:21:12,623 INFO  [org.jboss.as.server] (Thread-8) WFLYSRV0220: Server shutdown has been requested via an OS signal
2018-08-22 08:21:12,684 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0022: Unregistered web context: '/hawkular/metrics' from server 'default-server'
2018-08-22 08:21:12,737 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
2018-08-22 08:21:12,743 ERROR [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (MSC service thread 1-8) HAWKMETRICS200009: Unexcepted exception while shutting down: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
	at com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1464)
	at com.google.common.util.concurrent.Futures.get(Futures.java:1453)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.destroy(MetricsServiceLifecycle.java:670)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:97)
	at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.preDestroy(DefaultLifecycleCallbackInvoker.java:90)
	at org.jboss.weld.injection.producer.BasicInjectionTarget.preDestroy(BasicInjectionTarget.java:127)
	at org.jboss.weld.bean.ManagedBean.destroy(ManagedBean.java:191)
	at org.jboss.weld.util.bean.IsolatedForwardingBean.destroy(IsolatedForwardingBean.java:50)
	at org.jboss.weld.context.AbstractContext.destroyContextualInstance(AbstractContext.java:139)
	at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:153)
	at org.jboss.weld.context.AbstractSharedContext.destroy(AbstractSharedContext.java:61)
	at org.jboss.weld.context.AbstractSharedContext.invalidate(AbstractSharedContext.java:56)
	at org.jboss.weld.bootstrap.WeldRuntime.shutdown(WeldRuntime.java:56)
	at org.jboss.weld.bootstrap.WeldBootstrap.shutdown(WeldBootstrap.java:113)
	at org.jboss.as.weld.WeldStartService.stop(WeldStartService.java:128)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.stopServices(MetricsServiceLifecycle.java:682)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	... 3 more

2018-08-22 08:21:12,762 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2018-08-22 08:21:12,831 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2018-08-22 08:21:12,836 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0019: Stopped Driver service with driver-name = h2
2018-08-22 08:21:12,844 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment activemq-rar.rar (runtime-name: activemq-rar.rar) in 204ms
2018-08-22 08:21:12,861 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0008: Undertow HTTPS listener https suspending
2018-08-22 08:21:12,861 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTP listener default suspending
2018-08-22 08:21:12,862 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 0.0.0.0:8443
2018-08-22 08:21:12,871 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:8080
2018-08-22 08:21:12,871 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0004: Undertow 1.4.18.SP5-redhat-1 stopping
2018-08-22 08:21:12,908 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0028: Stopped deployment hawkular-metrics.war (runtime-name: hawkular-metrics.war) in 271ms
2018-08-22 08:21:12,917 INFO  [org.jboss.as] (MSC service thread 1-8) WFLYSRV0050: JBoss EAP 7.1.2.GA (WildFly Core 3.0.13.Final-redhat-1) stopped in 279ms
*** JBossAS process (375) received TERM signal ***

Comment 7 Junqi Zhao 2018-09-14 05:47:37 UTC
Issue is not fixed with images:
metrics-cassandra:v3.9.43-2
metrics-hawkular-metrics:v3.9.43-2
metrics-heapster:v3.9.43-2

Comment 8 John Sanda 2018-09-17 19:19:38 UTC
The latest 3.9 image is 3.9.44 and the problem still exist because that image uses Hawkular Metrics 0.29.2.

The schema installer was introduced in OCP 3.10. We wanted to back port it to 3.9. The changes were introduced upstream in Hawkular Metrics 0.29.2. Because og the ongoing problems we have had with trying to back port, we are going to update the 3.9 image to use Hawkular Metrics 0.29.1.

Comment 10 Junqi Zhao 2018-09-18 04:10:09 UTC
HAWKULAR_METRICS_VERSION=0.29.2.Final-redhat-1 in metrics-hawkular-metrics-v3.9.43-3

Comment 12 Junqi Zhao 2018-09-19 05:36:55 UTC
Tested with metrics-hawkular-metrics-v3.9.43-4, hawkular-metrics pods failed to start up, there is error in pod logs.

other images
metrics-cassandra-v3.9.43-3
metrics-heapster-v3.9.43-3

# oc get pod -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-qchg4   1/1       Running   0          17m
hawkular-metrics-rs4gr       0/1       Running   1          17m
heapster-g2n62               0/1       Running   2          17m


2018-09-19 05:12:29,290 INFO  [org.jboss.weld.Version] (MSC service thread 1-7) WELD-000900: 2.4.7 (redhat)
2018-09-19 05:12:29,587 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "hawkular-metrics.war")]) - failure description: {
    "WFLYCTL0412: Required services that are not installed:" => [
        "jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics",
        "jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks"
    ],
    "WFLYCTL0180: Services with missing/unavailable dependencies" => [
        "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.cache.locks is missing [jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks]",
        "jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.container.hawkular-metrics is missing [jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics]"
    ]
}
2018-09-19 05:12:29,611 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "hawkular-metrics.war" (runtime-name : "hawkular-metrics.war")
2018-09-19 05:12:29,612 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "activemq-rar.rar" (runtime-name : "activemq-rar.rar")
2018-09-19 05:12:29,616 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0184:    New missing/unsatisfied dependencies:
      service jboss.naming.context.java.jboss.infinispan.cache.hawkular-metrics.locks (missing) dependents: [service jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.cache.locks] 
      service jboss.naming.context.java.jboss.infinispan.container.hawkular-metrics (missing) dependents: [service jboss.naming.context.java.module.hawkular-metrics.hawkular-metrics.env.container.hawkular-metrics] 

2018-09-19 05:12:29,763 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2018-09-19 05:12:29,766 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
2018-09-19 05:12:29,766 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
2018-09-19 05:12:29,766 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: JBoss EAP 7.1.2.GA (WildFly Core 3.0.13.Final-redhat-1) started (with errors) in 10478ms - Started 422 of 689 services (22 services failed or missing dependencies, 372 services are lazy, passive or on-demand)

Comment 13 Junqi Zhao 2018-09-19 05:37:43 UTC
Created attachment 1484573 [details]
metrics logs - metrics-hawkular-metrics:v3.9.43-4

Comment 14 John Sanda 2018-09-19 14:05:16 UTC
Ruben,

I bet I know what the problem is. I think we left out another, necessary change in addition to reverting back to hawkular-metrics 0.29.1. We need to add back the infinispan configurtation in standalone.xml in the image.

Comment 15 Junqi Zhao 2018-09-20 02:10:19 UTC
fixed with metrics-hawkular-metrics:v3.9.43-5

other images
metrics-cassandra-v3.9.43-3
metrics-heapster-v3.9.43-3

# oc get pod -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-4k9hl   1/1       Running   0          32m
hawkular-metrics-55mrs       1/1       Running   0          32m
heapster-6zvzt               1/1       Running   0          32m

Comment 16 Junqi Zhao 2018-09-21 11:30:23 UTC
please change to ON_QA, so we can close it.

Comment 17 Junqi Zhao 2018-09-25 00:18:48 UTC
Per Comment 15, set it to VERIFIED.

Comment 23 errata-xmlrpc 2018-12-13 19:27:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3748


Note You need to log in before you can comment on or make changes to this bug.