Bug 1619497

Summary: [3.11]hawkular-metrics pod failed to start up due to unsuccessful version check
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: HawkularAssignee: John Sanda <jsanda>
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: ahaile, aos-bugs, dhananjay.lambe, jsanda, juzhao, ricardo.arguello, rvargasp, ssadhale, wsun
Target Milestone: ---Keywords: Regression, Reopened, TestBlocker
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1611941 Environment:
Last Closed: 2019-11-21 18:38:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1611941    
Bug Blocks: 1612648, 1612813, 1613095    

Comment 1 Junqi Zhao 2018-08-21 04:17:13 UTC
metrics-cassandra-v3.11.0-0.19.0.0
metrics-hawkular-metrics-v3.11.0-0.19.0.0
metrics-heapster-v3.11.0-0.19.0.0

# oc get pod -n openshift-infra
NAME                         READY     STATUS             RESTARTS   AGE
hawkular-cassandra-1-cm6wx   1/1       Running            0          1h
hawkular-metrics-2kkln       0/1       Running            0          1h
heapster-dgrrf               0/1       Running            9          1h

# oc logs hawkular-metrics-2kkln -n openshift-infra
2018-08-21 03:09:19,914 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-21 03:09:29,920 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: unconfigured table sys_config
2018-08-21 03:09:29,920 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-21 03:09:39,926 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Version check failed: unconfigured table sys_config
2018-08-21 03:09:39,927 INFO  [org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker] (metricsservice-lifecycle-thread) Trying again in 10000 ms
2018-08-21 03:09:49,927 FATAL [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) The schema version check failed. Start up cannot proceed.: org.hawkular.metrics.api.jaxrs.util.SchemaVersionCheckException: Version check unsuccessful after 30 attempts
	at org.hawkular.metrics.api.jaxrs.util.SchemaVersionChecker.waitForSchemaUpdates(SchemaVersionChecker.java:72)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.doSchemaVersionCheck(MetricsServiceLifecycle.java:531)
	at org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle.startMetricsService(MetricsServiceLifecycle.java:368)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Comment 7 dhananajay lambe 2019-10-21 09:46:46 UTC
hi, 
i am also facing same issue please update