Bug 1632852

Summary:	The schema installer job may not complete successfully
Product:	OpenShift Container Platform	Reporter:	John Sanda <jsanda>
Component:	Hawkular	Assignee:	Ruben Vargas Palma <rvargasp>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Junqi Zhao <juzhao>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.11.0	CC:	aos-bugs
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1632853 (view as bug list)		Environment:
Last Closed:	2021-08-06 01:41:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1632853

Description John Sanda 2018-09-25 17:06:26 UTC

Description of problem:
The schema installer job was introduced in 3.10. It applies Hawkular Metrics schema changes to Cassandra. This was previously done by hawkular-metrics at start up prior to 3.10. 

The hawkular-metrics pod(s) cannot start until the schema installer job finishes successfully. The job updates a row in the sys_config table to indicate that the schema is up to date. Hawkular Metrics polls Cassandra, checking that row to determine if the schema is up to date.

The job is configured with a restart policy of OnFailure which made me think that the job would be continually run until it completed successfully. https://github.com/kubernetes/kubernetes/issues/54870 explains that the backoffLimit will cap the number of retries for a job. The schema installer does not set the backoffLimit, so it gets the default value of 6.

This has led to situations where hawkular-metrics cannot start up because the job has failed after six attempts. This has happened for example when the Cassandra cluster is in a bad state and requires some manual intervention to get it healthy. 

We could increase the backoffLimit, but that does not really address the problem. The installer has to be updated with additional retry logic so that it keeps running until changes are applied successfully.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: