Description of problem:
The schema installer job was introduced in 3.10. It applies Hawkular Metrics schema changes to Cassandra. This was previously done by hawkular-metrics at start up prior to 3.10. 

The hawkular-metrics pod(s) cannot start until the schema installer job finishes successfully. The job updates a row in the sys_config table to indicate that the schema is up to date. Hawkular Metrics polls Cassandra, checking that row to determine if the schema is up to date.

The job is configured with a restart policy of OnFailure which made me think that the job would be continually run until it completed successfully. https://github.com/kubernetes/kubernetes/issues/54870 explains that the backoffLimit will cap the number of retries for a job. The schema installer does not set the backoffLimit, so it gets the default value of 6.

This has led to situations where hawkular-metrics cannot start up because the job has failed after six attempts. This has happened for example when the Cassandra cluster is in a bad state and requires some manual intervention to get it healthy. 

We could increase the backoffLimit, but that does not really address the problem. The installer has to be updated with additional retry logic so that it keeps running until changes are applied successfully.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: