Red Hat Bugzilla – Bug 1520377
Logging upgrade fails from 3.6 to 3.7
Last modified: 2018-04-05 05:33:53 EDT
Description of problem: After upgrading to OCP 3.7 I run the openshift-logging.yml playbook. It finished without errors but the elasticsearch replication controllers fail to start. Somehow the readiness probe fails with "Readiness probe failed: Incorrect SG document count, expected 5 [received doc count: ] Traceback (most recent call last): File "<string>", line 1, in <module> KeyError: 'hits" Elastic search logs show below errors : [2017-12-04 10:40:11,411][INFO ][container.run ] Begin Elasticsearch startup script [2017-12-04 10:40:11,437][INFO ][container.run ] Comparing the specified RAM to the maximum recommended for Elasticsearch... [2017-12-04 10:40:11,438][INFO ][container.run ] Inspecting the maximum RAM available... [2017-12-04 10:40:11,444][INFO ][container.run ] ES_HEAP_SIZE: '4096m' [2017-12-04 10:40:11,445][INFO ][container.run ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof [2017-12-04 10:40:11,450][INFO ][container.run ] Checking if Elasticsearch is ready on https://localhost:9200 Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'POD_LABEL' at org.elasticsearch.common.property.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:128) at org.elasticsearch.common.property.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:81) at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1179) at org.elasticsearch.node.internal.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:131) at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100) at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107) at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100) at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:242) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45) Refer to the log for complete error details. Version-Release number of selected component (if applicable): Actual results: elastic search pod fails to start Expected results: It should suceed without errors Additional info: seems because of the upgrade issue mentioned here : https://bugzilla.redhat.com/show_bug.cgi?id=1503563#c19
Can you provide the contents of the logging-es configmap, the yaml output of one of the ES dc, and the version of the openshift-ansible rpm used for doing this install?
Issue from c#6 will be resolved by: https://github.com/openshift/origin-aggregated-logging/pull/812
The ose-ansible:3.7.40 works.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0636