Bug 1520377

Summary:	Logging upgrade fails from 3.6 to 3.7
Product:	OpenShift Container Platform	Reporter:	Jaspreet Kaur <jkaur>
Component:	Logging	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	3.7.0	CC:	aos-bugs, ewolinet, hgomes, jcantril, jkaur, jwozniak, rmeggins
Target Milestone:	---
Target Release:	3.7.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-05 09:33:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jaspreet Kaur 2017-12-04 11:30:37 UTC

Description of problem:  After upgrading to OCP 3.7 I run the openshift-logging.yml   playbook. It finished without errors but the elasticsearch replication controllers fail to start. Somehow the readiness probe fails with

"Readiness probe failed: Incorrect SG document count, expected 5 [received doc count: ] Traceback (most recent call last): File "<string>", line 1, in <module> KeyError: 'hits"

Elastic search logs show below errors :


[2017-12-04 10:40:11,411][INFO ][container.run            ] Begin Elasticsearch startup script
[2017-12-04 10:40:11,437][INFO ][container.run            ] Comparing the specified RAM to the maximum recommended for Elasticsearch...
[2017-12-04 10:40:11,438][INFO ][container.run            ] Inspecting the maximum RAM available...
[2017-12-04 10:40:11,444][INFO ][container.run            ] ES_HEAP_SIZE: '4096m'
[2017-12-04 10:40:11,445][INFO ][container.run            ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof
[2017-12-04 10:40:11,450][INFO ][container.run            ] Checking if Elasticsearch is ready on https://localhost:9200
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'POD_LABEL'
	at org.elasticsearch.common.property.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:128)
	at org.elasticsearch.common.property.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:81)
	at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1179)
	at org.elasticsearch.node.internal.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:131)
	at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
	at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
	at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:242)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45)
Refer to the log for complete error details.



Version-Release number of selected component (if applicable):


Actual results: elastic search pod fails to start


Expected results: It should suceed without errors


Additional info: seems because of the upgrade issue mentioned here :
https://bugzilla.redhat.com/show_bug.cgi?id=1503563#c19

Comment 1 ewolinet 2017-12-04 19:33:04 UTC

Can you provide the contents of the logging-es configmap, the yaml output of one of the ES dc, and the version of the openshift-ansible rpm used for doing this install?

Comment 7 Jeff Cantrill 2017-12-08 14:31:19 UTC

Issue from c#6 will be resolved by: https://github.com/openshift/origin-aggregated-logging/pull/812

Comment 8 Anping Li 2018-03-29 01:51:24 UTC

The ose-ansible:3.7.40 works.

Comment 14 errata-xmlrpc 2018-04-05 09:33:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636