Description of problem: After running the playbooks to upgrade aggregated-logging it causes several errors due to the lack of synchronization between configurations. Version-Release number of selected component (if applicable): openshift-ansible-3.6.173.0.48-1.git.0.1609d30.el7.noarch logging-curator-v3.6.173.0.21-17 logging-elasticsearch-v3.6.173.0.5-5 logging-fluentd-v3.6.173.0.21-17 logging-kibana-v3.6.173.0.21-22 logging-auth-proxy-v3.6.173.0.21-17 How reproducible: Always Steps to Reproduce: 1. yum update on masters 2. ansible-playbook openshift-logging.yml Actual results: Error 1. Elasticsearch pod is unable to start [2017-10-24 11:53:27,760][INFO ][container.run ] Checking if Elasticsearch is ready on https://localhost:9200 Exception in thread "main" java.lang.IllegalArgumentException: Unknown Discovery type [kubernetes] at org.elasticsearch.discovery.DiscoveryModule.configure(DiscoveryModule.java:100) at <<<guice>>> at org.elasticsearch.node.Node.<init>(Node.java:213) at org.elasticsearch.node.Node.<init>(Node.java:140) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:45) This is caused by the livenessProbes configuration in the configMap cloud: kubernetes: pod_label: ${POD_LABEL} pod_port: 9300 namespace: ${NAMESPACE} Still needs to be this: cloud: kubernetes: service: ${SERVICE_DNS} namespace: ${NAMESPACE} After changing it the Elasticsearch pod starts Error 2. Fluentd does not send any log to elasticsearch. This is caused by the addition of the <label @OUTPUT> to the configmap but fluentd images still don't add this label. The following errors are appreciated: 2017-10-24 07:57:51 -0400 [warn]: no patterns matched tag="journal.system" The logging-fluentd configmap looks like this: @include configs.d/openshift/filter-post-*.conf ## </label> <label @OUTPUT> ## matches @include configs.d/openshift/output-pre-*.conf Should still be: @include configs.d/openshift/filter-post-*.conf ## ## matches @include configs.d/openshift/output-pre-*.conf After removing the </label> and <label @OUTPUT> it works again. Expected results: Elasticsearch pods to be deployed successfully Fluentd pods to forward logs to elasticsearch Additional info:
The ES image with readiness probe has been waiting for a release for more than a month and a half. It is not out of sync config between ansible and image, repositories have been in sync since it merged, we only need the image to be rebuilt and pushed to the registries. For more information, take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1503563
*** This bug has been marked as a duplicate of bug 1503563 ***