Bug 1845947

This bug was initially created as a copy of Bug #1834576

I am copying this bug because: 



Description of problem:
The Fluentd couldn't connect to ES after the secret/master-certs regenerated. Looks like the Kibana and ES didn't use the new secrets, but the Fluentd was updated to use the new secrets. 

Logs in the Fluentd pod:
$ oc logs fluentd-x2g6h
2020-05-12 00:41:11 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-05-12 00:41:12 +0000 chunk="5a568aa5350d18f130acd5aa90c78a59" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): Connection refused - connect(2) for 172.30.106.176:9200 (Errno::ECONNREFUSED)"
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:962:in `rescue in send_bulk'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:924:in `send_bulk'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:758:in `block in write'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:757:in `each'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:757:in `write'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2020-05-12 00:41:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-05-12 00:41:44 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-05-12 00:41:45 +0000 chunk="5a568aa5350d18f130acd5aa90c78a59" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [503] Open Distro not initialized"
  2020-05-12 00:41:44 +0000 [warn]: suppressed same stacktrace
2020-05-12 00:41:46 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=2 next_retry_seconds=2020-05-12 00:41:48 +0000 chunk="5a568aa5350d18f130acd5aa90c78a59" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [503] Open Distro not initialized"
  2020-05-12 00:41:46 +0000 [warn]: suppressed same stacktrace
2020-05-12 00:41:48 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=3 next_retry_seconds=2020-05-12 00:41:52 +0000 chunk="5a568aa5350d18f130acd5aa90c78a59" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [503] Open Distro not initialized"
  2020-05-12 00:41:48 +0000 [warn]: suppressed same stacktrace
2020-05-12 00:41:52 +0000 [warn]: [clo_default_output_es] retry succeeded. chunk_id="5a568aa5350d18f130acd5aa90c78a59"
2020-05-12 00:42:12 +0000 [warn]: [clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=64.46370552199733 slow_flush_log_threshold=20.0 plugin_id="clo_default_output_es"
2020-05-12 00:43:25 +0000 [warn]: [retry_clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=64.77023854700019 slow_flush_log_threshold=20.0 plugin_id="retry_clo_default_output_es"
2020-05-12 00:43:25 +0000 [warn]: [clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=60.05987231400286 slow_flush_log_threshold=20.0 plugin_id="clo_default_output_es"
2020-05-12 00:43:29 +0000 [warn]: [clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=60.05105542599995 slow_flush_log_threshold=20.0 plugin_id="clo_default_output_es"
2020-05-12 00:43:56 +0000 [warn]: [clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=31.028722502000164 slow_flush_log_threshold=20.0 plugin_id="clo_default_output_es"
2020-05-12 00:43:56 +0000 [warn]: [retry_clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=98.55270952800129 slow_flush_log_threshold=20.0 plugin_id="retry_clo_default_output_es"
2020-05-12 00:43:57 +0000 [warn]: [clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=27.71531252299974 slow_flush_log_threshold=20.0 plugin_id="clo_default_output_es"
2020-05-12 00:43:57 +0000 [warn]: [retry_clo_default_output_es] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=32.02300949600249 slow_flush_log_threshold=20.0 plugin_id="retry_clo_default_output_es"
2020-05-12 01:03:11 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=0 next_retry_seconds=2020-05-12 01:03:12 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:962:in `rescue in send_bulk'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:924:in `send_bulk'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:758:in `block in write'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:757:in `each'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.0.5/lib/fluent/plugin/out_elasticsearch.rb:757:in `write'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2020-05-12 01:03:11 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.9.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-05-12 01:03:11 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=1 next_retry_seconds=2020-05-12 01:03:12 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:11 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:12 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=2 next_retry_seconds=2020-05-12 01:03:14 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:12 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:14 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=3 next_retry_seconds=2020-05-12 01:03:18 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:14 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:14 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-05-12 01:03:22 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:14 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:22 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=5 next_retry_seconds=2020-05-12 01:03:39 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:22 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:22 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=6 next_retry_seconds=2020-05-12 01:03:54 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:22 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:54 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=7 next_retry_seconds=2020-05-12 01:05:05 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:54 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:03:54 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=8 next_retry_seconds=2020-05-12 01:06:15 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:03:54 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:06:15 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=9 next_retry_seconds=2020-05-12 01:10:23 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:06:15 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:06:15 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=10 next_retry_seconds=2020-05-12 01:11:45 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:06:15 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:11:45 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=11 next_retry_seconds=2020-05-12 01:17:02 +0000 chunk="5a56906eb5d3a871b8d3443702693564" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:11:45 +0000 [warn]: suppressed same stacktrace
2020-05-12 01:11:50 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=12 next_retry_seconds=2020-05-12 01:16:33 +0000 chunk="5a56906dc2438626c68475e23c2fc181" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n            `Excon.defaults[:ssl_ca_path] = path_to_certs`\n            `ENV['SSL_CERT_DIR'] = path_to_certs`\n            `Excon.defaults[:ssl_ca_file] = path_to_file`\n            `ENV['SSL_CERT_FILE'] = path_to_file`\n            `Excon.defaults[:ssl_verify_callback] = callback`\n                (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n            `Excon.defaults[:ssl_verify_peer] = false` (less secure).\n"
  2020-05-12 01:11:50 +0000 [warn]: suppressed same stacktrace

EO log:
time="2020-05-12T01:02:36Z" level=info msg="Kibana status successfully updated"
time="2020-05-12T01:02:40Z" level=info msg="Timed out waiting for node elasticsearch-cdm-7copec07-1 to rollout"
time="2020-05-12T01:02:40Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-7copec07-1: timed out waiting for the condition"
time="2020-05-12T01:02:40Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:40Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
{"level":"error","ts":1589245360.3139858,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2020-05-12T01:02:41Z" level=info msg="Beginning full cluster restart for cert redeploy on elasticsearch"
time="2020-05-12T01:02:41Z" level=warning msg="Unable to disable shard allocation: Put https://elasticsearch.openshift-logging.svc:9200/_cluster/settings: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:41Z" level=warning msg="Unable to perform synchronized flush: Post https://elasticsearch.openshift-logging.svc:9200/_flush/synced: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:41Z" level=warning msg="Unable to enable shard allocation: Put https://elasticsearch.openshift-logging.svc:9200/_cluster/settings: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:41Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:41Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
{"level":"error","ts":1589245361.7059393,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2020-05-12T01:02:42Z" level=info msg="Waiting for cluster to complete recovery:  / green"
time="2020-05-12T01:02:43Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"
time="2020-05-12T01:02:43Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"openshift-cluster-logging-signer\")"


Version-Release number of selected component (if applicable):
cluster version: 4.5.0-0.nightly-2020-05-10-180138
logging images are from 4.5.0-0.ci-2020-05-11-212141
manifests are copied from master branch

How reproducible:
Always

Steps to Reproduce:
1. deploy clusterlogging
2. scale down cluster-logging-operator to 0
3. delete secret/master-certs
4. scale up cluster-logging-operator to 1
5. wait until the CLO recreate secret/master-certs, check the indices in the ES, and check logs in the Fluentd pod.

Actual results:
The Fluentd couldn't connect to ES after the secret/master-certs regenerated.

Expected results:
The logging stack could work well after secrets regenerated.

Additional info: