Bug 1835083

Description of problem:
Deploy logging 4.4, then upgrade logging from 4.4 to 4.5, all the curator and index management pods are in Error status:

$ oc get pod
NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-d857bb6db-2fjrr        1/1     Running     0          30m
curator-1589333400-7vl4w                        0/1     Completed   0          56m
curator-1589335200-rg5z2                        0/1     Error       0          26m
elasticsearch-cdm-m2j2lxw9-1-596649ffc8-nrz5k   2/2     Running     0          29m
elasticsearch-cdm-m2j2lxw9-2-78787b64cf-vvwvk   2/2     Running     0          26m
elasticsearch-cdm-m2j2lxw9-3-74d7bd8db4-n9fbf   2/2     Running     0          24m
elasticsearch-delete-app-1589336100-vg46f       0/1     Error       0          11m
elasticsearch-delete-audit-1589336100-9mqhr     0/1     Error       0          11m
elasticsearch-delete-infra-1589336100-7694k     0/1     Error       0          11m
elasticsearch-rollover-app-1589336100-7cc4s     0/1     Error       0          11m
elasticsearch-rollover-audit-1589336100-hz4jz   0/1     Error       0          11m
elasticsearch-rollover-infra-1589336100-pdgdz   0/1     Error       0          11m
fluentd-8dvgn                                   1/1     Running     0          27m
fluentd-q89wn                                   1/1     Running     0          29m
fluentd-r57cg                                   1/1     Running     0          27m
fluentd-sfvhg                                   1/1     Running     0          29m
fluentd-t2bdg                                   1/1     Running     0          28m
fluentd-xf9mm                                   1/1     Running     0          26m
kibana-6ff5c8d8f-s5tgq                          2/2     Running     0          65m

$ oc logs curator-1589335200-rg5z2
2020-05-13 02:00:17,460 INFO	Found curator configuration in [/etc/curator/settings/config.yaml]
2020-05-13 02:00:17,474 INFO	Converting config file.
Traceback (most recent call last):
  File "/usr/bin/curator", line 9, in <module>
    load_entry_point('elasticsearch-curator==5.2.0', 'console_scripts', 'curator')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/curator/cli.py", line 211, in cli
    run(config, action_file, dry_run)
  File "/usr/lib/python2.7/site-packages/curator/cli.py", line 158, in run
    client = get_client(**client_args)
  File "/usr/lib/python2.7/site-packages/curator/utils.py", line 800, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: TransportError(503, u'Open Distro not initialized')

$ oc logs elasticsearch-rollover-app-1589336100-7cc4s 
+++ echo eyJjb25kaXRpb25zIjp7Im1heF9hZ2UiOiI4aCIsIm1heF9kb2NzIjoxMjI4ODAsIm1heF9zaXplIjoiMTIwZ2IifX0=
+++ base64 -d
++ decoded='{"conditions":{"max_age":"8h","max_docs":122880,"max_size":"120gb"}}'
++++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+++ curl 'https://elasticsearch:9200/app-write/_rollover?pretty' -w '%{response_code}' -sv --cacert /etc/indexmanagement/keys/admin-ca -HContent-Type:application/json -XPOST '-HAuthorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InNqRFR2NXhoeHNocFB5WnVpQjNUTFZDSTRFZmJQY1gteGxSS3c4bXBnRXMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbG9nZ2luZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJlbGFzdGljc2VhcmNoLXRva2VuLXpsNTVxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImVsYXN0aWNzZWFyY2giLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmOGY0YTVmNi04ZGY3LTQ4NzQtYjQ1Ny00ZjNhNzIxNjRkMzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6b3BlbnNoaWZ0LWxvZ2dpbmc6ZWxhc3RpY3NlYXJjaCJ9.KCzRKnTaIPaNQjaStjbq1fKc2VfAB5BLtv4Qif9h8bB40HPoH72NSFNnFJQMNlpZdYnzZV8zqK_9bOI4EXnsioL8NkW5-YB3hHq3OROFZ1g6d2T1vQQM-I8MerXm_ELMc80XmSn-ugtEWOUXXBnRzRFcVjyuljFWxq_YDOcQW2t_bvlvESqaQrzF-nhkjKst6xrmbfOr0CvjRDdNOWSA3gz1hZ72RWkIl325PCkAaxO5BZ7XvnQNG0nyi42bDiacWxDh9S-YbzJTR4umFQnKaUSV9hqctS_X5XTVSW9yPGxX_RVDSQ3MVVyGB9IjnASS3MWfWea4hAKfDjeW5pZjOQ' -o /tmp/response.txt -d '{"conditions":{"max_age":"8h","max_docs":122880,"max_size":"120gb"}}'
* About to connect() to elasticsearch port 9200 (#0)
*   Trying 172.30.202.177...
* Connected to elasticsearch (172.30.202.177) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/indexmanagement/keys/admin-ca
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=logging-es,OU=OpenShift,O=Logging
* 	start date: May 13 00:26:17 2020 GMT
* 	expire date: May 13 00:26:17 2022 GMT
* 	common name: logging-es
* 	issuer: CN=openshift-cluster-logging-signer
> POST /app-write/_rollover?pretty HTTP/1.1
> User-Agent: curl/7.29.0
> Host: elasticsearch:9200
> Accept: */*
> Content-Type:application/json
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InNqRFR2NXhoeHNocFB5WnVpQjNUTFZDSTRFZmJQY1gteGxSS3c4bXBnRXMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbG9nZ2luZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJlbGFzdGljc2VhcmNoLXRva2VuLXpsNTVxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImVsYXN0aWNzZWFyY2giLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmOGY0YTVmNi04ZGY3LTQ4NzQtYjQ1Ny00ZjNhNzIxNjRkMzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6b3BlbnNoaWZ0LWxvZ2dpbmc6ZWxhc3RpY3NlYXJjaCJ9.KCzRKnTaIPaNQjaStjbq1fKc2VfAB5BLtv4Qif9h8bB40HPoH72NSFNnFJQMNlpZdYnzZV8zqK_9bOI4EXnsioL8NkW5-YB3hHq3OROFZ1g6d2T1vQQM-I8MerXm_ELMc80XmSn-ugtEWOUXXBnRzRFcVjyuljFWxq_YDOcQW2t_bvlvESqaQrzF-nhkjKst6xrmbfOr0CvjRDdNOWSA3gz1hZ72RWkIl325PCkAaxO5BZ7XvnQNG0nyi42bDiacWxDh9S-YbzJTR4umFQnKaUSV9hqctS_X5XTVSW9yPGxX_RVDSQ3MVVyGB9IjnASS3MWfWea4hAKfDjeW5pZjOQ
> Content-Length: 68
> 
} [data not shown]
* upload completely sent off: 68 out of 68 bytes
< HTTP/1.1 401 Unauthorized
< content-type: text/plain; charset=UTF-8
< content-length: 29
< 
{ [data not shown]
* Connection #0 to host elasticsearch left intact
++ code=401
++ '[' 401 == 200 ']'
++ cat /tmp/response.txt
Authentication finally failed++ exit 1


$ oc logs elasticsearch-delete-app-1589336100-vg46f 
++++ cat /var/run/secrets/kubernetes.io/serviceaccount/token
+++ curl -s 'https://elasticsearch:9200/app-*/_alias/app-write' --cacert /etc/indexmanagement/keys/admin-ca '-HAuthorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InNqRFR2NXhoeHNocFB5WnVpQjNUTFZDSTRFZmJQY1gteGxSS3c4bXBnRXMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbG9nZ2luZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJlbGFzdGljc2VhcmNoLXRva2VuLXpsNTVxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImVsYXN0aWNzZWFyY2giLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmOGY0YTVmNi04ZGY3LTQ4NzQtYjQ1Ny00ZjNhNzIxNjRkMzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6b3BlbnNoaWZ0LWxvZ2dpbmc6ZWxhc3RpY3NlYXJjaCJ9.KCzRKnTaIPaNQjaStjbq1fKc2VfAB5BLtv4Qif9h8bB40HPoH72NSFNnFJQMNlpZdYnzZV8zqK_9bOI4EXnsioL8NkW5-YB3hHq3OROFZ1g6d2T1vQQM-I8MerXm_ELMc80XmSn-ugtEWOUXXBnRzRFcVjyuljFWxq_YDOcQW2t_bvlvESqaQrzF-nhkjKst6xrmbfOr0CvjRDdNOWSA3gz1hZ72RWkIl325PCkAaxO5BZ7XvnQNG0nyi42bDiacWxDh9S-YbzJTR4umFQnKaUSV9hqctS_X5XTVSW9yPGxX_RVDSQ3MVVyGB9IjnASS3MWfWea4hAKfDjeW5pZjOQ' -HContent-Type:application/json
++ writeIndices='Authentication finally failed'
+++ cat
++ CMD='import json,sys
r=json.load(sys.stdin)
alias="app-write"
indices = [index for index in r if r[index]['\''aliases'\''][alias]['\''is_write_index'\'']]
if len(indices) > 0:
  print indices[0] '
+++ python -c 'import json,sys
r=json.load(sys.stdin)
alias="app-write"
indices = [index for index in r if r[index]['\''aliases'\''][alias]['\''is_write_index'\'']]
if len(indices) > 0:
  print indices[0] '
+++ echo 'Authentication finally failed'
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
++ writeIndex=


Version-Release number of selected component (if applicable):
logging 4.4: 4.4.0-202005120551 
logging 4.5 images are from 4.5.0-0.ci-2020-05-12-205117
4.5 manifests are copied from master branch.

How reproducible:
Always

Steps to Reproduce:
1. deploy logging 4.4 
2. upgrade logging to 4.5
3. check curator/index management pods status

Actual results:


Expected results:


Additional info:

Fluentd could connect to ES after upgrade to 4.5.
No this issue when deploy logging 4.5