Description of problem: After upgrade logging from v3.10 to v3.11. The kibana is CrashLoopBackOff. It report error main.go:134: Invalid configuration: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher when pass_access_token == true. [anli@upg_slave_qeos10 310]$ oc logs -c kibana-proxy logging-kibana-2-txw24 2018/08/13 07:55:20 provider.go:526: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server 2018/08/13 07:55:20 provider.go:572: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server { "issuer": "https://ec2-54-152-157-183.compute-1.amazonaws.com:8443", "authorization_endpoint": "https://ec2-54-152-157-183.compute-1.amazonaws.com:8443/oauth/authorize", "token_endpoint": "https://ec2-54-152-157-183.compute-1.amazonaws.com:8443/oauth/token", "scopes_supported": [ "user:check-access", "user:full", "user:info", "user:list-projects", "user:list-scoped-projects" ], "response_types_supported": [ "code", "token" ], "grant_types_supported": [ "authorization_code", "implicit" ], "code_challenge_methods_supported": [ "plain", "S256" ] } 2018/08/13 07:55:20 main.go:134: Invalid configuration: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher when pass_access_token == true or cookie_refresh != 0, but is 152 bytes. note: cookie secret was base64 decoded from "rxD7rpeIwFzDQMY3twdzItK2qND80yjtJ7N3wEVgmDKEK3psZwXeDP6czzu5thRKkzo Version-Release number of selected component (if applicable): openshift-ansible-v3.11.0-0.14.0.0 Logging v3.11.0-0.14.0.0 How reproducible: always Steps to Reproduce: 1. Deploy logging v3.10 on OCP v3.10 using openshift-ansible-3.10 2. Upgrade Logging v3.10 to Logging v3.11 on OCP 3.10 using openshift-ansible-3.11.0-0.14.0.0 Notes: The OCP couldn't be upgrade from v3.10 to v3.11. So we don't peform OCP upgrade in this case. To deploy logging v3.11 to OCP 3.10, You must set the following inventory file. oreg_url=registry.xxx.com/openshift3/ose-${component}:v3.11 openshift_image_tag=v3.11.0 openshift_pkg_version=-3.11.0 openshift_logging_curator_image=registry.xxx.com/openshift3/openshift3/ose-logging-curator5:v3.11.0 openshift_logging_kibana_image=registry.xxx.com/openshift3/ose-logging-kibana5:v3.11.0 openshift_logging_elasticsearch_image=registry.xxx.com/openshift3/ose-logging-elasticsearch5:v3.11.0 3. Check the kibana status after upgrade Actual results: The kibana5 couldn't be deployed. [anli@upg_slave_qeos10 310]$ oc get pods NAME READY STATUS RESTARTS AGE logging-es-data-master-n9dmyn0x-2-tjcc4 2/2 Running 0 1m logging-fluentd-58gv5 1/1 Running 0 2m logging-fluentd-g8jb5 1/1 Running 0 2m logging-fluentd-k59pr 1/1 Running 0 1m logging-fluentd-s4stw 1/1 Running 0 2m logging-kibana-1-n2sc5 2/2 Running 0 30m logging-kibana-2-deploy 1/1 Running 0 4m logging-kibana-2-txw24 1/2 CrashLoopBackOff 5 4m Expected results: The kibana can be updated to kibana5
@Mo, Can you provide any advice here?
I am moving this to the auth team since we own the proxy. Can we get the full YAML of the templates used to create these components as well as the output of: oc get all -o yaml oc get secrets -o yaml
This error comes from the newly used oauth-proxy. The problem is that the previously used fabric8io/openshift-auth-proxy had different requirements on cookie secrets. I suggest that we re-generate the cookie secret during upgrade. The upgrade playbook reads secrets and certs from the file system, if they exist. So deleting the secret file before running the generate certs task should solve the problem. Page refresh in browser and a fresh log-in will be necessary after the update.
Simo, Is there something here to be 'fixed' in the component or simply a usage bug as identified in the PR? Do we simply need to regen the secret
On our side we'll eventually take care of this, but for now on your side just regen the secret.
Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/a696933a304d00cb3891ec49d938084037aa7954 Bug 1615275. Regenerate session_secret if it can't be used with oauth-proxy session_secret generated by 3.10 is 200 bytes. oauth-proxy can use 16, 24 or 32 bytes session_secret. https://github.com/openshift/openshift-ansible/commit/2fb6224c12fffd7862a0e0cceba4eac57279c652 Merge pull request #9613 from t0ffel/master Bug 1615275. Regenerate session_secret if it can't be used with oauth-proxy
*** Bug 1618581 has been marked as a duplicate of this bug. ***
The kibana can be started and login after update by openshift-anisble:v3.11.0-0.17.0.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652