Bug 1930368

Summary: kibana-proxy CrashLoopBackoff with error Invalid configuration cookie_secret must be 16, 24, or 32 bytes to create an AES cipher
Product: OpenShift Container Platform Reporter: Mani <mmohan>
Component: LoggingAssignee: Sergey Yedrikov <syedriko>
Status: CLOSED ERRATA QA Contact: Kabir Bharti <kbharti>
Severity: high Docs Contact:
Priority: high    
Version: 4.6CC: anli, aos-bugs, dgautam, jcantril, llopezmo, msweiker, rguilme1, scott.worthington, syedriko
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: logging-core
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: In some circumstances, the Kibana session secret is generated with the wrong length. Consequence: kibana-proxy container goes into CrashLoopBackoff state with error "Invalid configuration cookie_secret must be 16, 24, or 32 bytes to create an AES cipher" Fix: Use correct hexdump parameters so it does not deviate from the "1 byte -> 2 hex nibbles" scheme. Result: The length of the Kibana session secret is always 32 bytes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-04 19:49:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1972718    

Description Mani 2021-02-18 18:26:50 UTC
Description of problem:

Kibana crasshloop with below error on kibana_proxy.

kibana-proxy log:
2021/02/15 13:10:45 main.go:138: Invalid configuration:
  cookie_secret must be 16, 24, or 32 bytes to create an AES cipher when pass_access_token == true or cookie_refresh != 0, but is 29 bytes.

How reproducible:
N/a



Actual results:
Kibana pod crash looping


Additional info:
Facing this after the  upgrade of RHOCP 4.5.22 and 4.5.31,elasticsearch-operator.4.5.0-202011132127.p0 to elasticsearch-operator.4.5.0-202102041049.p0

Comment 8 Lucas López Montero 2021-06-09 14:16:25 UTC
Do you have news about this issue? Thank you.

Comment 9 Sergey Yedrikov 2021-06-09 14:25:49 UTC
Fix PR https://github.com/openshift/cluster-logging-operator/pull/1046 merged in the upstream.

Comment 12 Kabir Bharti 2021-07-08 01:04:26 UTC
Tested on the below CLO and EO version.

[kbharti@cube ~]$ oc get csv
NAME                                        DISPLAY                            VERSION              REPLACES   PHASE
clusterlogging.4.6.0-202107070256           Cluster Logging                    4.6.0-202107070256              Succeeded
elasticsearch-operator.4.6.0-202107070256   OpenShift Elasticsearch Operator   4.6.0-202107070256              Succeeded

Managed mode CLO instance:
kibana-session-secret is regenerated when removed/changed to length > 32 bytes. Regenerated secret == 32 bytes in length.
Works fine.

Unmanaged mode CLO instance:
Changed kibana-session-secret to length > 32 bytes but Kibana-proxy pod does not fail with CrashLoopBackOff error.

Secret changed under tmp/ocp-clo/kibana-session-secret to AB3D39BDA98B127E23B9TETDGDDAA308400B95F2 (length == 40)

Logs from Kibana-proxy when working with the above secret.
2021/07/08 01:01:41 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:5601/"
2021/07/08 01:01:41 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-logging:kibana
2021/07/08 01:01:41 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:24h0m0s domain:<default> samesite: refresh:disabled
2021/07/08 01:01:41 http.go:61: HTTP: listening on 127.0.0.1:4180
2021/07/08 01:01:41 http.go:107: HTTPS: listening on [::]:3000
I0708 01:01:41.877270       1 dynamic_serving_content.go:130] Starting serving::/secret/server-cert::/secret/server-key

No error seen in the logs.

Marking retest in 4.6 as failure.

Comment 18 errata-xmlrpc 2021-08-04 19:49:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.41 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2889