Bug 1930368 - kibana-proxy CrashLoopBackoff with error Invalid configuration cookie_secret must be 16, 24, or 32 bytes to create an AES cipher
Summary: kibana-proxy CrashLoopBackoff with error Invalid configuration cookie_secret...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Sergey Yedrikov
QA Contact: Kabir Bharti
URL:
Whiteboard: logging-core
Depends On:
Blocks: 1972718
TreeView+ depends on / blocked
 
Reported: 2021-02-18 18:26 UTC by Mani
Modified: 2024-12-20 19:39 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: In some circumstances, the Kibana session secret is generated with the wrong length. Consequence: kibana-proxy container goes into CrashLoopBackoff state with error "Invalid configuration cookie_secret must be 16, 24, or 32 bytes to create an AES cipher" Fix: Use correct hexdump parameters so it does not deviate from the "1 byte -> 2 hex nibbles" scheme. Result: The length of the Kibana session secret is always 32 bytes.
Clone Of:
Environment:
Last Closed: 2021-08-04 19:49:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:2889 0 None None None 2021-08-04 19:49:52 UTC

Description Mani 2021-02-18 18:26:50 UTC
Description of problem:

Kibana crasshloop with below error on kibana_proxy.

kibana-proxy log:
2021/02/15 13:10:45 main.go:138: Invalid configuration:
  cookie_secret must be 16, 24, or 32 bytes to create an AES cipher when pass_access_token == true or cookie_refresh != 0, but is 29 bytes.

How reproducible:
N/a



Actual results:
Kibana pod crash looping


Additional info:
Facing this after the  upgrade of RHOCP 4.5.22 and 4.5.31,elasticsearch-operator.4.5.0-202011132127.p0 to elasticsearch-operator.4.5.0-202102041049.p0

Comment 8 Lucas López Montero 2021-06-09 14:16:25 UTC
Do you have news about this issue? Thank you.

Comment 9 Sergey Yedrikov 2021-06-09 14:25:49 UTC
Fix PR https://github.com/openshift/cluster-logging-operator/pull/1046 merged in the upstream.

Comment 12 Kabir Bharti 2021-07-08 01:04:26 UTC
Tested on the below CLO and EO version.

[kbharti@cube ~]$ oc get csv
NAME                                        DISPLAY                            VERSION              REPLACES   PHASE
clusterlogging.4.6.0-202107070256           Cluster Logging                    4.6.0-202107070256              Succeeded
elasticsearch-operator.4.6.0-202107070256   OpenShift Elasticsearch Operator   4.6.0-202107070256              Succeeded

Managed mode CLO instance:
kibana-session-secret is regenerated when removed/changed to length > 32 bytes. Regenerated secret == 32 bytes in length.
Works fine.

Unmanaged mode CLO instance:
Changed kibana-session-secret to length > 32 bytes but Kibana-proxy pod does not fail with CrashLoopBackOff error.

Secret changed under tmp/ocp-clo/kibana-session-secret to AB3D39BDA98B127E23B9TETDGDDAA308400B95F2 (length == 40)

Logs from Kibana-proxy when working with the above secret.
2021/07/08 01:01:41 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:5601/"
2021/07/08 01:01:41 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-logging:kibana
2021/07/08 01:01:41 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:24h0m0s domain:<default> samesite: refresh:disabled
2021/07/08 01:01:41 http.go:61: HTTP: listening on 127.0.0.1:4180
2021/07/08 01:01:41 http.go:107: HTTPS: listening on [::]:3000
I0708 01:01:41.877270       1 dynamic_serving_content.go:130] Starting serving::/secret/server-cert::/secret/server-key

No error seen in the logs.

Marking retest in 4.6 as failure.

Comment 18 errata-xmlrpc 2021-08-04 19:49:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.41 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2889


Note You need to log in before you can comment on or make changes to this bug.