Bug 1834311 - Kibana OAuth HTTP 500 error - x509: certificate signed by unknown authority
Summary: Kibana OAuth HTTP 500 error - x509: certificate signed by unknown authority
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: 4.5.0
Assignee: Periklis Tsirakidis
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks: 1833273 1833288 1855220
TreeView+ depends on / blocked
 
Reported: 2020-05-11 13:56 UTC by Periklis Tsirakidis
Modified: 2023-10-06 19:58 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Cluster-logging-operator did not reconcile the injected CA Bundle contents for fluentd. Same applies to elasticsearch-operator for kibana. Consequence: Fluentd and Kibana missing volume mounts to config maps with injected CA bundle Fix: Fetch anew the config map contents during reconciliation to ensure volume mounts. Result: Fluentd and Kibana mount the CA bundle config maps appropriately and certification works again
Clone Of:
Environment:
Last Closed: 2020-07-13 17:37:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
A stranage Login Popup Windows. (33.50 KB, image/png)
2020-05-19 08:40 UTC, Anping Li
no flags Details
Ingress logs (64.79 KB, text/plain)
2020-05-19 10:38 UTC, Anping Li
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 517 0 None closed Bug 1834311: Fix trusted ca bundle and hash reconciliation 2021-01-20 17:08:13 UTC
Github openshift elasticsearch-operator pull 351 0 None closed Bug 1834311: Fix trusted ca bundle and hash reconciliation 2021-01-20 17:08:13 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:37:48 UTC

Description Periklis Tsirakidis 2020-05-11 13:56:37 UTC
This bug was initially created as a copy of Bug #1833288

I am copying this bug because: 



Description of problem: Encountering https://bugzilla.redhat.com/show_bug.cgi?id= 1766187 in 4.4. The problem has already been fixed in 4.3


Version-Release number of selected component (if applicable): 4.4


Steps to Reproduce:
1. Install ClusterLogging with 4.4 subscription channel. CSV is clusterlogging.4.4.0-202004261927. The cluster is using a custom CA bundle.
2. Create ClusterLogging resource
3. Try to login to Kibana using Oauth

Actual results:
HTTP error 500

Logs Kibana Oauth proxy:
2020/05/08 07:15:26 oauthproxy.go:645: error redeeming code (client:172.28.20.20:40992): Post https://oauth-openshift.apps.dx01.od.sdx.corp/oauth/token: x509: certificate signed by unknown authority
2020/05/08 07:15:26 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error


Expected results:
Login to Kibana succeeds.


Additional info:
Kibana CM trusted ca bundle is created

─➤  oc get cm kibana-trusted-ca-bundle
NAME                       DATA   AGE
kibana-trusted-ca-bundle   1      2m58s


However it is not mounted in Kibana deployment.

  volumes:
    - name: kibana
      secret:
        secretName: kibana
        defaultMode: 420
    - name: kibana-proxy
      secret:
        secretName: kibana-proxy
        defaultMode: 420
    - name: kibana-token-9pswp
      secret:
        secretName: kibana-token-9pswp
        defaultMode: 420

Comment 3 Anping Li 2020-05-19 08:39:07 UTC
Blocked  in 4.5,  A Login Popup windows appears. and It didn't accept the correct user/password.

Comment 4 Anping Li 2020-05-19 08:40:03 UTC
Created attachment 1689820 [details]
A stranage Login Popup Windows.

Comment 5 Anping Li 2020-05-19 10:37:53 UTC
#All kibana logs looks good.
#I get the TLS handshake error in the oauth-openshift-599c4f66f4-xzdmf

I0519 10:26:06.795083       1 log.go:172] http: TLS handshake error from 10.129.2.9:51520: remote error: tls: unknown certificate
I0519 10:26:07.246785       1 log.go:172] http: TLS handshake error from 10.129.2.9:51550: EOF
I0519 10:26:32.536787       1 log.go:172] http: TLS handshake error from 10.129.2.9:51982: remote error: tls: unknown certificate


#10.129.2.9 is the ingress pod IP

oc get pods -o wide -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
router-default-66d49d799b-2zt9f   1/1     Running   0          21h   10.131.0.10   ip-10-0-54-195.us-east-2.compute.internal   <none>           <none>
router-default-66d49d799b-lrcrp   1/1     Running   0          21h   10.129.2.9    ip-10-0-69-202.us-east-2.compute.internal   <none>           <none>

#the ingress log attached.

Comment 6 Anping Li 2020-05-19 10:38:37 UTC
Created attachment 1689839 [details]
Ingress logs

Comment 7 Anping Li 2020-05-25 10:11:20 UTC
1) trust-ca-bundle Configmap are mountd
 oc get configmap kibana-trusted-ca-bundle  -o name -n openshift-logging
configmap/kibana-trusted-ca-bundle
$ oc get pod kibana-79d74cd97-ffn75 -o json |jq '.spec.volumes'
[
   {
    "configMap": {
      "defaultMode": 420,
      "items": [
        {
          "key": "ca-bundle.crt",
          "path": "tls-ca-bundle.pem"
        }
      ],
      "name": "kibana-trusted-ca-bundle"
    },
    "name": "kibana-trusted-ca-bundle"
  }
 ]
$ oc get pod kibana-79d74cd97-ffn75 -o json |jq '.spec.containers[].volumeMounts'
[
  {
    "mountPath": "/etc/kibana/keys",
    "name": "kibana",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kibana-token-z7fr4",
    "readOnly": true
  }
]
[
  {
    "mountPath": "/secret",
    "name": "kibana-proxy",
    "readOnly": true
  },
  {
    "mountPath": "/etc/pki/ca-trust/extracted/pem/",
    "name": "kibana-trusted-ca-bundle",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kibana-token-z7fr4",
    "readOnly": true
  }
]
2) Login kibana. no x509 error.

Comment 8 errata-xmlrpc 2020-07-13 17:37:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.