Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1834311

Summary: Kibana OAuth HTTP 500 error - x509: certificate signed by unknown authority
Product: OpenShift Container Platform Reporter: Periklis Tsirakidis <periklis>
Component: LoggingAssignee: Periklis Tsirakidis <periklis>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, jritter, qitang
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Cluster-logging-operator did not reconcile the injected CA Bundle contents for fluentd. Same applies to elasticsearch-operator for kibana. Consequence: Fluentd and Kibana missing volume mounts to config maps with injected CA bundle Fix: Fetch anew the config map contents during reconciliation to ensure volume mounts. Result: Fluentd and Kibana mount the CA bundle config maps appropriately and certification works again
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:37:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1833273, 1833288, 1855220    
Attachments:
Description Flags
A stranage Login Popup Windows.
none
Ingress logs none

Description Periklis Tsirakidis 2020-05-11 13:56:37 UTC
This bug was initially created as a copy of Bug #1833288

I am copying this bug because: 



Description of problem: Encountering https://bugzilla.redhat.com/show_bug.cgi?id= 1766187 in 4.4. The problem has already been fixed in 4.3


Version-Release number of selected component (if applicable): 4.4


Steps to Reproduce:
1. Install ClusterLogging with 4.4 subscription channel. CSV is clusterlogging.4.4.0-202004261927. The cluster is using a custom CA bundle.
2. Create ClusterLogging resource
3. Try to login to Kibana using Oauth

Actual results:
HTTP error 500

Logs Kibana Oauth proxy:
2020/05/08 07:15:26 oauthproxy.go:645: error redeeming code (client:172.28.20.20:40992): Post https://oauth-openshift.apps.dx01.od.sdx.corp/oauth/token: x509: certificate signed by unknown authority
2020/05/08 07:15:26 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error


Expected results:
Login to Kibana succeeds.


Additional info:
Kibana CM trusted ca bundle is created

─➤  oc get cm kibana-trusted-ca-bundle
NAME                       DATA   AGE
kibana-trusted-ca-bundle   1      2m58s


However it is not mounted in Kibana deployment.

  volumes:
    - name: kibana
      secret:
        secretName: kibana
        defaultMode: 420
    - name: kibana-proxy
      secret:
        secretName: kibana-proxy
        defaultMode: 420
    - name: kibana-token-9pswp
      secret:
        secretName: kibana-token-9pswp
        defaultMode: 420

Comment 3 Anping Li 2020-05-19 08:39:07 UTC
Blocked  in 4.5,  A Login Popup windows appears. and It didn't accept the correct user/password.

Comment 4 Anping Li 2020-05-19 08:40:03 UTC
Created attachment 1689820 [details]
A stranage Login Popup Windows.

Comment 5 Anping Li 2020-05-19 10:37:53 UTC
#All kibana logs looks good.
#I get the TLS handshake error in the oauth-openshift-599c4f66f4-xzdmf

I0519 10:26:06.795083       1 log.go:172] http: TLS handshake error from 10.129.2.9:51520: remote error: tls: unknown certificate
I0519 10:26:07.246785       1 log.go:172] http: TLS handshake error from 10.129.2.9:51550: EOF
I0519 10:26:32.536787       1 log.go:172] http: TLS handshake error from 10.129.2.9:51982: remote error: tls: unknown certificate


#10.129.2.9 is the ingress pod IP

oc get pods -o wide -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
router-default-66d49d799b-2zt9f   1/1     Running   0          21h   10.131.0.10   ip-10-0-54-195.us-east-2.compute.internal   <none>           <none>
router-default-66d49d799b-lrcrp   1/1     Running   0          21h   10.129.2.9    ip-10-0-69-202.us-east-2.compute.internal   <none>           <none>

#the ingress log attached.

Comment 6 Anping Li 2020-05-19 10:38:37 UTC
Created attachment 1689839 [details]
Ingress logs

Comment 7 Anping Li 2020-05-25 10:11:20 UTC
1) trust-ca-bundle Configmap are mountd
 oc get configmap kibana-trusted-ca-bundle  -o name -n openshift-logging
configmap/kibana-trusted-ca-bundle
$ oc get pod kibana-79d74cd97-ffn75 -o json |jq '.spec.volumes'
[
   {
    "configMap": {
      "defaultMode": 420,
      "items": [
        {
          "key": "ca-bundle.crt",
          "path": "tls-ca-bundle.pem"
        }
      ],
      "name": "kibana-trusted-ca-bundle"
    },
    "name": "kibana-trusted-ca-bundle"
  }
 ]
$ oc get pod kibana-79d74cd97-ffn75 -o json |jq '.spec.containers[].volumeMounts'
[
  {
    "mountPath": "/etc/kibana/keys",
    "name": "kibana",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kibana-token-z7fr4",
    "readOnly": true
  }
]
[
  {
    "mountPath": "/secret",
    "name": "kibana-proxy",
    "readOnly": true
  },
  {
    "mountPath": "/etc/pki/ca-trust/extracted/pem/",
    "name": "kibana-trusted-ca-bundle",
    "readOnly": true
  },
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kibana-token-z7fr4",
    "readOnly": true
  }
]
2) Login kibana. no x509 error.

Comment 8 errata-xmlrpc 2020-07-13 17:37:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409