Bug 1669368
| Summary: | No openshift metadata info in log when log collection type is rsyslog. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> | ||||
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Anping Li <anli> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 4.1.0 | CC: | aos-bugs, eparis, jcantril, jialiu, jokerman, mmccomas, mrogers, nhosoi, qitang, rmeggins, xtian | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.2.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-04-03 20:36:00 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1668534, 1670282 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Qiaoling Tang
2019-01-25 03:48:33 UTC
Please attach rsyslog pod log I could not reproduce the problem. This is a similar search result. As you see in the pastebin, it contains kubernetes metadata. http://pastebin.test.redhat.com/702329 Only a difference is my cluster-logging-operator is built from the master branch with a small tweak (but it should not affect the metadata handling.) When are we having the next respin? Note: oc logs on my rsyslog pod is very quiet. INFO: Disabling Prometheus endpoint rsyslogd 8.39.0: running as pid 1, enabling container-specific defaults, press ctl-c to terminate rsyslog Rsyslog pod logs: $ oc logs rsyslog-nhrst INFO: Disabling Prometheus endpoint rsyslogd 8.39.0: running as pid 1, enabling container-specific defaults, press ctl-c to terminate rsyslog rsyslogd: mmkubernetes: failed to connect to [https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/openshift-sdn] - 60:Peer certificate cannot be authenticated with given CA certificates [v8.39.0] rsyslogd: action 'action-5-mmkubernetes' (module 'mmkubernetes') message lost, could not be processed. Check for additional error messages before this one. [v8.39.0] rsyslogd: omelasticsearch: we are suspending ourselfs due to server failure 7: Failed to connect to elasticsearch port 9200: No route to host [v8.39.0 try http://www.rsyslog.com/e/2007 ] Please provide the output of the following: oc get ds rsyslog -o yaml oc get cm rsyslog -o yaml then - get the rsyslog pod and oc rsh $rsyslog_pod from inside the rsyslog_pod ls -alrtF /var/run/secret/kubernetes.io/serviceaccount cat /var/run/secret/kubernetes.io/serviceaccount/ca.crt curl -vs --cacert /var/run/secret/kubernetes.io/serviceaccount/ca.cr https://kubernetes.default.svc.cluster.local:443 Created attachment 1524178 [details]
Rsyslog info
(In reply to Rich Megginson from comment #4) > Please provide the output of the following: > > oc get ds rsyslog -o yaml > oc get cm rsyslog -o yaml > > then - get the rsyslog pod and oc rsh $rsyslog_pod > > from inside the rsyslog_pod > > ls -alrtF /var/run/secret/kubernetes.io/serviceaccount > cat /var/run/secret/kubernetes.io/serviceaccount/ca.crt Sorry - this is /var/run/secrets/kubernetes.io/serviceaccount and /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > > curl -vs --cacert /var/run/secret/kubernetes.io/serviceaccount/ca.cr > https://kubernetes.default.svc.cluster.local:443 Sorry - this should be curl -vs --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc.cluster.local:443 $ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-657c4f64bc-prspn 1/1 Running 0 2h elasticsearch-clientdatamaster-0-1-6cf8db77f4-qst69 1/1 Running 0 2h elasticsearch-clientdatamaster-0-2-588bcf7b7f-9v42c 1/1 Running 0 2h elasticsearch-operator-86cc7cb548-lslxf 1/1 Running 0 2h kibana-6c9d89bc58-kcthh 2/2 Running 0 2h rsyslog-49xzf 1/1 Running 0 2h rsyslog-92z6v 1/1 Running 0 2h rsyslog-rsf4t 1/1 Running 0 2h rsyslog-xkcpr 1/1 Running 0 2h rsyslog-z4ptj 1/1 Running 0 2h [qitang@192 aws]$ oc rsh ^C [qitang@192 aws]$ oc exec elasticsearch-clientdatamaster-0-1-6cf8db77f4-qst69 -- indices Mon Jan 28 12:35:12 UTC 2019 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .operations.2019.01.28 F4QKlCRtQXiqYM8jvwy62g 2 1 646622 0 1327 674 green open .kibana cRVcWJL8Q_i-0dFwZs7tHw 1 1 5 0 0 0 green open .searchguard G2pdQok8TuKzfYuiDmbQqQ 1 1 5 0 0 0 [qitang@192 aws]$ oc rsh rsyslog-49xzf sh-4.4# ls -alrtF /var/run/secrets/kubernetes.io/serviceaccount total 0 lrwxrwxrwx. 1 root root 12 Jan 28 10:06 token -> ..data/token lrwxrwxrwx. 1 root root 16 Jan 28 10:06 namespace -> ..data/namespace lrwxrwxrwx. 1 root root 13 Jan 28 10:06 ca.crt -> ..data/ca.crt lrwxrwxrwx. 1 root root 31 Jan 28 10:06 ..data -> ..2019_01_28_10_06_35.837725545/ drwxr-xr-x. 2 root root 100 Jan 28 10:06 ..2019_01_28_10_06_35.837725545/ drwxrwxrwt. 3 root root 140 Jan 28 10:06 ./ drwxr-xr-x. 3 root root 60 Jan 28 10:06 ../ sh-4.4# cat /var/run/secrets/kubernetes.io/serviceaccount/ ..2019_01_28_10_06_35.837725545/ ..data/ ca.crt namespace token sh-4.4# cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -----BEGIN CERTIFICATE----- MIIDMDCCAhigAwIBAgIIG208sgthFqkwDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE CxMJb3BlbnNoaWZ0MRAwDgYDVQQDEwdyb290LWNhMB4XDTE5MDEyODA2NTY0N1oX DTI5MDEyNTA2NTY0N1owJTERMA8GA1UECxMIYm9vdGt1YmUxEDAOBgNVBAMTB2t1 YmUtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQClzGxvtN/wB5FQ vQ4iQecJpWJJ+Z3pGJ6PhkJkXYLse4a1HIxq4NKaV4mBRXMg7w/HlQUlrVwOSNl0 XJ6PTKajuz2nnE1zW9F8ueBY5j5pvo8SNfOvp3BACjFcIZIa4623pnCPLwvMI9tA hyCAcPqbGhSZNtQ7R61pffuk2evMZWtbwKUBWRTKKNRNWsfdD88kzLy6KvIZvL8j Aare2Ut8yxjsAYCGLUh2dd0LoRgrhUM2iySzyTWQOf7qdvrabFjtxe03ZedQ14NN 2kwgGKAnndbOdc5eUsmlUP/ChPNJonWL5xWwz89sShihmkXNqOIUWubIgyONWign PH3937VvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/ MB0GA1UdDgQWBBQ5Yg4xGjy1Iqs7pXXiFWTKaEbhsjAfBgNVHSMEGDAWgBQ5Yg4x Gjy1Iqs7pXXiFWTKaEbhsjANBgkqhkiG9w0BAQsFAAOCAQEAiYDsCalFsi4uOSGg sVejnPL98WsPHrTFAzqj7dp9UzBJGcNKXhBhTf/zKASzNgDCAhT0XyMDUvlW2wed 3MOFBjh7yRjEB6SCWACu1RkjylmjT3utkOvQSIkZXeEkePD/hXxUQexKtdXJnKR1 mjTgm4UBllpXYFzqnpJub6Y/vvadTXwBoQr2XyV7ldNZKvS+rNMP1AihAKkY18xJ UqYHgmzdpMHD6gUFavB/FhhvVXJ47vBWvOfzetr4h7zG6m9y5YAstu+Da4Yu5g+g bRdCONkN4Yyg1MxYD4VAE9Cq88IJcrqPrJ2HyQBI4wFxFeDCMn73AWk3BqZwKdwO aFO7Tg== -----END CERTIFICATE----- sh-4.4# curl -vs --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc.cluster.local:443 * Rebuilt URL to: https://kubernetes.default.svc.cluster.local:443/ * Trying 172.30.0.1... * TCP_NODELAY set * Connected to kubernetes.default.svc.cluster.local (172.30.0.1) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: unable to get issuer certificate * Closing connection 0 sh-4.4# The problem is that the ca cert is only the intermediate cert and does not contain the full CA cert chain. rsyslog requires the full CA cert chain all the way up to the root.
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1976302546878273193 (0x1b6d3cb20b6116a9)
Signature Algorithm: sha256WithRSAEncryption
Issuer: OU = openshift, CN = root-ca
Validity
Not Before: Jan 28 06:56:47 2019 GMT
Not After : Jan 25 06:56:47 2029 GMT
Subject: OU = bootkube, CN = kube-ca
We are missing the CA cert for OU = openshift, CN = root-ca
There was a previous bug for this, and I thought that OpenShift had fixed it by including the entire cert chain in the ca.crt file.
What version of the openshift-installer did you use?
I tested it on the beta drop candidate build, it was: openshift-install v4.0.0-0.148.0.0-dirty , v0.11.0 oc version: v4.0.0-0.148.0 RHCOS:47.280 Payload:4.0.0-0.nightly-2019-01-25-205123 , 4.0.0-0.2 Also tested in 4.0.0-0.alpha-2019-01-29-032610, got the same result as comment 7. This is a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1670282 (In reply to Rich Megginson from comment #11) > This is a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1670282 However - we can't seem to rely on getting a CA cert that we can use reliably in rsyslog - therefore, we need to address this in rsyslog. Design: The CLO will add some code to find the cluster Root CA cert, and concatenate it with the service CA cert (e.g. /var/run/secrets/kubernetes.io/serviceaccount/ca.crt or service-ca.crt). The CLO will add this to the secret used by rsyslog. rsyslog will use this CA cert instead of trusting the one from the service (e.g. /var/run/secrets/kubernetes.io/serviceaccount/ca.crt or service-ca.crt) Moving to 4.2 Closing WONTFIX as this was resolved by the auth team. c#13 is an RFE for rsyslog |