Description of problem: Using rsyslog as log collector, there is no openshift metadata info in log entities. All the logs are sent to .operations.* index in ES. $ oc get pod -n qitang1 NAME READY STATUS RESTARTS AGE centos-logtest-kdd87 1/1 Running 0 10m $ oc exec elasticsearch-clientdatamaster-0-1-f96df8ccf-cppcz -- indices Fri Jan 25 03:33:07 UTC 2019 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .operations.2019.01.25 GZNEwnghTFia5Ey2Qn8Ldw 3 1 18078 0 35 17 green open .kibana 6qP_NtaHRVeJJNX6y1sfLA 1 1 5 0 0 0 green open .searchguard la7RYgWtQom58-0rXC9e9w 1 1 5 2 0 0 Log collected by rsyslog: $ oc exec elasticsearch-clientdatamaster-0-1-f96df8ccf-cppcz -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key 'https://localhost:9200/*/_search?pretty&q=centos-logtest-kdd87' { "took" : 23, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 935, "max_score" : 16.30296, "hits" : [ { "_index" : ".operations.2019.01.25", "_type" : "com.redhat.viaq.common", "_id" : "6545C78121C14CDFA95261BCEAE6C730", "_score" : 16.30296, "_source" : { "pipeline_metadata" : { "collector" : { "original_raw_message" : "2019-01-25 03:24:12,791 - SVTLogger - INFO - centos-logtest-kdd87 : 2 : BQjwNHCrKSIZoDsW3sUwYpVYMu12Z73MLWtDn6c5X6sdYGvRK8UzpOk4I3CD4QIj9mSwX237UPOG4s3MFjLXiH9VyE CoTros2EL4xn8eCuCDXguVU5jHjNVe3t8blPkNg5kPNFtWWDonTOrzz5t1tHbKIuH1gNrqrFNNxYLRb7bQWvTZtL3n WRH3id2ATf65UdY19R2UCGWgCVwY101MdRAMTLxIpIK78PYTHRQEEvNeR2oFxVnCujcufx75acC9MosXT1sQmTJyMG EeNMkvdb78UFUfKOFwAZomioDfWszAwf80BqHhpiRX3WvNppKuB2y0AEOIWaZq3w3hsyG8DgD8cXUj6ViH0JOTnApf yLQU1pBG85n4iDCAOi5HVXKfxyj8jlKGlwrSLDsQNbqNkpM3porRmxC5fxXOoyR6sOIFsY7GtFR6sJUz25iTTR7I8P 3grzEO0GuBf7756iiIzLjlxVo0PSIbsSXKogaNf72aWlnQmYwR9mq69XS3eKaKRF68XVODtoX7NWqywrxYvHgmbmcI WLvcCm8OozPNil8i4p9tRBeCcVw5S3eaxsQ9fzmDrDbPcgP5crH6UAwNGK1508eMi5PpCFcdTmmofqEbQrORYXjoQu 0exgWLQl9SvNsUFtdVTN14ZYj2Ue2sWKbmN0Bax0COoyYUqN9JIN8k6MlkHaehFEJPE8YmJwBFMXc1iSNfguZB1UqZ uZRtgjblNJTJFgOWsWah8W0e6QUN3WWHECAwTsf8g6NcFLiKuapNTsTuS8Gy6Z28kauU2Ass8Mhf1S9IGehwMDcOVI xbeaWarnNCSEU60slwBGR3I99tGjHzV153H7euj40BVd6Px0vhFCdqAyBGwkzma69dLZm1XH9whlDMhx8DVXFdi0b1 htn2aWJztZwhrugbQtgcl3qCF6pgFFFulNcHLVZMNLfQgDUF0js9hl6CUdN1fMlGmDB2B4D1iUuSmtaSe3CtJzQNre YUaY2CIWJTmojDRQlpDa2ExnlWnqVgAjlD7f06uNTGyGYdOS4ASuGNVxHjCtnic7NtZj7b2ggVfEFIJfwP8PKGi7Dt sWofnWmK1vFUKW41P0k2sW57urYG7bq0Kro5vK4M651r0gOy1NFn6rKhHRZiAqQ9GGZZwvzqW7yKufNn0tiRY0HJpR tax8AVB1SGQNGU28s7jO0uBAFCcBchWdKLbLYBsJimk8gbORHQoc6jz1HDFr1P8giMrOHSR8MS040F7hsnYirbayex IuxgWjfPjQLr2okAe8bR5PTIrm8RPol0LPtbV6rT8A2hvxiWUvEwyR9D6SBgSS0PLUqH6xluIMU2n7XJdKTeGqEd0Z AGjpQVJYrcLsFrqRT661H1jGopq3SuYsqhF2mcw3UBqy67IO2NgNYigqP4NJOPdM6Luw11tTtbk7JMxKIUyMoXSwYz WDZ6tgUWsnCfMgmVJT6iaIUOgfuHvGtsazUC6W37inHeUeGrfoCqM0pBcXLzP1V2Lojt1GBDqzn3Tvar9vRcrEXA7A Phm5Fv1V9eVlAm8YRkbKkiqpg4Ll5xTxULaHKFHp5j42yqz4G6hedo53qhpsnM7dfbklnmrKE63uM9H34FsaRUjQLi THB5c0My0B4LeeIyYyCQN5bKXUkfOiqV4zWqALgku2LRfZqsdqhBSoP5VEl0ho8C3t8VCqHV9I7YgzHuLg3S5RKGkn kYOTIXjeFvZMy2cooX26CLV1b4C3O5sci1qtXztr6k7JGAQgi4kUN6lCKyVlCw1jpAXarP56QlFLh4TqsAAAMz1xuz PUIkRbdoVfk92govM86u4XNzc3zofq0PX615kAR8VSctHkRupPxb9Wcyipu59kfjPFMNzGHFxBuUhIG0q2VnebhLHw 47ydYCL90f0MyTDz3vkNCiIxiqe9fLdZsd2IESDqPQniUjp0tLbPCl9bw6yGVjNcz0n0G2tWDiBe9KDJXnrx5eTcJ", "name" : "rsyslog", "inputname" : "imfile", "received_at" : "2019-01-25T03:24:12.791954+00:00", "ipaddr4" : "127.0.0.1", "ipaddr6" : "::ffff:127.0.0.1", "version" : "8.39.0 " } }, "@timestamp" : "2019-01-25T03:24:12.791775362+00:00", "stream" : "stderr", "level" : "err", "file" : "/var/log/containers/centos-logtest-kdd87_qitang1_centos-logtest-97dd38682c0c18932cb7688c5dac176e9164dec79608aff23cf39ee4e6850362.log", "offset" : "2118", "message" : "2019-01-25 03:24:12,791 - SVTLogger - INFO - centos-logtest-kdd87 : 2 : BQjwNHCrKSIZoDsW3sUwYpVYMu12Z73MLWtDn6c5X6sdYGvRK8UzpOk4I3CD4QIj9mSwX237UPOG4s3MFjLXiH9VyE CoTros2EL4xn8eCuCDXguVU5jHjNVe3t8blPkNg5kPNFtWWDonTOrzz5t1tHbKIuH1gNrqrFNNxYLRb7bQWvTZtL3n WRH3id2ATf65UdY19R2UCGWgCVwY101MdRAMTLxIpIK78PYTHRQEEvNeR2oFxVnCujcufx75acC9MosXT1sQmTJyMG EeNMkvdb78UFUfKOFwAZomioDfWszAwf80BqHhpiRX3WvNppKuB2y0AEOIWaZq3w3hsyG8DgD8cXUj6ViH0JOTnApf yLQU1pBG85n4iDCAOi5HVXKfxyj8jlKGlwrSLDsQNbqNkpM3porRmxC5fxXOoyR6sOIFsY7GtFR6sJUz25iTTR7I8P 3grzEO0GuBf7756iiIzLjlxVo0PSIbsSXKogaNf72aWlnQmYwR9mq69XS3eKaKRF68XVODtoX7NWqywrxYvHgmbmcI WLvcCm8OozPNil8i4p9tRBeCcVw5S3eaxsQ9fzmDrDbPcgP5crH6UAwNGK1508eMi5PpCFcdTmmofqEbQrORYXjoQu 0exgWLQl9SvNsUFtdVTN14ZYj2Ue2sWKbmN0Bax0COoyYUqN9JIN8k6MlkHaehFEJPE8YmJwBFMXc1iSNfguZB1UqZ uZRtgjblNJTJFgOWsWah8W0e6QUN3WWHECAwTsf8g6NcFLiKuapNTsTuS8Gy6Z28kauU2Ass8Mhf1S9IGehwMDcOVI xbeaWarnNCSEU60slwBGR3I99tGjHzV153H7euj40BVd6Px0vhFCdqAyBGwkzma69dLZm1XH9whlDMhx8DVXFdi0b1 htn2aWJztZwhrugbQtgcl3qCF6pgFFFulNcHLVZMNLfQgDUF0js9hl6CUdN1fMlGmDB2B4D1iUuSmtaSe3CtJzQNre YUaY2CIWJTmojDRQlpDa2ExnlWnqVgAjlD7f06uNTGyGYdOS4ASuGNVxHjCtnic7NtZj7b2ggVfEFIJfwP8PKGi7Dt sWofnWmK1vFUKW41P0k2sW57urYG7bq0Kro5vK4M651r0gOy1NFn6rKhHRZiAqQ9GGZZwvzqW7yKufNn0tiRY0HJpR tax8AVB1SGQNGU28s7jO0uBAFCcBchWdKLbLYBsJimk8gbORHQoc6jz1HDFr1P8giMrOHSR8MS040F7hsnYirbayex IuxgWjfPjQLr2okAe8bR5PTIrm8RPol0LPtbV6rT8A2hvxiWUvEwyR9D6SBgSS0PLUqH6xluIMU2n7XJdKTeGqEd0Z AGjpQVJYrcLsFrqRT661H1jGopq3SuYsqhF2mcw3UBqy67IO2NgNYigqP4NJOPdM6Luw11tTtbk7JMxKIUyMoXSwYz WDZ6tgUWsnCfMgmVJT6iaIUOgfuHvGtsazUC6W37inHeUeGrfoCqM0pBcXLzP1V2Lojt1GBDqzn3Tvar9vRcrEXA7A Phm5Fv1V9eVlAm8YRkbKkiqpg4Ll5xTxULaHKFHp5j42yqz4G6hedo53qhpsnM7dfbklnmrKE63uM9H34FsaRUjQLi THB5c0My0B4LeeIyYyCQN5bKXUkfOiqV4zWqALgku2LRfZqsdqhBSoP5VEl0ho8C3t8VCqHV9I7YgzHuLg3S5RKGkn kYOTIXjeFvZMy2cooX26CLV1b4C3O5sci1qtXztr6k7JGAQgi4kUN6lCKyVlCw1jpAXarP56QlFLh4TqsAAAMz1xuz PUIkRbdoVfk92govM86u4XNzc3zofq0PX615kAR8VSctHkRupPxb9Wcyipu59kfjPFMNzGHFxBuUhIG0q2VnebhLHw 47ydYCL90f0MyTDz3vkNCiIxiqe9fLdZsd2IESDqPQniUjp0tLbPCl9bw6yGVjNcz0n0G2tWDiBe9KDJXnrx5eTcJ" } }, Version-Release number of selected component (if applicable): $ oc get pod rsyslog-jnmrp -o yaml |grep image image: docker.io/viaq/rsyslog:latest imagePullPolicy: IfNotPresent imagePullSecrets: image: docker.io/viaq/rsyslog:latest imageID: docker.io/viaq/rsyslog@sha256:fd5fafa8843029219729611f9b52be249ea9c05381acc88fbd5c050ef909227c $ oc rsh rsyslog-jnmrp sh-4.4# rpm -qa |grep rsyslog rsyslog-8.39.0-1.fc29.x86_64 rsyslog-relp-8.39.0-1.fc29.x86_64 rsyslog-gssapi-8.39.0-1.fc29.x86_64 rsyslog-mmkubernetes-8.39.0-1.fc29.x86_64 rsyslog-kafka-8.39.0-1.fc29.x86_64 rsyslog-elasticsearch-8.39.0-1.fc29.x86_64 rsyslog-mmjsonparse-8.39.0-1.fc29.x86_64 rsyslog-mmnormalize-8.39.0-1.fc29.x86_64 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-01-24-184525 True False 1h Cluster version is 4.0.0-0.nightly-2019-01-24-184525 How reproducible: Always Steps to Reproduce: 1. Deploy logging, set log collection type: rsyslog in clusterlogging object 2. create a project, then create app in the new project to generate some logs 3. check indices and logs in ES Actual results: Expected results: Additional info:
Please attach rsyslog pod log
I could not reproduce the problem. This is a similar search result. As you see in the pastebin, it contains kubernetes metadata. http://pastebin.test.redhat.com/702329 Only a difference is my cluster-logging-operator is built from the master branch with a small tweak (but it should not affect the metadata handling.) When are we having the next respin? Note: oc logs on my rsyslog pod is very quiet. INFO: Disabling Prometheus endpoint rsyslogd 8.39.0: running as pid 1, enabling container-specific defaults, press ctl-c to terminate rsyslog
Rsyslog pod logs: $ oc logs rsyslog-nhrst INFO: Disabling Prometheus endpoint rsyslogd 8.39.0: running as pid 1, enabling container-specific defaults, press ctl-c to terminate rsyslog rsyslogd: mmkubernetes: failed to connect to [https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/openshift-sdn] - 60:Peer certificate cannot be authenticated with given CA certificates [v8.39.0] rsyslogd: action 'action-5-mmkubernetes' (module 'mmkubernetes') message lost, could not be processed. Check for additional error messages before this one. [v8.39.0] rsyslogd: omelasticsearch: we are suspending ourselfs due to server failure 7: Failed to connect to elasticsearch port 9200: No route to host [v8.39.0 try http://www.rsyslog.com/e/2007 ]
Please provide the output of the following: oc get ds rsyslog -o yaml oc get cm rsyslog -o yaml then - get the rsyslog pod and oc rsh $rsyslog_pod from inside the rsyslog_pod ls -alrtF /var/run/secret/kubernetes.io/serviceaccount cat /var/run/secret/kubernetes.io/serviceaccount/ca.crt curl -vs --cacert /var/run/secret/kubernetes.io/serviceaccount/ca.cr https://kubernetes.default.svc.cluster.local:443
Created attachment 1524178 [details] Rsyslog info
(In reply to Rich Megginson from comment #4) > Please provide the output of the following: > > oc get ds rsyslog -o yaml > oc get cm rsyslog -o yaml > > then - get the rsyslog pod and oc rsh $rsyslog_pod > > from inside the rsyslog_pod > > ls -alrtF /var/run/secret/kubernetes.io/serviceaccount > cat /var/run/secret/kubernetes.io/serviceaccount/ca.crt Sorry - this is /var/run/secrets/kubernetes.io/serviceaccount and /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > > curl -vs --cacert /var/run/secret/kubernetes.io/serviceaccount/ca.cr > https://kubernetes.default.svc.cluster.local:443 Sorry - this should be curl -vs --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc.cluster.local:443
$ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-657c4f64bc-prspn 1/1 Running 0 2h elasticsearch-clientdatamaster-0-1-6cf8db77f4-qst69 1/1 Running 0 2h elasticsearch-clientdatamaster-0-2-588bcf7b7f-9v42c 1/1 Running 0 2h elasticsearch-operator-86cc7cb548-lslxf 1/1 Running 0 2h kibana-6c9d89bc58-kcthh 2/2 Running 0 2h rsyslog-49xzf 1/1 Running 0 2h rsyslog-92z6v 1/1 Running 0 2h rsyslog-rsf4t 1/1 Running 0 2h rsyslog-xkcpr 1/1 Running 0 2h rsyslog-z4ptj 1/1 Running 0 2h [qitang@192 aws]$ oc rsh ^C [qitang@192 aws]$ oc exec elasticsearch-clientdatamaster-0-1-6cf8db77f4-qst69 -- indices Mon Jan 28 12:35:12 UTC 2019 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .operations.2019.01.28 F4QKlCRtQXiqYM8jvwy62g 2 1 646622 0 1327 674 green open .kibana cRVcWJL8Q_i-0dFwZs7tHw 1 1 5 0 0 0 green open .searchguard G2pdQok8TuKzfYuiDmbQqQ 1 1 5 0 0 0 [qitang@192 aws]$ oc rsh rsyslog-49xzf sh-4.4# ls -alrtF /var/run/secrets/kubernetes.io/serviceaccount total 0 lrwxrwxrwx. 1 root root 12 Jan 28 10:06 token -> ..data/token lrwxrwxrwx. 1 root root 16 Jan 28 10:06 namespace -> ..data/namespace lrwxrwxrwx. 1 root root 13 Jan 28 10:06 ca.crt -> ..data/ca.crt lrwxrwxrwx. 1 root root 31 Jan 28 10:06 ..data -> ..2019_01_28_10_06_35.837725545/ drwxr-xr-x. 2 root root 100 Jan 28 10:06 ..2019_01_28_10_06_35.837725545/ drwxrwxrwt. 3 root root 140 Jan 28 10:06 ./ drwxr-xr-x. 3 root root 60 Jan 28 10:06 ../ sh-4.4# cat /var/run/secrets/kubernetes.io/serviceaccount/ ..2019_01_28_10_06_35.837725545/ ..data/ ca.crt namespace token sh-4.4# cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -----BEGIN CERTIFICATE----- MIIDMDCCAhigAwIBAgIIG208sgthFqkwDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE CxMJb3BlbnNoaWZ0MRAwDgYDVQQDEwdyb290LWNhMB4XDTE5MDEyODA2NTY0N1oX DTI5MDEyNTA2NTY0N1owJTERMA8GA1UECxMIYm9vdGt1YmUxEDAOBgNVBAMTB2t1 YmUtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQClzGxvtN/wB5FQ vQ4iQecJpWJJ+Z3pGJ6PhkJkXYLse4a1HIxq4NKaV4mBRXMg7w/HlQUlrVwOSNl0 XJ6PTKajuz2nnE1zW9F8ueBY5j5pvo8SNfOvp3BACjFcIZIa4623pnCPLwvMI9tA hyCAcPqbGhSZNtQ7R61pffuk2evMZWtbwKUBWRTKKNRNWsfdD88kzLy6KvIZvL8j Aare2Ut8yxjsAYCGLUh2dd0LoRgrhUM2iySzyTWQOf7qdvrabFjtxe03ZedQ14NN 2kwgGKAnndbOdc5eUsmlUP/ChPNJonWL5xWwz89sShihmkXNqOIUWubIgyONWign PH3937VvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/ MB0GA1UdDgQWBBQ5Yg4xGjy1Iqs7pXXiFWTKaEbhsjAfBgNVHSMEGDAWgBQ5Yg4x Gjy1Iqs7pXXiFWTKaEbhsjANBgkqhkiG9w0BAQsFAAOCAQEAiYDsCalFsi4uOSGg sVejnPL98WsPHrTFAzqj7dp9UzBJGcNKXhBhTf/zKASzNgDCAhT0XyMDUvlW2wed 3MOFBjh7yRjEB6SCWACu1RkjylmjT3utkOvQSIkZXeEkePD/hXxUQexKtdXJnKR1 mjTgm4UBllpXYFzqnpJub6Y/vvadTXwBoQr2XyV7ldNZKvS+rNMP1AihAKkY18xJ UqYHgmzdpMHD6gUFavB/FhhvVXJ47vBWvOfzetr4h7zG6m9y5YAstu+Da4Yu5g+g bRdCONkN4Yyg1MxYD4VAE9Cq88IJcrqPrJ2HyQBI4wFxFeDCMn73AWk3BqZwKdwO aFO7Tg== -----END CERTIFICATE----- sh-4.4# curl -vs --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc.cluster.local:443 * Rebuilt URL to: https://kubernetes.default.svc.cluster.local:443/ * Trying 172.30.0.1... * TCP_NODELAY set * Connected to kubernetes.default.svc.cluster.local (172.30.0.1) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: unable to get issuer certificate * Closing connection 0 sh-4.4#
The problem is that the ca cert is only the intermediate cert and does not contain the full CA cert chain. rsyslog requires the full CA cert chain all the way up to the root. Certificate: Data: Version: 3 (0x2) Serial Number: 1976302546878273193 (0x1b6d3cb20b6116a9) Signature Algorithm: sha256WithRSAEncryption Issuer: OU = openshift, CN = root-ca Validity Not Before: Jan 28 06:56:47 2019 GMT Not After : Jan 25 06:56:47 2029 GMT Subject: OU = bootkube, CN = kube-ca We are missing the CA cert for OU = openshift, CN = root-ca There was a previous bug for this, and I thought that OpenShift had fixed it by including the entire cert chain in the ca.crt file. What version of the openshift-installer did you use?
I tested it on the beta drop candidate build, it was: openshift-install v4.0.0-0.148.0.0-dirty , v0.11.0 oc version: v4.0.0-0.148.0 RHCOS:47.280 Payload:4.0.0-0.nightly-2019-01-25-205123 , 4.0.0-0.2
Also tested in 4.0.0-0.alpha-2019-01-29-032610, got the same result as comment 7.
This is a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1670282
(In reply to Rich Megginson from comment #11) > This is a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1670282 However - we can't seem to rely on getting a CA cert that we can use reliably in rsyslog - therefore, we need to address this in rsyslog. Design: The CLO will add some code to find the cluster Root CA cert, and concatenate it with the service CA cert (e.g. /var/run/secrets/kubernetes.io/serviceaccount/ca.crt or service-ca.crt). The CLO will add this to the secret used by rsyslog. rsyslog will use this CA cert instead of trusting the one from the service (e.g. /var/run/secrets/kubernetes.io/serviceaccount/ca.crt or service-ca.crt)
Moving to 4.2
Closing WONTFIX as this was resolved by the auth team. c#13 is an RFE for rsyslog