If the information is being gathered from the kubelet, then the SDN is not involved.
$ cat csr.json | jq '.items[] | {name:.metadata.name, message:.status.conditions[0].message, type:.status.conditions[0].type}' { "name": "csr-4cgnb", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-9qcsb", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } { "name": "csr-bhxg4", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } { "name": "csr-bm8bh", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-dh6lw", "message": null, "type": null } { "name": "csr-gkdbg", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } { "name": "csr-gtg65", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-hnbss", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } { "name": "csr-jq8jq", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-k54ds", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } { "name": "csr-lw5d6", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-rcpl4", "message": "This CSR was approved by the Node CSR Approver", "type": "Approved" } { "name": "csr-wzpgc", "message": "This CSR was approved by kubectl certificate approve.", "type": "Approved" } The one with no status { "apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": { "creationTimestamp": "2019-08-29T09:34:07Z", "generateName": "csr-", "name": "csr-dh6lw", "resourceVersion": "8313", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-dh6lw", "uid": "25d56045-ca40-11e9-b754-12e8f2535c10" }, "spec": { "groups": [ "system:nodes", "system:authenticated" ], "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlJUQ0I3QUlCQURCS01SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14TVRBdkJnTlZCQU1US0hONQpjM1JsYlRwdWIyUmxPbWx3TFRFd0xUQXRNVE15TFRFMk5pNWxZekl1YVc1MFpYSnVZV3d3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRL2xhVlJBR3V4V2d6N3htV3pxZnBOa0JEdDhuM3I5OEVlQ1V5Q0d4SGgKSExURHJDS3hMZm5HallWZTZtWk05Y2RDS1VCSnR3OTgxQ0tRL0dKaUswd09vRUF3UGdZSktvWklodmNOQVFrTwpNVEV3THpBdEJnTlZIUkVFSmpBa2doeHBjQzB4TUMwd0xURXpNaTB4TmpZdVpXTXlMbWx1ZEdWeWJtRnNod1FLCkFJU21NQW9HQ0NxR1NNNDlCQU1DQTBnQU1FVUNJUUQ4eEdiU3hKeFlwNW1maTZ3djdHOU50U3FUWGF5K0VySjgKTmxPbkRVbEl1Z0lnUURTVU9XV1g4RTMwUjY5REhOZk4yYXJsdFVWWVZWL3doa1NjQlNnVzAzRT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==", "usages": [ "digital signature", "key encipherment", "server auth" ], "username": "system:node:ip-10-0-132-166.ec2.internal" }, "status": {} }, ip-10-0-132-166 is the node where any tests the need to read logs, exec, etc are failing machine-approver is dropping CSRs. Ryan was saying that it is watching, but not listing, which might cause it to miss seeing new CSRs.
From the logs: ``` I0829 09:34:07.302930 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.310565 1 main.go:132] CSR csr-dh6lw not authorized: No target machine I0829 09:34:07.310588 1 main.go:164] Error syncing csr csr-dh6lw: No target machine I0829 09:34:07.315773 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.326631 1 main.go:132] CSR csr-dh6lw not authorized: No target machine I0829 09:34:07.326747 1 main.go:164] Error syncing csr csr-dh6lw: No target machine I0829 09:34:07.336960 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.343642 1 main.go:132] CSR csr-dh6lw not authorized: No target machine I0829 09:34:07.343663 1 main.go:164] Error syncing csr csr-dh6lw: No target machine I0829 09:34:07.363872 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.369072 1 main.go:132] CSR csr-dh6lw not authorized: No target machine I0829 09:34:07.369095 1 main.go:164] Error syncing csr csr-dh6lw: No target machine I0829 09:34:07.409320 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.415788 1 main.go:132] CSR csr-dh6lw not authorized: No target machine I0829 09:34:07.415815 1 main.go:164] Error syncing csr csr-dh6lw: No target machine I0829 09:34:07.496074 1 main.go:107] CSR csr-dh6lw added I0829 09:34:07.501979 1 main.go:132] CSR csr-dh6lw not authorized: No target machine E0829 09:34:07.502004 1 main.go:174] No target machine I0829 09:34:07.502013 1 main.go:175] Dropping CSR "csr-dh6lw" out of the queue: No target machine ``` The csr was dropped after 0.2s.
This is likely to be fixed by: https://bugzilla.redhat.com/show_bug.cgi?id=1746521
The following PR increases the number of retries before a csr is dropped. That should give enough time to avoid dropping any crs in less than 1s. https://github.com/openshift/cluster-machine-approver/pull/41
(In reply to Jan Chaloupka from comment #5) > The following PR increases the number of retries before a csr is dropped. > That should give enough time to avoid dropping any crs in less than 1s. > > https://github.com/openshift/cluster-machine-approver/pull/41 Related BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746521
Verified in 4.2.0-0.nightly-2019-09-08-180038 This has not been reproduced in the large scale test
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922