See https://github.com/openshift/installer/pull/3871 https://github.com/openshift/installer/pull/3974
Example from the no-op PR: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/3974/pull-ci-openshift-installer-master-e2e-aws-upi/1288496223979835392 Pasting the (not helpful but last) error message here to aid finding this BZ from search.ci: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state I think what we really need at this point is to extract more information (particularly the console) from the CF stack run or something like that?
This might be related to https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-aws-upi/1288243630917554176#1:build-log.txt%3A184 - discussion on Slack about `oc` and the kube rebase and failing to approve CSRs, but that job has different symptoms.
here is an example of AWS UPI job failing because oc cannot approve the CSR. > https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-aws-upi/1288538277200007168#1:build-log.txt%3A759 ``` no kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" in scheme "k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28" ``` > https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-aws-upi/1288538277200007168/artifacts/e2e-aws-upi/csr.json ``` $ for i in $(curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-aws-upi/1288538277200007168/artifacts/e2e-aws-upi/csr.json | jq -r '.items[] | select(.status == {}) | .spec.request'); do echo "$i" | base64 -d | openssl req -noout -text; done Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-67-141.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:95:2c:92:d1:2a:2b:04:d4:55:5b:58:5b:a5:a0: 4b:49:9b:69:20:68:b7:a9:03:cc:ca:9f:8f:38:cb: a7:ad:42:92:55:b3:3d:b7:c2:9e:7f:ad:70:3e:a6: 0a:46:3c:71:39:7c:be:b4:e6:f3:6e:30:dc:98:9d: a7:17:87:9e:07 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:45:7c:c3:87:f5:68:5d:da:e9:2a:0f:19:60:de: a3:28:1c:8d:ad:d1:66:cc:b8:77:12:56:11:6c:70:8e:99:22: 02:20:5a:75:c8:55:c9:35:73:72:87:9a:ea:8d:46:50:e8:7a: 5e:1e:8f:73:35:5c:72:d7:23:7e:e5:21:00:bf:7a:9f Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-67-141.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:6d:66:de:6e:5a:36:43:ed:04:83:66:44:29:65: 41:f6:49:35:a9:25:40:af:57:70:ef:0c:34:b4:cc: 7c:7b:e2:25:77:13:0a:a6:2f:84:e2:0a:12:b7:e3: bd:ed:08:22:5e:ef:4e:46:90:11:29:63:aa:8c:28: 39:22:c8:3f:2b ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:40:93:0d:ec:b5:09:e4:3f:21:0f:fc:ff:03:33: 0e:67:56:f2:98:9d:df:28:1a:c0:ce:f9:89:c4:49:d7:78:e1: 02:20:6c:3e:78:43:33:2b:91:97:21:d8:74:50:5e:51:b0:65: 10:69:82:6f:0a:b9:50:3d:6b:46:f2:11:20:78:38:30 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-86-165.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:19:a6:54:03:10:80:fb:19:da:00:82:19:1d:92: b9:8b:2e:8b:59:f0:7e:2d:82:f3:33:10:ad:b7:2d: c6:2d:06:94:23:41:c0:92:7d:fc:aa:21:06:e3:99: 83:9c:60:03:e5:ad:b4:0e:20:fc:e7:43:85:9f:e0: 67:11:b6:53:f9 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:46:02:21:00:ab:ca:d8:50:61:99:e1:29:b2:dd:d8:bc:a7: 51:38:cb:2f:bc:37:38:b9:cc:8c:73:ee:0e:1f:8d:cb:cc:32: 94:02:21:00:d4:d6:6a:35:19:e4:31:35:69:6c:a5:4f:a8:cc: 80:85:9a:10:3f:03:ee:fa:bb:af:5d:e4:17:75:ae:26:bf:1f Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-86-165.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:25:56:5a:57:8d:4c:09:14:10:db:95:f4:4d:19: 6e:36:94:86:22:85:b8:be:84:6c:13:12:a8:a4:df: eb:3c:85:04:85:cf:f6:48:7a:92:ad:98:27:e9:65: 18:bf:aa:ef:2c:e7:12:d1:17:40:1c:a1:6a:81:ce: 53:b8:08:48:87 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:19:f8:c3:36:f6:89:ea:c1:f9:30:40:a5:a4:b8: 62:6b:c8:eb:46:2b:3e:d8:d0:ae:d5:65:2c:a7:ad:78:a1:e7: 02:20:41:c5:30:7f:eb:3e:e4:09:bd:3c:50:24:f0:16:c9:e5: 56:7d:2c:2e:35:31:a8:cd:e6:77:fc:51:2d:a7:78:f8 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-51-79.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:4a:5e:ae:55:ec:c2:7f:7e:e5:c7:76:88:db:8a: ce:3c:a3:d1:b3:10:5b:55:42:2d:31:5b:de:fe:6a: 8c:43:49:0a:e7:cd:3f:79:f5:6c:df:9b:6c:b4:7f: 94:c5:2b:e0:f4:2d:30:d7:6c:25:b8:7c:ab:8d:fb: 99:55:48:d7:3a ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:46:02:21:00:aa:5c:e3:bb:5b:76:43:38:f1:a5:96:0e:61: e2:07:ec:41:91:4f:05:ae:a1:5c:d7:1b:07:ed:9f:6e:61:7b: d5:02:21:00:80:de:cb:9c:3f:57:a0:2e:eb:ce:c7:a6:14:7b: f5:97:4f:22:ba:8f:3e:e0:ed:92:0c:20:47:e0:aa:44:7a:9a Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:ip-10-0-51-79.ec2.internal Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:a2:2d:7f:7d:9f:1b:69:d0:88:aa:f1:9c:d0:29: 1a:cd:21:8f:f5:44:47:48:f1:e2:c5:bf:65:ea:2f: c2:a8:93:22:91:1e:8c:a6:87:0a:5c:4a:5f:5b:84: 22:52:f2:72:95:52:d6:c3:2c:64:c6:b0:d0:63:d0: ec:64:f4:2e:6f ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:45:02:21:00:ae:1f:14:eb:b4:ec:cf:a7:71:31:d9:3c:04: 08:08:72:0e:bd:8d:2a:ad:08:53:97:1b:8d:79:a9:ea:3b:5c: d0:02:20:52:1a:ec:70:a4:32:2f:1d:50:7e:da:a5:74:17:29: e2:fd:fd:78:e9:71:69:43:5d:47:bf:d4:3f:b2:fb:90:bf ```
here is an example of vSphere UPI job failing exactly due to the same reason. > https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-vsphere-upi/1288495552496930816 https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-vsphere-upi/1288495552496930816#1:build-log.txt%3A433 ``` error: no kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" in scheme "k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28" ``` ``` $ for i in $(curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/3871/pull-ci-openshift-installer-master-e2e-vsphere-upi/1288495552496930816/artifacts/e2e-vsphere-upi/gather-extra/csr.json | jq -r '.items[] | select(.status == {}) | .spec.request'); do echo "$i" | base64 -d | openssl req -noout -text; done Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-2 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:64:d3:40:ff:64:9e:de:dd:ff:01:37:00:f3:fb: 18:d9:3e:48:e7:f0:2e:cd:aa:86:e4:a6:53:23:fd: ee:d3:a1:aa:32:a3:0f:79:4a:ab:a3:52:0d:c4:33: ca:64:97:52:e7:1b:a6:a3:16:4d:0b:8d:c5:99:de: fc:07:39:1f:37 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:46:02:21:00:fe:76:8d:f3:7d:2f:9a:21:65:da:b5:aa:2a: aa:f3:82:52:e8:15:d1:a4:6c:f6:b5:ac:ce:2b:25:75:6e:19: 8b:02:21:00:f3:d0:cf:4f:04:7b:b0:59:29:37:e7:a4:f5:75: bc:92:03:c7:94:ce:ae:3b:2a:5d:bb:52:6c:09:6e:2e:a7:74 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-0 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:8a:c7:5a:3a:80:30:a7:b7:75:14:87:1e:78:18: c3:83:1d:48:19:8d:5b:b5:10:cb:de:f9:46:9b:35: a9:76:38:93:df:f1:42:87:06:18:04:d0:1f:e9:ed: 8d:ec:56:c9:47:f7:71:74:64:44:29:ae:db:dd:e4: ae:f7:30:25:97 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:02:60:a1:4b:e4:fe:2d:02:f6:ea:9a:91:af:92: aa:a6:f7:4c:9b:a0:f1:a8:6c:bc:ee:90:ea:c0:de:25:98:e7: 02:20:6f:dd:9c:e2:61:9d:b8:90:cc:25:77:45:9f:6b:f8:69: 6f:6c:b9:9a:75:27:67:66:12:49:76:4a:8c:a7:03:55 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-2 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:4d:e1:91:6c:ba:ac:e8:35:d6:c4:be:d8:78:05: cc:39:cf:24:1e:49:52:39:9e:49:6d:da:d3:d8:b0: 1f:93:ce:a9:1d:20:74:77:3f:30:71:d2:3b:4e:aa: 29:c0:58:98:a5:1d:cd:42:dc:ab:5e:e8:7c:ce:fd: 09:e6:30:d5:b1 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:12:db:78:a8:76:59:0e:c7:b7:c3:7b:e3:27:f2: 68:09:34:c8:df:4e:7e:f6:fc:cc:9d:7a:66:b2:4d:27:34:73: 02:20:76:77:b2:8a:83:82:7d:6e:30:a0:61:9a:ef:70:25:40: 42:f6:71:e1:12:4b:30:0a:e8:36:21:ea:5a:39:1f:41 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-2 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:f1:15:34:b9:ad:eb:ad:53:14:ef:46:40:4b:04: 22:72:91:c0:aa:5e:0a:7a:91:35:a5:71:a2:32:bb: 6d:02:03:df:12:93:c6:4e:5e:27:9a:f3:3f:5c:aa: b0:0e:88:bf:2b:a7:e5:24:cb:2b:fd:3f:81:59:27: 48:a0:bd:b6:89 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:7a:04:64:33:a8:04:33:f1:0e:b8:c2:64:17:a1: fa:a6:a0:de:c6:01:82:0a:f0:fd:e8:f0:08:71:7f:0b:79:31: 02:20:35:2f:c1:d4:ba:4b:26:6c:e2:8f:89:61:e2:f5:d6:92: a0:c5:66:cd:a9:33:96:4e:cf:2a:11:5a:75:1b:c4:66 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-1 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:e9:ea:91:90:ff:bb:bd:26:86:67:a4:3b:db:d4: 12:3f:61:bf:66:89:ce:29:81:a0:3f:b0:1f:7b:58: 14:36:ef:13:db:d0:a4:1e:08:07:bf:3c:2e:41:6f: f6:6d:11:4b:7e:18:b4:91:c5:43:ec:43:e2:25:56: 59:c6:09:ef:a3 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:46:02:21:00:f4:5f:2a:ca:a0:d5:f2:bf:c6:25:f9:13:0b: b9:3b:a8:01:3f:5d:70:3d:1d:68:1f:a2:4a:29:fa:f4:b5:7a: ac:02:21:00:a7:83:6b:95:65:29:b2:b6:5f:b6:85:9c:b4:8f: b3:ed:87:26:53:47:40:f4:4d:88:d5:3c:b5:ec:c5:9b:a7:2b Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-1 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:a8:30:c4:1e:b5:45:62:18:77:e1:61:4d:8b:75: 00:d2:cb:e8:62:9b:97:1a:27:ee:0f:95:b2:5a:b9: 75:2a:a5:7e:b0:5f:11:4f:42:3a:ac:a3:42:87:95: 9d:c9:b5:97:79:56:4e:65:fe:e7:8b:d0:d3:56:b5: 16:28:0b:9e:2e ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:45:02:20:4e:e1:3b:5f:81:09:b0:ae:26:fc:9e:fc:42:d1: a1:46:f6:1b:fe:8f:0d:84:88:b8:cc:43:3f:c7:46:4e:42:95: 02:21:00:a2:9b:be:dc:c7:7e:c1:e1:a9:32:b4:be:a0:38:95: a6:4d:95:e8:10:b9:99:ae:47:b9:3c:70:40:51:3e:8f:ad Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-0 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:b7:c6:31:cb:6d:50:1e:43:d1:d9:32:22:d1:82: 0e:6b:6b:76:0d:8c:ae:db:1f:27:6c:c3:1d:03:00: a4:7e:e1:e2:81:13:54:d7:b0:ec:f9:2b:f8:f9:8e: 69:38:82:fc:a7:08:b1:5b:4f:03:cf:f1:53:7a:40: ac:50:ab:d7:7c ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:44:02:20:78:0c:ff:6e:99:4c:21:e2:b1:c8:e7:23:6f:7c: 7b:21:df:ca:6f:6c:88:ee:6c:64:11:75:5e:eb:51:43:77:41: 02:20:4c:c8:1d:3b:08:19:bc:cb:4f:d9:39:4f:dc:fe:73:10: 79:9f:24:d6:86:7d:7f:7a:44:8b:ac:d9:1c:31:a4:81 Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-1 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:7b:c6:06:15:7b:af:09:51:a3:11:75:92:c6:7b: ec:ed:04:24:dd:d5:5d:29:d8:a7:7b:70:80:2c:13: a0:26:8c:d1:7b:e4:29:ca:08:e8:4e:7a:5d:e4:29: 93:8d:c8:b1:89:c0:79:51:93:5e:ae:a8:37:a4:3c: 50:30:1c:14:c7 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:46:02:21:00:a9:00:69:33:07:60:78:78:87:16:c3:22:e1: b9:64:a5:0b:d6:fb:83:6b:d3:50:b5:58:ad:60:e8:43:65:6b: a5:02:21:00:ed:3f:7d:40:ef:a1:2c:0d:23:49:6a:17:67:46: dd:40:9c:e6:ce:a6:d5:19:b2:46:08:b2:10:81:4a:89:6b:cd Certificate Request: Data: Version: 1 (0x0) Subject: O = system:nodes, CN = system:node:compute-0 Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:44:b1:c3:75:01:03:af:40:8a:29:30:e7:d6:3f: 55:cb:84:e3:77:b7:7f:4d:99:d1:8e:06:4c:eb:61: 42:86:70:af:94:d3:b6:f7:3b:80:e1:c3:52:0e:5d: 72:a7:97:c1:81:2d:fb:14:e0:f6:dc:ee:46:0b:88: 77:d8:09:df:0b ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: a0:00 Signature Algorithm: ecdsa-with-SHA256 30:45:02:21:00:a8:89:b9:df:2f:ef:fb:5b:bf:a2:4b:82:f8: ce:b0:f8:a2:76:b4:ce:61:f6:02:6d:16:50:bb:99:bd:e6:43: 16:02:20:46:4c:ef:ec:b2:52:49:70:fb:b5:35:d2:47:d1:de: ec:c5:6c:ee:07:90:0e:e7:57:ba:a3:0e:4a:87:ea:49:4b ``` shows that the compute is booting and sending CSR but those are not getting approved.
UPI on GCP also has the issue. Adding testblocker.
UPI on vsphere also met this issue. NAME AGE SIGNERNAME REQUESTOR CONDITION csr-bdph8 6m25s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending csr-j9q7x 16m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-mhfgn 6m25s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending csr-p75jc 16m kubernetes.io/kubelet-serving system:node:control-plane-1 Approved,Issued csr-qlp8k 16m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-s74tv 16m kubernetes.io/kubelet-serving system:node:control-plane-2 Approved,Issued csr-wzcnm 16m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-z6skx 16m kubernetes.io/kubelet-serving system:node:control-plane-0 Approved,Issued Trying to approve CSR requests for nodes...... No resources found no kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28" no kind "CertificateSigningRequest" is registered for version "certificates.k8s.io/v1" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28" No resources found
We were hoping this would be fixed by https://github.com/kubernetes/kubernetes/pull/91691 which was pulled into the oc 1.19-rc.1 bump (https://github.com/openshift/oc/pull/491) but it seems that it wasn't. Looking more into this
Mike, is this a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1860789 ? I'm not sure if these jobs are using 4.5 or 4.6 oc.
That is a good question, Scott. I would also like to find out if these jobs are using 4.5 or 4.6. There was a 1.18 fix that was backported into 4.5 (https://github.com/openshift/kubernetes-kubectl/pull/26) and a workaround until oc 1.19/4.6 were available (https://github.com/openshift/kubernetes/commit/444618c8a5e8c2a8f17e2c111c4ee9c1875ef72e) in order to land the rebase. However reverting that workaround still doesn't work post oc 1.19 (see https://github.com/openshift/kubernetes/pull/298). It seems like the fix from 1.19 isn't working
*** Bug 1866316 has been marked as a duplicate of this bug. ***
*** This bug has been marked as a duplicate of bug 1860789 ***