Description of problem: Workers nodes CSRs are not automatically approved, it stuck on Pending status and increase CRSs number each couple of mins Version-Release number of the following components: 4.3.0-0.nightly-2020-02-21-091838-ipv6.3 How reproducible: constantly Steps to Reproduce: 1. Deploy 4.3 baremetal ipv6 2. wait until successful deployment 3. verify all nodes (including workers) in Ready state 4. verify csrs status: oc -n openshift-machine-api get csr Actual results: Depend on how long time passed since deployment, there are a lot of worker nodes csrs in Pending status Expected results: Approved,Issued status for all csrs Additional info: Please attach logs from ansible-playbook with the -vvv flag
After few hours of running: [kni@provisionhost-0 ~]$ oc -n openshift-machine-api get csr NAME AGE REQUESTOR CONDITION csr-28qvl 178m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-2kx7j 9h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-2sfdp 11h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-48bnt 7h36m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-4fs2j 13h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-4lwqf 11h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-4t6sp 8h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-4vxwc 70m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-4zzc4 7h36m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5fw78 11h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5gh92 11h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5ltwb 12h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5lwxw 13h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5mbff 8m25s system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-5nc8g 3h29m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-66gmw 7h36m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6dcjq 6h3m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6ft2s 6h18m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6h97p 4h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6l4x4 5h2m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6stmt 12h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-6t2cj 147m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7ccjs 12h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7lb94 12h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7nkpz 3h59m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7pfxq 146m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7sbwz 39m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7v5z5 6h19m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-7vppv 100m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-82ctp 9h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8cn24 11h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8fxz7 8h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8ggk4 5h2m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8gmc2 8h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8hqh6 8h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8ptzk 10h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8qm99 54m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-8s7qw 12h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-92qkz 3h44m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-99tqf 6h50m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-9qcx2 7h5m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-9r762 10h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-bkzz2 116m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-bt9zc 10h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-bvdj2 162m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-bxs7h 116m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-ccrw6 9h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-ccxfm 9h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-cj5nm 10h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-clxbl 11h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-crffc 13h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-csxvp 5h16m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-d9kh8 13h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-ds5k5 11h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-dzwjm 162m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-g488f 101m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-g6sq7 23m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-gbnrk 7h52m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-gczkt 3h13m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-gfm9d 3h28m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-h56rp 39m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-h5dsb 9h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-h6vqp 11h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-h6wt4 11h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hdlx7 7m29s system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hfgkp 54m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hg7rj 22m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hk98z 8h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hm8jx 100m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hr956 4h45m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-hrz9k 3h44m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-j9g85 10h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jd66k 11h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jfgz6 5h33m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jkmz9 7h6m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jpcqp 11h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jr8mg 8h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jrc4v 13h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-jxlcr 85m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-k5jpr 3h44m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-k7x9r 131m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-k8gbc 8h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-kbshf 4h31m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-khwb9 6h34m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-kmzdr 3h13m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-l8fs2 9h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-lk5bc 5h1m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-ltwsb 6h3m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-lvpz5 4h15m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-m4hbb 3h29m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-m5wdb 5h17m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-m6jn8 8h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-mhqcr 84m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-mnznh 85m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-mx56n 132m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-n6gz2 4h31m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nkb7c 9h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nv8cd 5h17m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nwpmh 5h48m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nx6d5 12h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nxbxg 8m8s system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-nzwts 6h19m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-p496v 6h49m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-p5ksm 10h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pdzpb 23m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pg52x 69m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pkbkd 8h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-plpf5 10h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pm7dm 13h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pq5xw 5h48m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-pzqgd 6h35m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-q5k5w 53m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-q99wd 14h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-q9spl 4h14m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-qn7hx 7h52m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-qs62f 7h21m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-qxr9v 70m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-r5jrl 178m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-rfv6g 5h47m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-rggbg 4h15m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-rgxbq 13h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-rlzmt 10h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-rzgsv 8h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-s2cpq 131m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sdb4k 4h46m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sg6p7 7h51m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sg8q7 11h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-shc8t 4h46m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sxdrl 13h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sxwzc 163m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sz4ws 7h21m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-sz776 13h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-t7g79 12h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-t98kg 8h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-tb55v 8h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-tb8zc 7h5m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-tk74d 9h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-txhh5 38m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-v4m5g 10h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-v74dr 3h14m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-vck8b 13h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-vcngm 12h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-vdm74 14h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-vrklw 5h32m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-vskrq 9h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wgglm 5h32m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wlt6c 14h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wmf9x 6h50m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wq8xt 10h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wt67f 13h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-wz7cb 9h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-x8t2g 12h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-xbkkj 177m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-xmbvt 13h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-xnw9l 115m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-xpztc 10h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-xsvz5 7h20m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-z5sfr 4h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-z7b66 147m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-zb8w5 6h34m system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-zdpww 10h system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-zmxdb 4h30m system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com Pending csr-znksq 9h system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com Pending csr-zrzxz 9h system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending csr-zwkbx 6h4m system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com Pending [kni@provisionhost-0 ~]$ oc get nodes NAME STATUS ROLES AGE VERSION master-0.ocp-edge-cluster.qe.lab.redhat.com Ready master 14h v1.16.2 master-1.ocp-edge-cluster.qe.lab.redhat.com Ready master 14h v1.16.2 master-2.ocp-edge-cluster.qe.lab.redhat.com Ready master 14h v1.16.2 worker-0.ocp-edge-cluster.qe.lab.redhat.com Ready worker 14h v1.16.2 worker-1.ocp-edge-cluster.qe.lab.redhat.com Ready worker 14h v1.16.2 worker-2.ocp-edge-cluster.qe.lab.redhat.com Ready worker 14h v1.16.2
There is WA exist: [kni@provisionhost-0 ~]$ for csr in $(oc -n openshift-machine-api get csr | awk '/Pending/ {print $1}'); do oc adm certificate approve $csr;done
I believe this may be related due to machines not being assigned to nodes: [kni@provisionhost-0 ~]$ oc -n openshift-machine-api get machines NAME PHASE TYPE REGION ZONE AGE ocp-edge-cluster-master-0 12h ocp-edge-cluster-master-1 12h ocp-edge-cluster-master-2 12h ocp-edge-cluster-worker-0-787g6 12h ocp-edge-cluster-worker-0-b8vv6 12h
Do you still see this same behavior on 4.3.z w/ IPv6?
yes, and for 4.4 as well
I think I found the root cause for this https://github.com/metal3-io/baremetal-operator/pull/457 proposed upstream, I'll propose it to the openshift fork after doing some testing.
https://github.com/openshift/baremetal-operator/pull/52 merged and so should be in recent 4.5 nightly builds, setting to MODIFIED so we can get this QA verified for the 4.4/4.3 backports
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409