Bug 1807854 - [Edge][ipv6] Workers nodes CSRs are not automatically approved
Summary: [Edge][ipv6] Workers nodes CSRs are not automatically approved
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Beth White
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: 1771572 1816121
TreeView+ depends on / blocked
 
Reported: 2020-02-27 11:23 UTC by Elena German
Modified: 2020-07-13 17:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1816121 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:21:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift baremetal-operator pull 52 0 None closed Bug 1807854 : Don't assume IPv4 in introspection data 2020-11-23 21:55:37 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:22:11 UTC

Description Elena German 2020-02-27 11:23:33 UTC
Description of problem:
Workers nodes CSRs are not automatically approved, it stuck on Pending status and increase CRSs number each couple of mins

Version-Release number of the following components:
4.3.0-0.nightly-2020-02-21-091838-ipv6.3

How reproducible:
constantly

Steps to Reproduce:
1. Deploy 4.3 baremetal ipv6
2. wait until successful deployment
3. verify all nodes (including workers) in Ready state
4. verify csrs status: oc -n openshift-machine-api get csr

Actual results:
Depend on how long time passed since deployment, there are a lot of worker nodes csrs in Pending status

Expected results:
Approved,Issued status for all csrs

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Elena German 2020-02-27 11:25:49 UTC
After few hours of running:
[kni@provisionhost-0 ~]$ oc -n openshift-machine-api get csr
NAME        AGE     REQUESTOR                                                 CONDITION
csr-28qvl   178m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-2kx7j   9h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-2sfdp   11h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-48bnt   7h36m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-4fs2j   13h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-4lwqf   11h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-4t6sp   8h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-4vxwc   70m     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-4zzc4   7h36m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5fw78   11h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5gh92   11h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5ltwb   12h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5lwxw   13h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5mbff   8m25s   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-5nc8g   3h29m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-66gmw   7h36m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6dcjq   6h3m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6ft2s   6h18m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6h97p   4h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6l4x4   5h2m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6stmt   12h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-6t2cj   147m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7ccjs   12h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7lb94   12h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7nkpz   3h59m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7pfxq   146m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7sbwz   39m     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7v5z5   6h19m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-7vppv   100m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-82ctp   9h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8cn24   11h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8fxz7   8h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8ggk4   5h2m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8gmc2   8h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8hqh6   8h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8ptzk   10h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8qm99   54m     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-8s7qw   12h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-92qkz   3h44m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-99tqf   6h50m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-9qcx2   7h5m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-9r762   10h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-bkzz2   116m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-bt9zc   10h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-bvdj2   162m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-bxs7h   116m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-ccrw6   9h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-ccxfm   9h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-cj5nm   10h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-clxbl   11h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-crffc   13h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-csxvp   5h16m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-d9kh8   13h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-ds5k5   11h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-dzwjm   162m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-g488f   101m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-g6sq7   23m     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-gbnrk   7h52m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-gczkt   3h13m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-gfm9d   3h28m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-h56rp   39m     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-h5dsb   9h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-h6vqp   11h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-h6wt4   11h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hdlx7   7m29s   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hfgkp   54m     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hg7rj   22m     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hk98z   8h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hm8jx   100m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hr956   4h45m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-hrz9k   3h44m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-j9g85   10h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jd66k   11h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jfgz6   5h33m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jkmz9   7h6m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jpcqp   11h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jr8mg   8h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jrc4v   13h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-jxlcr   85m     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-k5jpr   3h44m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-k7x9r   131m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-k8gbc   8h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-kbshf   4h31m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-khwb9   6h34m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-kmzdr   3h13m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-l8fs2   9h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-lk5bc   5h1m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-ltwsb   6h3m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-lvpz5   4h15m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-m4hbb   3h29m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-m5wdb   5h17m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-m6jn8   8h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-mhqcr   84m     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-mnznh   85m     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-mx56n   132m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-n6gz2   4h31m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nkb7c   9h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nv8cd   5h17m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nwpmh   5h48m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nx6d5   12h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nxbxg   8m8s    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-nzwts   6h19m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-p496v   6h49m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-p5ksm   10h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pdzpb   23m     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pg52x   69m     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pkbkd   8h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-plpf5   10h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pm7dm   13h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pq5xw   5h48m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-pzqgd   6h35m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-q5k5w   53m     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-q99wd   14h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-q9spl   4h14m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-qn7hx   7h52m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-qs62f   7h21m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-qxr9v   70m     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-r5jrl   178m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-rfv6g   5h47m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-rggbg   4h15m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-rgxbq   13h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-rlzmt   10h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-rzgsv   8h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-s2cpq   131m    system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sdb4k   4h46m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sg6p7   7h51m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sg8q7   11h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-shc8t   4h46m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sxdrl   13h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sxwzc   163m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sz4ws   7h21m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-sz776   13h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-t7g79   12h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-t98kg   8h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-tb55v   8h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-tb8zc   7h5m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-tk74d   9h      system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-txhh5   38m     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-v4m5g   10h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-v74dr   3h14m   system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-vck8b   13h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-vcngm   12h     system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-vdm74   14h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-vrklw   5h32m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-vskrq   9h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wgglm   5h32m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wlt6c   14h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wmf9x   6h50m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wq8xt   10h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wt67f   13h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-wz7cb   9h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-x8t2g   12h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-xbkkj   177m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-xmbvt   13h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-xnw9l   115m    system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-xpztc   10h     system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-xsvz5   7h20m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-z5sfr   4h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-z7b66   147m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-zb8w5   6h34m   system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-zdpww   10h     system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-zmxdb   4h30m   system:node:worker-2.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-znksq   9h      system:node:worker-0.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-zrzxz   9h      system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
csr-zwkbx   6h4m    system:node:worker-1.ocp-edge-cluster.qe.lab.redhat.com   Pending
[kni@provisionhost-0 ~]$ oc get nodes
NAME                                          STATUS   ROLES    AGE   VERSION
master-0.ocp-edge-cluster.qe.lab.redhat.com   Ready    master   14h   v1.16.2
master-1.ocp-edge-cluster.qe.lab.redhat.com   Ready    master   14h   v1.16.2
master-2.ocp-edge-cluster.qe.lab.redhat.com   Ready    master   14h   v1.16.2
worker-0.ocp-edge-cluster.qe.lab.redhat.com   Ready    worker   14h   v1.16.2
worker-1.ocp-edge-cluster.qe.lab.redhat.com   Ready    worker   14h   v1.16.2
worker-2.ocp-edge-cluster.qe.lab.redhat.com   Ready    worker   14h   v1.16.2

Comment 2 Elena German 2020-02-27 11:28:13 UTC
There is WA exist: [kni@provisionhost-0 ~]$ for csr in $(oc -n openshift-machine-api get csr | awk '/Pending/ {print $1}'); do oc adm certificate approve $csr;done

Comment 3 Marius Cornea 2020-02-27 15:47:53 UTC
I believe this may be related due to machines not being assigned to nodes:

[kni@provisionhost-0 ~]$ oc -n openshift-machine-api get machines
NAME                              PHASE   TYPE   REGION   ZONE   AGE
ocp-edge-cluster-master-0                                        12h
ocp-edge-cluster-master-1                                        12h
ocp-edge-cluster-master-2                                        12h
ocp-edge-cluster-worker-0-787g6                                  12h
ocp-edge-cluster-worker-0-b8vv6                                  12h

Comment 4 Stephen Benjamin 2020-03-19 17:25:32 UTC
Do you still see this same behavior on 4.3.z w/ IPv6?

Comment 5 Elena German 2020-03-20 03:21:33 UTC
yes, and for 4.4 as well

Comment 6 Steven Hardy 2020-03-20 16:07:49 UTC
I think I found the root cause for this https://github.com/metal3-io/baremetal-operator/pull/457 proposed upstream, I'll propose it to the openshift fork after doing some testing.

Comment 7 Steven Hardy 2020-03-23 10:10:48 UTC
https://github.com/openshift/baremetal-operator/pull/52 merged and so should be in recent 4.5 nightly builds, setting to MODIFIED so we can get this QA verified for the 4.4/4.3 backports

Comment 11 errata-xmlrpc 2020-07-13 17:21:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.