Description of problem: Version-Release number of the following components: $ rpm -q openshift-ansible openshift-ansible-3.11.157-1.git.0.10b76ed.el7.noarch $ rpm -q ansible ansible-2.6.20-1.el7ae.noarch $ ansible --version ansible 2.6.20 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/ec2-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] How reproducible: Always Steps to Reproduce: 1. ansible-playbook -vvv /usr/share/ansible/openshift-ansible/playbooks/openshift-master/redeploy-openshift-ca.yml 2. ansible-playbook -vvv /usr/share/ansible/openshift-ansible/playbooks/redeploy-certificates.yml -e openshift_redeploy_openshift_ca=true 3. Actual results: The /etc/origin/node/node.kubeconfig is not updated with the updated current kubelet client cert. The sync pod uses this file to get node ConfigMap and failed with "error: You must be logged in to the server (Unauthorized)" forever due to old client cert. The /etc/origin/node/certificates/kubelet-client-current.pem is updated so the kubelet/atomic-openshift-node service itself is fine. Expected results: The /etc/origin/node/node.kubeconfig is updated, or sync pod needs to use the kubelet-client-current.pem file instead. Additional info:
This bug leads sync pod CrashLoop in some cases, and other playbook failures because of sync pod rollout healthchecks. After the redeploy: $ oc logs sync-dsmn2 -n openshift-node error: You must be logged in to the server (Unauthorized) $ sudo oc --config=/etc/origin/node/node.kubeconfig whoami error: You must be logged in to the server (Unauthorized) $ sudo oc --config=/etc/origin/node/bootstrap.kubeconfig whoami system:serviceaccount:openshift-infra:node-bootstrapper The workaround is to renew the .kubeconfig files manually: Manually recreate OpenShift Node TLS bootstrapped certificates and kubeconfig files. https://access.redhat.com/solutions/3782361
https://github.com/openshift/openshift-ansible/pull/12119 is proposed fix for https://bugzilla.redhat.com/show_bug.cgi?id=1808068 I think this may be a dupe but I haven't looked deeply.
I don't think this is a duplicate. Reproduced the error on the current code.
To be reviewed as part of https://issues.redhat.com/browse/CORS-1470
Jira issue https://issues.redhat.com/browse/CORS-1470 was not scheduled for the current sprint.
Possible dup of https://bugzilla.redhat.com/show_bug.cgi?id=1772580
Not a dup of Bug 1772580 as proposed in comment # 15. 1772580 is related to bootstrap kubeconfig and this is not.
Checked and met the expection, so move to verified. openshift-ansible-3.11.306-1.git.0.75ef178.el7.noarch openshift-ansible-roles-3.11.306-1.git.0.75ef178.el7.noarch openshift-ansible-docs-3.11.306-1.git.0.75ef178.el7.noarch openshift-ansible-playbooks-3.11.306-1.git.0.75ef178.el7.noarch ==============> Before redeploy: [root@wj311osp1009cmaster-etcd-nfs-1 ~]# ls /etc/origin/node/certificates/ -al total 12 drwxr-xr-x. 2 root root 212 Oct 9 05:29 . drwx------. 5 root root 198 Oct 9 05:28 .. -rw-------. 1 root root 2826 Oct 9 05:24 kubelet-client-2020-10-09-05-24-18.pem -rw-------. 1 root root 1167 Oct 9 05:25 kubelet-client-2020-10-09-05-25-21.pem lrwxrwxrwx. 1 root root 68 Oct 9 05:25 kubelet-client-current.pem -> /etc/origin/node/certificates/kubelet-client-2020-10-09-05-25-21.pem -rw-------. 1 root root 1285 Oct 9 05:29 kubelet-server-2020-10-09-05-29-52.pem lrwxrwxrwx. 1 root root 68 Oct 9 05:29 kubelet-server-current.pem -> /etc/origin/node/certificates/kubelet-server-2020-10-09-05-29-52.pem [root@wj311osp1009cmaster-etcd-nfs-1 ~]# ls /etc/origin/node/node.kubeconfig -al -rw-------. 1 root root 7668 Oct 9 05:22 /etc/origin/node/node.kubeconfig ==============> redeploy: #ansible-playbook -i ~/qe-inventory-host-file -vvv ./playbooks/openshift-master/redeploy-openshift-ca.yml ...... PLAY RECAP *********************************************************************************************************************************************************************************************************************************** ci-vm-10-0-149-112.hosted.upshift.rdu2.redhat.com : ok=21 changed=7 unreachable=0 failed=0 skipped=38 rescued=0 ignored=0 ci-vm-10-0-149-23.hosted.upshift.rdu2.redhat.com : ok=21 changed=7 unreachable=0 failed=0 skipped=38 rescued=0 ignored=0 ci-vm-10-0-150-163.hosted.upshift.rdu2.redhat.com : ok=98 changed=27 unreachable=0 failed=0 skipped=79 rescued=0 ignored=0 localhost : ok=17 changed=0 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 INSTALLER STATUS ***************************************************************************************************************************************************************************************************************************** Initialization : Complete (0:00:22) Friday 09 October 2020 06:07:55 -0400 (0:00:00.512) 0:04:10.273 ******** =============================================================================== Restart docker ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 26.13s /usr/share/ansible/openshift-ansible/playbooks/openshift-node/private/restart.yml:47 -------------------------------------------------------------------------------------------------------------------------------------------------------- openshift_control_plane : verify API server ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 18.56s /usr/share/ansible/openshift-ansible/roles/openshift_control_plane/handlers/main.yml:13 ----------------------------------------------------------------------------------------------------------------------------------------------------- openshift_ca : Install the base package for admin tooling ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.03s /usr/share/ansible/openshift-ansible/roles/openshift_ca/tasks/main.yml:6 -------------------------------------------------------------------------------------------------------------------------------------------------------------------- openshift_control_plane : restart master ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.83s /usr/share/ansible/openshift-ansible/roles/openshift_control_plane/tasks/restart.yml:2 ------------------------------------------------------------------------------------------------------------------------------------------------------ openshift_master_facts : Set master facts --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.77s /usr/share/ansible/openshift-ansible/roles/openshift_master_facts/tasks/main.yml:2 ---------------------------------------------------------------------------------------------------------------------------------------------------------- Gather Cluster facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.73s /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:27 -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Initialize openshift.node.sdn_mtu ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.42s /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:61 -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Deploy CA certificate, key, bundle and serial ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.24s /usr/share/ansible/openshift-ansible/playbooks/openshift-master/private/redeploy-openshift-ca.yml:153 --------------------------------------------------------------------------------------------------------------------------------------- openshift_ca : Create the master certificates if they do not already exist ------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.09s /usr/share/ansible/openshift-ansible/roles/openshift_ca/tasks/main.yml:131 ------------------------------------------------------------------------------------------------------------------------------------------------------------------ restart node -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.83s /usr/share/ansible/openshift-ansible/playbooks/openshift-node/private/restart.yml:68 -------------------------------------------------------------------------------------------------------------------------------------------------------- openshift_named_certificates : openshift_facts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.68s /usr/share/ansible/openshift-ansible/roles/openshift_named_certificates/tasks/main.yml:8 ---------------------------------------------------------------------------------------------------------------------------------------------------- update ca trust ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.90s /usr/share/ansible/openshift-ansible/playbooks/openshift-master/private/redeploy-openshift-ca.yml:244 --------------------------------------------------------------------------------------------------------------------------------------- Retrieve CA certificate, key, bundle and serial --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.58s /usr/share/ansible/openshift-ansible/playbooks/openshift-master/private/redeploy-openshift-ca.yml:131 --------------------------------------------------------------------------------------------------------------------------------------- Gathering Facts ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.51s /usr/share/ansible/openshift-ansible/playbooks/init/basic_facts.yml:2 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy current OpenShift CA to legacy directory ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.49s /usr/share/ansible/openshift-ansible/playbooks/openshift-master/private/redeploy-openshift-ca.yml:76 ---------------------------------------------------------------------------------------------------------------------------------------- Gathering Facts ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.30s /usr/share/ansible/openshift-ansible/playbooks/init/basic_facts.yml:7 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Wait for node to be ready ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.19s /usr/share/ansible/openshift-ansible/playbooks/openshift-node/private/restart.yml:73 -------------------------------------------------------------------------------------------------------------------------------------------------------- openshift_ca : Copy generated loopback master client config to master config dir ------------------------------------------------------------------------------------------------------------------------------------------------------ 1.13s /usr/share/ansible/openshift-ansible/roles/openshift_ca/tasks/main.yml:256 ------------------------------------------------------------------------------------------------------------------------------------------------------------------ get openshift_current_version --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.09s /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:10 -------------------------------------------------------------------------------------------------------------------------------------------------------------------- copy ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.06s /usr/share/ansible/openshift-ansible/playbooks/openshift-master/private/redeploy-openshift-ca.yml:223 --------------------------------------------------------------------------------------------------------------------------------------- # ansible-playbook -i ~/qe-inventory-host-file -vvv /usr/share/ansible/openshift-ansible/playbooks/redeploy-certificates.yml -e openshift_redeploy_openshift_ca=true PLAY RECAP *********************************************************************************************************************************************************************************************************************************** ci-vm-10-0-149-112.hosted.upshift.rdu2.redhat.com : ok=26 changed=6 unreachable=0 failed=0 skipped=90 rescued=0 ignored=0 ci-vm-10-0-149-23.hosted.upshift.rdu2.redhat.com : ok=26 changed=6 unreachable=0 failed=0 skipped=90 rescued=0 ignored=0 ci-vm-10-0-150-163.hosted.upshift.rdu2.redhat.com : ok=390 changed=129 unreachable=0 failed=0 skipped=289 rescued=0 ignored=0 localhost : ok=16 changed=0 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 INSTALLER STATUS ***************************************************************************************************************************************************************************************************************************** Initialization : Complete (0:00:24) Node Join : Complete (0:00:27) Friday 09 October 2020 06:19:40 -0400 (0:00:43.052) 0:10:04.680 ******** =============================================================================== openshift_service_catalog : Verify that the controller-manager is running ------------------------------------------------------------------------------------------------------------------------------------------------------------ 43.15s /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/restart_pods.yml:40 ---------------------------------------------------------------------------------------------------------------------------------------------- template_service_broker : Verify that the apiserver is running ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43.05s /usr/share/ansible/openshift-ansible/roles/template_service_broker/tasks/restart_pods.yml:9 ------------------------------------------------------------------------------------------------------------------------------------------------- openshift_service_catalog : Verify that the apiserver is running --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43.04s /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/restart_pods.yml:19 ---------------------------------------------------------------------------------------------------------------------------------------------- openshift_console : Waiting for console rollout to complete -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 36.23s /usr/share/ansible/openshift-ansible/roles/openshift_console/tasks/start.yml:2 -------------------------------------------------------------------------------------------------------------------------------------------------------------- ansible_service_broker : Verify that the ASB is running ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 32.49s /usr/share/ansible/openshift-ansible/roles/ansible_service_broker/tasks/restart_pods.yml:20 ------------------------------------------------------------------------------------------------------------------------------------------------- openshift_control_plane : verify API server ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 22.85s /usr/share/ansible/openshift-ansible/roles/openshift_control_plane/handlers/main.yml:13 ----------------------------------------------------------------------------------------------------------------------------------------------------- openshift_control_plane : verify API server ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 19.65s /usr/share/ansible/openshift-ansible/roles/openshift_control_plane/handlers/main.yml:13 ----------------------------------------------------------------------------------------------------------------------------------------------------- etcd : restart etcd ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 11.59s /usr/share/ansible/openshift-ansible/roles/etcd/tasks/restart.yml:2 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Verify that the Alertmanager is running ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.33s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:50 ------------------------------------------------------------------------------------------------------------------------------------ Verify that the Prometheus is running ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 11.20s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:29 ------------------------------------------------------------------------------------------------------------------------------------ Verify that the Grafana is running --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.15s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:92 ------------------------------------------------------------------------------------------------------------------------------------ Remove node-exporter pods ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9.56s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:106 ----------------------------------------------------------------------------------------------------------------------------------- openshift_master_facts : Set master facts --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.62s /usr/share/ansible/openshift-ansible/roles/openshift_master_facts/tasks/main.yml:2 ---------------------------------------------------------------------------------------------------------------------------------------------------------- Remove web console pods --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.23s /usr/share/ansible/openshift-ansible/playbooks/openshift-web-console/private/redeploy-certificates.yml:16 ----------------------------------------------------------------------------------------------------------------------------------- Remove grafana pods ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7.66s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:85 ------------------------------------------------------------------------------------------------------------------------------------ openshift_service_catalog : Remove controller-manager pods ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.95s /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/restart_pods.yml:33 ---------------------------------------------------------------------------------------------------------------------------------------------- template_service_broker : Remove apiserver pods --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.70s /usr/share/ansible/openshift-ansible/roles/template_service_broker/tasks/restart_pods.yml:2 ------------------------------------------------------------------------------------------------------------------------------------------------- Gather Cluster facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 6.20s /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:27 -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Approve node certificates when bootstrapping ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 6.18s /usr/share/ansible/openshift-ansible/playbooks/openshift-node/private/join.yml:43 ----------------------------------------------------------------------------------------------------------------------------------------------------------- Remove kube-state-metrics pods -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.69s /usr/share/ansible/openshift-ansible/playbooks/openshift-monitoring/private/redeploy-certificates.yml:64 ------------------------------------------------------------------------------------------------------------------------------------ =================> After redeploy [root@wj311osp1009cmaster-etcd-nfs-1 ~]# ls /etc/origin/node/certificates/ -al total 8 drwxr-xr-x. 2 root root 166 Oct 9 06:13 . drwx------. 5 root root 248 Oct 9 06:12 .. -rw-------. 1 root root 1167 Oct 9 06:12 kubelet-client-2020-10-09-06-12-55.pem lrwxrwxrwx. 1 root root 68 Oct 9 06:12 kubelet-client-current.pem -> /etc/origin/node/certificates/kubelet-client-2020-10-09-06-12-55.pem -rw-------. 1 root root 1289 Oct 9 06:13 kubelet-server-2020-10-09-06-13-02.pem lrwxrwxrwx. 1 root root 68 Oct 9 06:13 kubelet-server-current.pem -> /etc/origin/node/certificates/kubelet-server-2020-10-09-06-13-02.pem [root@wj311osp1009cmaster-etcd-nfs-1 ~]# ls /etc/origin/node/node.kubeconfig -al -rw-------. 1 root root 10524 Oct 9 06:10 /etc/origin/node/node.kubeconfig [root@wj311osp1009cmaster-etcd-nfs-1 ~]# oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME wj311osp1009cmaster-etcd-nfs-1 Ready master 56m v1.11.0+d4cacc0 10.0.150.163 <none> Red Hat Enterprise Linux Server 7.7 (Maipo) 3.10.0-1062.4.1.el7.x86_64 docker://1.13.1 wj311osp1009cnode-1 Ready compute 52m v1.11.0+d4cacc0 10.0.149.112 <none> Red Hat Enterprise Linux Server 7.7 (Maipo) 3.10.0-1062.4.1.el7.x86_64 docker://1.13.1 wj311osp1009cnode-registry-router-1 Ready <none> 52m v1.11.0+d4cacc0 10.0.149.23 <none> Red Hat Enterprise Linux Server 7.7 (Maipo) 3.10.0-1062.4.1.el7.x86_64 docker://1.13.1 [root@wj311osp1009cmaster-etcd-nfs-1 ~]# oc get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE default docker-registry-2-jsxgc 1/1 Running 0 7m 10.130.0.17 wj311osp1009cnode-registry-router-1 <none> default registry-console-1-kcrcj 1/1 Running 1 49m 10.128.0.10 wj311osp1009cmaster-etcd-nfs-1 <none> default router-2-zbbj6 1/1 Running 0 8m 10.0.149.23 wj311osp1009cnode-registry-router-1 <none> install-test mongodb-1-v8gwr 1/1 Running 1 43m 10.129.0.21 wj311osp1009cnode-1 <none> install-test nodejs-mongodb-example-1-build 0/1 Completed 0 43m 10.129.0.9 wj311osp1009cnode-1 <none> install-test nodejs-mongodb-example-1-cv8xc 1/1 Running 1 42m 10.129.0.19 wj311osp1009cnode-1 <none> kube-service-catalog apiserver-kxdtc 1/1 Running 0 5m 10.128.0.17 wj311osp1009cmaster-etcd-nfs-1 <none> kube-service-catalog controller-manager-nhjlp 1/1 Running 0 4m 10.128.0.18 wj311osp1009cmaster-etcd-nfs-1 <none> kube-system master-api-wj311osp1009cmaster-etcd-nfs-1 1/1 Running 5 56m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> kube-system master-controllers-wj311osp1009cmaster-etcd-nfs-1 1/1 Running 6 56m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> kube-system master-etcd-wj311osp1009cmaster-etcd-nfs-1 1/1 Running 3 55m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-ansible-service-broker asb-1-nkz4n 1/1 Running 0 4m 10.129.0.25 wj311osp1009cnode-1 <none> openshift-console console-6c6bdbd6f-v6x2v 1/1 Running 0 7m 10.128.0.15 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-monitoring alertmanager-main-0 3/3 Running 0 6m 10.130.0.19 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring alertmanager-main-1 3/3 Running 0 6m 10.129.0.24 wj311osp1009cnode-1 <none> openshift-monitoring alertmanager-main-2 3/3 Running 0 6m 10.130.0.20 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring cluster-monitoring-operator-576c6b8b55-qmm8s 1/1 Running 0 49m 10.129.0.20 wj311osp1009cnode-1 <none> openshift-monitoring grafana-6dc585b845-mdtxr 2/2 Running 0 6m 10.130.0.22 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring kube-state-metrics-585c47c777-xrp8s 3/3 Running 0 6m 10.130.0.21 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring node-exporter-9cmfw 2/2 Running 0 6m 10.0.149.112 wj311osp1009cnode-1 <none> openshift-monitoring node-exporter-wxcwx 2/2 Running 0 6m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-monitoring node-exporter-x24vr 2/2 Running 0 6m 10.0.149.23 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring prometheus-k8s-0 4/4 Running 1 6m 10.130.0.18 wj311osp1009cnode-registry-router-1 <none> openshift-monitoring prometheus-k8s-1 4/4 Running 1 6m 10.129.0.23 wj311osp1009cnode-1 <none> openshift-monitoring prometheus-operator-754d586f64-tzxm2 1/1 Running 0 49m 10.129.0.15 wj311osp1009cnode-1 <none> openshift-node sync-lq5jp 1/1 Running 2 52m 10.0.149.23 wj311osp1009cnode-registry-router-1 <none> openshift-node sync-r89bd 1/1 Running 2 53m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-node sync-wsvph 1/1 Running 0 52m 10.0.149.112 wj311osp1009cnode-1 <none> openshift-sdn ovs-465m6 1/1 Running 2 52m 10.0.149.23 wj311osp1009cnode-registry-router-1 <none> openshift-sdn ovs-qhd8b 1/1 Running 2 52m 10.0.149.112 wj311osp1009cnode-1 <none> openshift-sdn ovs-wd8p5 1/1 Running 2 53m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-sdn sdn-pxwg2 1/1 Running 2 53m 10.0.150.163 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-sdn sdn-qtxp2 1/1 Running 2 52m 10.0.149.112 wj311osp1009cnode-1 <none> openshift-sdn sdn-wzzlq 1/1 Running 2 52m 10.0.149.23 wj311osp1009cnode-registry-router-1 <none> openshift-template-service-broker apiserver-dq2vp 1/1 Running 0 3m 10.128.0.19 wj311osp1009cmaster-etcd-nfs-1 <none> openshift-web-console webconsole-7f485486c7-6nfv9 1/1 Running 0 7m 10.128.0.16 wj311osp1009cmaster-etcd-nfs-1 <none> [root@wj311osp1009cmaster-etcd-nfs-1 ~]# ls /etc/origin/master/*.crt -al -rw-r--r--. 1 root root 1151 Oct 9 06:10 /etc/origin/master/admin.crt -rw-r--r--. 1 root root 1090 Oct 9 06:10 /etc/origin/master/aggregator-front-proxy.crt -rw-r--r--. 1 root root 2141 Oct 9 06:10 /etc/origin/master/ca-bundle.crt -rw-r--r--. 1 root root 1070 Oct 9 06:04 /etc/origin/master/ca.crt -rw-r--r--. 1 root root 2140 Oct 9 06:04 /etc/origin/master/client-ca-bundle.crt -rw-r--r--. 1 root root 2587 Oct 9 06:10 /etc/origin/master/etcd.server.crt -rw-r--r--. 1 root root 1070 Oct 9 06:10 /etc/origin/master/front-proxy-ca.crt -rw-r--r--. 1 root root 1078 Oct 9 05:22 /etc/origin/master/frontproxy-ca.crt -rw-------. 1 root root 1895 Oct 9 05:21 /etc/origin/master/master.etcd-ca.crt -rw-------. 1 root root 5941 Oct 9 06:10 /etc/origin/master/master.etcd-client.crt -rw-r--r--. 1 root root 1135 Oct 9 06:10 /etc/origin/master/master.kubelet-client.crt -rw-r--r--. 1 root root 1086 Oct 9 06:10 /etc/origin/master/master.proxy-client.crt -rw-r--r--. 1 root root 2587 Oct 9 06:10 /etc/origin/master/master.server.crt -rw-r--r--. 1 root root 1099 Oct 9 05:22 /etc/origin/master/openshift-aggregator.crt -rw-r--r--. 1 root root 1123 Oct 9 06:10 /etc/origin/master/openshift-master.crt -rw-r--r--. 1 root root 2270 Oct 9 06:13 /etc/origin/master/openshift-router.crt -rw-r--r--. 1 root root 2372 Oct 9 06:13 /etc/origin/master/registry.crt -rw-r--r--. 1 root root 1115 Oct 9 06:10 /etc/origin/master/service-signer.crt
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4170