Bug 1855479
Summary: | Post-upgrade the MCO failing to evict "build" pods during drain | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
Component: | kube-apiserver | Assignee: | Stefan Schimanski <sttts> |
Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> |
Severity: | high | Docs Contact: | |
Priority: | urgent | ||
Version: | 4.5 | CC: | aos-bugs, ccoleman, deads, dwalsh, jokerman, lmohanty, mfojtik, mgugino, mpatel, rphillips, scuppett, wking, xxia |
Target Milestone: | --- | Keywords: | Upgrades |
Target Release: | 4.3.z | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-09-23 13:52:39 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1855478 | ||
Bug Blocks: |
Comment 1
W. Trevor King
2020-07-10 01:24:06 UTC
Ryan is on leave - Tried upgrade 4.2.z to 4.3 nightly build, IPI install on AWS, upgrade from 4.2.36 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.36 True False 6m22s Cluster version is 4.2.36 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-131-241.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.131.241 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8 ip-10-0-143-227.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.143.227 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8 ip-10-0-157-23.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.157.23 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8 ip-10-0-159-130.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.159.130 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8 ip-10-0-160-238.us-east-2.compute.internal Ready master 19m v1.14.6-152-g117ba1f 10.0.160.238 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8 ip-10-0-164-122.us-east-2.compute.internal Ready worker 13m v1.14.6-152-g117ba1f 10.0.164.122 <none> Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8_2 $ cat newapp.sh #!/usr/bin/env bash newapp(){ oc new-app ./ruby-hello-world/ --strategy=docker oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex oc new-app https://github.com/sclorg/cakephp-ex oc new-app https://github.com/sclorg/nodejs-ex } for i in {1..10}; do oc new-project test$i if [ $i -eq 9 -o $i -eq 10 ];then oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder fi newapp done Using above script to create some apps. $ oc get pods -A | grep ^test test1 cakephp-ex-1-build 0/1 Completed 0 23m test1 cakephp-ex-6b884dd687-jdtdx 1/1 Running 0 19m test1 nodejs-ex-1-build 0/1 Completed 0 22m test1 nodejs-ex-7bdf958f7f-f4dl5 1/1 Running 0 18m test1 ruby-ex-1-build 0/1 Completed 0 23m test1 ruby-ex-7577b4dd55-fpdtt 1/1 Running 0 20m test1 ruby-hello-world-1-build 0/1 Completed 0 23m test1 ruby-hello-world-59589497-bbxxf 1/1 Running 0 21m test10 cakephp-ex-1-build 0/1 Completed 0 15m test10 cakephp-ex-6d77b74995-5jfkt 1/1 Running 0 5m7s test10 nodejs-ex-1-build 0/1 OOMKilled 0 15m test10 nodejs-ex-7698c47fd6-zggf2 1/1 Running 0 9m8s test10 ruby-ex-1-build 0/1 Completed 0 15m test10 ruby-ex-7bc9777745-zhzhf 1/1 Running 0 5m51s test10 ruby-hello-world-1-build 0/1 Completed 0 15m test10 ruby-hello-world-54bc4b48df-zmzft 1/1 Running 0 5m2s test2 cakephp-ex-1-build 0/1 Completed 0 22m test2 cakephp-ex-5dd565f6b6-wb67m 1/1 Running 0 17m test2 nodejs-ex-1-build 0/1 Completed 0 21m test2 nodejs-ex-6cc88874bb-tsqqn 1/1 Running 0 14m test2 ruby-ex-1-build 0/1 Completed 0 22m test2 ruby-ex-747bcdb76c-j4pkp 1/1 Running 0 18m test2 ruby-hello-world-1-build 0/1 Completed 0 22m test2 ruby-hello-world-6f7f4ccf98-6cbrw 1/1 Running 0 16m test3 cakephp-ex-1-build 0/1 Completed 0 21m test3 cakephp-ex-657f7bf957-wnds5 1/1 Running 0 13m test3 nodejs-ex-1-build 0/1 OOMKilled 0 21m test3 nodejs-ex-8b86c7d86-862ck 1/1 Running 0 16m test3 ruby-ex-1-build 0/1 Completed 0 21m test3 ruby-ex-7b77499c6-54d7h 1/1 Running 0 13m test3 ruby-hello-world-1-build 0/1 Completed 0 21m test3 ruby-hello-world-848d6fffcc-fr2k8 1/1 Running 0 14m test4 cakephp-ex-1-build 0/1 Completed 0 20m test4 cakephp-ex-76f579894c-s6x7l 1/1 Running 0 11m test4 nodejs-ex-1-build 0/1 Completed 0 20m test4 nodejs-ex-757fb7f8c8-l46dj 1/1 Running 0 11m test4 ruby-ex-1-build 0/1 Completed 0 20m test4 ruby-ex-54bf97bbcb-sp49w 1/1 Running 0 11m test4 ruby-hello-world-1-build 0/1 Completed 0 20m test4 ruby-hello-world-7c9bdcfdf5-gnjpt 1/1 Running 0 12m test5 cakephp-ex-1-build 0/1 Completed 0 19m test5 cakephp-ex-57c877986b-6r9fp 1/1 Running 0 11m test5 nodejs-ex-1-build 0/1 Completed 0 19m test5 nodejs-ex-c7c5b667b-srm26 1/1 Running 0 11m test5 ruby-ex-1-build 0/1 Completed 0 19m test5 ruby-ex-8684b74769-5x62c 1/1 Running 0 11m test5 ruby-hello-world-1-build 0/1 Completed 0 20m test5 ruby-hello-world-7c8d887577-zb2m6 1/1 Running 0 11m test6 cakephp-ex-1-build 0/1 Completed 0 18m test6 cakephp-ex-698c7989c8-t7cqm 1/1 Running 0 9m15s test6 nodejs-ex-1-build 0/1 Completed 0 18m test6 nodejs-ex-86dfcb8c48-qpn65 1/1 Running 0 10m test6 ruby-ex-1-build 0/1 Completed 0 19m test6 ruby-ex-769798c47-km4kc 1/1 Running 0 11m test6 ruby-hello-world-1-build 0/1 Completed 0 19m test6 ruby-hello-world-669fc65c-kgtqv 1/1 Running 0 7m34s test7 cakephp-ex-1-build 0/1 Completed 0 18m test7 cakephp-ex-59c5b8d69c-w2pk4 1/1 Running 0 6m10s test7 nodejs-ex-1-build 0/1 Completed 0 17m test7 nodejs-ex-6bb9678bc4-r8zj7 1/1 Running 0 9m21s test7 ruby-ex-1-build 0/1 Completed 0 18m test7 ruby-ex-6c8c98f556-b88d8 1/1 Running 0 10m test7 ruby-hello-world-1-build 0/1 Completed 0 18m test7 ruby-hello-world-db8c8569b-zbr67 1/1 Running 0 6m12s test8 cakephp-ex-1-build 0/1 Completed 0 17m test8 cakephp-ex-d6bd47599-cpqpc 1/1 Running 0 9m46s test8 nodejs-ex-1-build 0/1 Completed 0 17m test8 nodejs-ex-74b9c9b6b7-pwsxx 1/1 Running 0 6m38s test8 ruby-ex-1-build 0/1 Completed 0 17m test8 ruby-ex-867497b6dd-g7ksm 1/1 Running 0 9m3s test8 ruby-hello-world-1-build 0/1 Completed 0 17m test8 ruby-hello-world-79797cffff-9kf7p 1/1 Running 0 5m52s test9 cakephp-ex-1-build 0/1 Completed 0 16m test9 cakephp-ex-857b798b57-cl77n 1/1 Running 0 9m47s test9 nodejs-ex-1-build 0/1 Completed 0 16m test9 nodejs-ex-67fcb884c9-bx48m 1/1 Running 0 6m2s test9 ruby-ex-1-build 0/1 Completed 0 16m test9 ruby-ex-55b8bf94c5-j6sk7 1/1 Running 0 6m41s test9 ruby-hello-world-1-build 0/1 Completed 0 16m test9 ruby-hello-world-7bb5858cdd-rkj9t 1/1 Running 0 8m40s $ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.0-0.nightly-2020-09-09-043522 True False 31m Cluster version is 4.3.0-0.nightly-2020-09-09-043522 $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT master rendered-master-59c6331d1421bfabeffbcd20bf8df0de True False False 3 3 3 0 worker rendered-worker-26925141dedf82d1c1313502a244a20a True False False 3 3 3 0 $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.0-0.nightly-2020-09-09-043522 True False False 127m cloud-credential 4.3.0-0.nightly-2020-09-09-043522 True False False 138m cluster-autoscaler 4.3.0-0.nightly-2020-09-09-043522 True False False 131m console 4.3.0-0.nightly-2020-09-09-043522 True False False 36m dns 4.3.0-0.nightly-2020-09-09-043522 True False False 137m image-registry 4.3.0-0.nightly-2020-09-09-043522 True False False 45m ingress 4.3.0-0.nightly-2020-09-09-043522 True False False 39m insights 4.3.0-0.nightly-2020-09-09-043522 True False False 138m kube-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 135m kube-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 135m kube-scheduler 4.3.0-0.nightly-2020-09-09-043522 True False False 135m machine-api 4.3.0-0.nightly-2020-09-09-043522 True False False 138m machine-config 4.3.0-0.nightly-2020-09-09-043522 True False False 137m marketplace 4.3.0-0.nightly-2020-09-09-043522 True False False 37m monitoring 4.3.0-0.nightly-2020-09-09-043522 True False False 45m network 4.3.0-0.nightly-2020-09-09-043522 True False False 137m node-tuning 4.3.0-0.nightly-2020-09-09-043522 True False False 38m openshift-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 35m openshift-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 136m openshift-samples 4.3.0-0.nightly-2020-09-09-043522 True False False 55m operator-lifecycle-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 134m operator-lifecycle-manager-catalog 4.3.0-0.nightly-2020-09-09-043522 True False False 134m operator-lifecycle-manager-packageserver 4.3.0-0.nightly-2020-09-09-043522 True False False 38m service-ca 4.3.0-0.nightly-2020-09-09-043522 True False False 138m service-catalog-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 134m service-catalog-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 134m storage 4.3.0-0.nightly-2020-09-09-043522 True False False 69m $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-131-241.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.131.241 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-143-227.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.143.227 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-157-23.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.157.23 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-159-130.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.159.130 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-160-238.us-east-2.compute.internal Ready master 140m v1.16.2+417b9fd 10.0.160.238 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-164-122.us-east-2.compute.internal Ready worker 134m v1.16.2+417b9fd 10.0.164.122 <none> Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 $ oc get pods -A | grep ^test test1 cakephp-ex-6b884dd687-ncd9m 1/1 Running 0 46m test1 nodejs-ex-7bdf958f7f-kh9lj 1/1 Running 0 46m test1 ruby-ex-7577b4dd55-z6bqj 1/1 Running 0 40m test1 ruby-hello-world-59589497-nqz9k 1/1 Running 0 40m test10 cakephp-ex-6d77b74995-zxxpw 1/1 Running 0 46m test10 nodejs-ex-7698c47fd6-4598c 1/1 Running 0 40m test10 ruby-ex-7bc9777745-mh9b5 1/1 Running 0 46m test10 ruby-hello-world-54bc4b48df-q9pbz 1/1 Running 0 40m test2 cakephp-ex-5dd565f6b6-6qbnw 1/1 Running 0 40m test2 nodejs-ex-6cc88874bb-9ptzz 1/1 Running 0 40m test2 ruby-ex-747bcdb76c-8lp6r 1/1 Running 0 46m test2 ruby-hello-world-6f7f4ccf98-mpdxt 1/1 Running 0 40m test3 cakephp-ex-657f7bf957-2b25d 1/1 Running 0 40m test3 nodejs-ex-8b86c7d86-xvjfd 1/1 Running 0 40m test3 ruby-ex-7b77499c6-rwdbl 1/1 Running 0 46m test3 ruby-hello-world-848d6fffcc-7zq6b 1/1 Running 0 40m test4 cakephp-ex-76f579894c-lhxvg 1/1 Running 0 40m test4 nodejs-ex-757fb7f8c8-k8hmm 1/1 Running 0 46m test4 ruby-ex-54bf97bbcb-pddpj 1/1 Running 0 46m test4 ruby-hello-world-7c9bdcfdf5-rbhv4 1/1 Running 0 46m test5 cakephp-ex-57c877986b-78vdg 1/1 Running 0 40m test5 nodejs-ex-c7c5b667b-7zkch 1/1 Running 0 46m test5 ruby-ex-8684b74769-qqpv5 1/1 Running 0 40m test5 ruby-hello-world-7c8d887577-c5l5c 1/1 Running 0 46m test6 cakephp-ex-698c7989c8-gc9wd 1/1 Running 0 46m test6 nodejs-ex-86dfcb8c48-kbn78 1/1 Running 0 46m test6 ruby-ex-769798c47-598nw 1/1 Running 0 40m test6 ruby-hello-world-669fc65c-km8mm 1/1 Running 0 46m test7 cakephp-ex-59c5b8d69c-8jd7c 1/1 Running 0 46m test7 nodejs-ex-6bb9678bc4-j5n5h 1/1 Running 0 40m test7 ruby-ex-6c8c98f556-5ptbk 1/1 Running 0 40m test7 ruby-hello-world-db8c8569b-4zk9l 1/1 Running 0 40m test8 cakephp-ex-d6bd47599-j9slp 1/1 Running 0 40m test8 nodejs-ex-74b9c9b6b7-vw5z8 1/1 Running 0 40m test8 ruby-ex-867497b6dd-bpp2l 1/1 Running 0 46m test8 ruby-hello-world-79797cffff-l4pmp 1/1 Running 0 46m test9 cakephp-ex-857b798b57-jk8zl 1/1 Running 0 40m test9 nodejs-ex-67fcb884c9-mk95p 1/1 Running 0 40m test9 ruby-ex-55b8bf94c5-47zxj 1/1 Running 0 40m test9 ruby-hello-world-7bb5858cdd-9dxxm 1/1 Running 0 46m - Tried upgrade between 4.3.z. IPI install on GCP, upgrade from 4.3.35 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.35 True False 71m Cluster version is 4.3.35 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kewang-p6sw8-m-0.c.openshift-qe.internal Ready master 93m v1.16.2+7279a4a 10.0.0.5 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-m-1.c.openshift-qe.internal Ready master 94m v1.16.2+7279a4a 10.0.0.3 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-m-2.c.openshift-qe.internal Ready master 93m v1.16.2+7279a4a 10.0.0.4 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.2 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.3 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal Ready worker 80m v1.16.2+7279a4a 10.0.32.4 Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 $ cat newapp.sh #!/usr/bin/env bash newapp(){ oc new-app ./ruby-hello-world/ --strategy=docker oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex oc new-app https://github.com/sclorg/cakephp-ex oc new-app https://github.com/sclorg/nodejs-ex } for i in {1..10}; do oc new-project test$i if [ $i -eq 9 -o $i -eq 10 ];then oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder fi newapp done Using above script to create some apps. $ oc get pods -A | grep ^test test1 cakephp-ex-1-build 0/1 Completed 0 18m test1 cakephp-ex-5669cf9595-9gd22 1/1 Running 0 16m test1 nodejs-ex-1-build 0/1 Completed 0 18m test1 nodejs-ex-bc9dc8559-2xl2s 1/1 Running 0 17m test1 ruby-ex-1-build 0/1 Completed 0 19m test1 ruby-ex-7857965989-6gnf8 1/1 Running 0 18m test1 ruby-hello-world-1-build 0/1 Completed 0 19m test1 ruby-hello-world-7797d99584-pjtmg 1/1 Running 0 17m test10 cakephp-ex-1-build 0/1 Completed 0 5m36s test10 cakephp-ex-57db887d6c-4h5kv 1/1 Running 0 4m14s test10 nodejs-ex-1-build 0/1 Completed 0 5m13s test10 nodejs-ex-67cd8b4844-6mpqb 1/1 Running 0 4m29s test10 ruby-ex-1-build 0/1 Completed 0 6m2s test10 ruby-ex-7476bfb668-wjrxs 1/1 Running 0 5m18s test10 ruby-hello-world-1-build 0/1 Completed 0 6m13s test10 ruby-hello-world-b4595bd64-zvx66 1/1 Running 0 5m5s test2 cakephp-ex-1-build 0/1 Completed 0 17m test2 cakephp-ex-dfd477ddd-c256r 1/1 Running 0 15m test2 nodejs-ex-1-build 0/1 Completed 0 16m test2 nodejs-ex-656f966b5b-p8wmq 1/1 Running 0 15m test2 ruby-ex-1-build 0/1 Completed 0 17m test2 ruby-ex-684bfc68df-ctmqj 1/1 Running 0 16m test2 ruby-hello-world-1-build 0/1 Completed 0 17m test2 ruby-hello-world-6c7d567c7d-87d6z 1/1 Running 0 16m test3 cakephp-ex-1-build 0/1 Completed 0 15m test3 cakephp-ex-5958fc74-pdx9k 1/1 Running 0 13m test3 nodejs-ex-1-build 0/1 Completed 0 15m test3 nodejs-ex-dbbcfc67f-n9j27 1/1 Running 0 14m test3 ruby-ex-1-build 0/1 Completed 0 16m test3 ruby-ex-69b7787699-5npfp 1/1 Running 0 15m test3 ruby-hello-world-1-build 0/1 Completed 0 16m test3 ruby-hello-world-86995d466b-wbm5q 1/1 Running 0 14m test4 cakephp-ex-1-build 0/1 Completed 0 14m test4 cakephp-ex-6dfd74d678-9zjwn 1/1 Running 0 12m test4 nodejs-ex-1-build 0/1 Completed 0 14m test4 nodejs-ex-7546c5bc8-d8xjj 1/1 Running 0 12m test4 ruby-ex-1-build 0/1 Completed 0 15m test4 ruby-ex-66cddc9d5-bbmgg 1/1 Running 0 14m test4 ruby-hello-world-1-build 0/1 Completed 0 15m test4 ruby-hello-world-6676457779-hwpxf 1/1 Running 0 13m test5 cakephp-ex-1-build 0/1 Completed 0 13m test5 cakephp-ex-5486f85f84-dvl96 1/1 Running 0 11m test5 nodejs-ex-1-build 0/1 Completed 0 12m test5 nodejs-ex-858cd79ff5-5r79z 1/1 Running 0 11m test5 ruby-ex-1-build 0/1 Completed 0 13m test5 ruby-ex-f548f5f5f-pfhkt 1/1 Running 0 12m test5 ruby-hello-world-1-build 0/1 Completed 0 13m test5 ruby-hello-world-5dd5db7576-mdvpp 1/1 Running 0 12m test6 cakephp-ex-1-build 0/1 Completed 0 11m test6 cakephp-ex-84cf68774f-62d84 1/1 Running 0 10m test6 nodejs-ex-1-build 0/1 Completed 0 11m test6 nodejs-ex-79bb7fc4fb-g4cjl 1/1 Running 0 10m test6 ruby-ex-1-build 0/1 Completed 0 12m test6 ruby-ex-544496f6dc-r962v 1/1 Running 0 11m test6 ruby-hello-world-1-build 0/1 Completed 0 12m test6 ruby-hello-world-6488bd9dd8-5xlzb 1/1 Running 0 11m test7 cakephp-ex-1-build 0/1 Completed 0 9m57s test7 cakephp-ex-7cf8d866c9-dpdtf 1/1 Running 0 8m33s test7 nodejs-ex-1-build 0/1 Completed 0 9m32s test7 nodejs-ex-844d6ccf64-h84l8 1/1 Running 0 8m39s test7 ruby-ex-1-build 0/1 Completed 0 10m test7 ruby-ex-5544c8879-n6kgk 1/1 Running 0 9m43s test7 ruby-hello-world-1-build 0/1 Completed 0 10m test7 ruby-hello-world-74594b4675-t5mjf 1/1 Running 0 9m52s test8 cakephp-ex-1-build 0/1 Completed 0 8m29s test8 cakephp-ex-7995f78697-c7v7d 1/1 Running 0 7m10s test8 nodejs-ex-1-build 0/1 Completed 0 8m8s test8 nodejs-ex-7b5c94ff88-gt25v 1/1 Running 0 7m7s test8 ruby-ex-1-build 0/1 Completed 0 8m56s test8 ruby-ex-59f8c8f748-rg4jz 1/1 Running 0 7m58s test8 ruby-hello-world-1-build 0/1 Completed 0 9m8s test8 ruby-hello-world-79d47b4d-trjgs 1/1 Running 0 7m50s test9 cakephp-ex-1-build 0/1 Completed 0 7m7s test9 cakephp-ex-56945f9ff-mvp4b 1/1 Running 0 5m24s test9 nodejs-ex-1-build 0/1 Completed 0 6m42s test9 nodejs-ex-64ff55cf59-8mrm8 1/1 Running 0 5m42s test9 ruby-ex-1-build 0/1 Completed 0 7m29s test9 ruby-ex-7bb4bfdd8f-gktr9 1/1 Running 0 6m9s test9 ruby-hello-world-1-build 0/1 Completed 0 7m40s test9 ruby-hello-world-86d76db4dd-9xzkn 1/1 Running 0 6m41s $ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.0-0.nightly-2020-09-09-043522 True False 3h5m Cluster version is 4.3.0-0.nightly-2020-09-09-043522 $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT master rendered-master-cbcdc8e72e97bbe782402cf26c204fbe True False False 3 3 3 0 worker rendered-worker-eeda4a8846152a8c46b17dbf377c8184 True False False 3 3 3 0 $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.0-0.nightly-2020-09-09-043522 True False False 5h16m cloud-credential 4.3.0-0.nightly-2020-09-09-043522 True False False 5h38m cluster-autoscaler 4.3.0-0.nightly-2020-09-09-043522 True False False 5h26m console 4.3.0-0.nightly-2020-09-09-043522 True False False 3h10m dns 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m image-registry 4.3.0-0.nightly-2020-09-09-043522 True False False 5h23m ingress 4.3.0-0.nightly-2020-09-09-043522 True False False 3h13m insights 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m kube-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 5h35m kube-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h34m kube-scheduler 4.3.0-0.nightly-2020-09-09-043522 True False False 5h34m machine-api 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m machine-config 4.3.0-0.nightly-2020-09-09-043522 True False False 3h7m marketplace 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m monitoring 4.3.0-0.nightly-2020-09-09-043522 True False False 5h17m network 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m node-tuning 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m openshift-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 3h7m openshift-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h35m openshift-samples 4.3.0-0.nightly-2020-09-09-043522 True False False 3h33m operator-lifecycle-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m operator-lifecycle-manager-catalog 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m operator-lifecycle-manager-packageserver 4.3.0-0.nightly-2020-09-09-043522 True False False 3h9m service-ca 4.3.0-0.nightly-2020-09-09-043522 True False False 5h36m service-catalog-apiserver 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m service-catalog-controller-manager 4.3.0-0.nightly-2020-09-09-043522 True False False 5h27m storage 4.3.0-0.nightly-2020-09-09-043522 True False False 3h33m $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kewang-p6sw8-m-0.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.5 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-m-1.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.3 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-m-2.c.openshift-qe.internal Ready master 5h38m v1.16.2+417b9fd 10.0.0.4 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal Ready worker 5h24m v1.16.2+417b9fd 10.0.32.2 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal Ready worker 5h25m v1.16.2+417b9fd 10.0.32.3 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal Ready worker 5h24m v1.16.2+417b9fd 10.0.32.4 Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa) 4.18.0-193.19.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 $ oc get pods -A | grep ^test test1 cakephp-ex-5669cf9595-rctd4 1/1 Running 0 3h14m test1 nodejs-ex-bc9dc8559-h55fh 1/1 Running 0 3h14m test1 ruby-ex-7857965989-9lk4t 1/1 Running 0 3h14m test1 ruby-hello-world-7797d99584-4brgs 1/1 Running 0 3h14m test10 cakephp-ex-57db887d6c-j8p2h 1/1 Running 0 3h16m test10 nodejs-ex-67cd8b4844-4wk82 1/1 Running 0 3h14m test10 ruby-ex-7476bfb668-7g5zw 1/1 Running 0 3h14m test10 ruby-hello-world-b4595bd64-lkbnx 1/1 Running 0 3h14m test2 cakephp-ex-dfd477ddd-m7n4v 1/1 Running 0 3h14m test2 nodejs-ex-656f966b5b-bmzkf 1/1 Running 0 3h14m test2 ruby-ex-684bfc68df-l6jbn 1/1 Running 0 3h14m test2 ruby-hello-world-6c7d567c7d-mn7tx 1/1 Running 0 3h16m test3 cakephp-ex-5958fc74-pqb5x 1/1 Running 0 3h16m test3 nodejs-ex-dbbcfc67f-rlrvf 1/1 Running 0 3h14m test3 ruby-ex-69b7787699-w2r8h 1/1 Running 0 3h16m test3 ruby-hello-world-86995d466b-l6p4d 1/1 Running 0 3h16m test4 cakephp-ex-6dfd74d678-2fbwc 1/1 Running 0 3h14m test4 nodejs-ex-7546c5bc8-2pkzh 1/1 Running 0 3h14m test4 ruby-ex-66cddc9d5-57n7c 1/1 Running 0 3h16m test4 ruby-hello-world-6676457779-vhxqt 1/1 Running 0 3h16m test5 cakephp-ex-5486f85f84-fph8x 1/1 Running 0 3h14m test5 nodejs-ex-858cd79ff5-8g828 1/1 Running 0 3h16m test5 ruby-ex-f548f5f5f-5f79v 1/1 Running 0 3h16m test5 ruby-hello-world-5dd5db7576-ztfwq 1/1 Running 0 3h16m test6 cakephp-ex-84cf68774f-dg67t 1/1 Running 0 3h14m test6 nodejs-ex-79bb7fc4fb-zb9lx 1/1 Running 0 3h16m test6 ruby-ex-544496f6dc-lfwjp 1/1 Running 0 3h14m test6 ruby-hello-world-6488bd9dd8-tzq76 1/1 Running 0 3h14m test7 cakephp-ex-7cf8d866c9-mc78k 1/1 Running 0 3h16m test7 nodejs-ex-844d6ccf64-9zssg 1/1 Running 0 3h14m test7 ruby-ex-5544c8879-qm459 1/1 Running 0 3h14m test7 ruby-hello-world-74594b4675-5nc5d 1/1 Running 0 3h16m test8 cakephp-ex-7995f78697-tjwbw 1/1 Running 0 3h16m test8 nodejs-ex-7b5c94ff88-4z8rt 1/1 Running 0 3h16m test8 ruby-ex-59f8c8f748-r9pnn 1/1 Running 0 3h16m test8 ruby-hello-world-79d47b4d-98lxq 1/1 Running 0 3h14m test9 cakephp-ex-56945f9ff-phpms 1/1 Running 0 3h14m test9 nodejs-ex-64ff55cf59-jkrvs 1/1 Running 0 3h16m test9 ruby-ex-7bb4bfdd8f-sh68t 1/1 Running 0 3h16m test9 ruby-hello-world-86d76db4dd-gwqg9 1/1 Running 0 3h16m Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.3.38 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3609 |