Bug 1855479

Summary: Post-upgrade the MCO failing to evict "build" pods during drain
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: kube-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, ccoleman, deads, dwalsh, jokerman, lmohanty, mfojtik, mgugino, mpatel, rphillips, scuppett, wking, xxia
Target Milestone: ---Keywords: Upgrades
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-23 13:52:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1855478    
Bug Blocks:    

Comment 1 W. Trevor King 2020-07-10 01:24:06 UTC
Depends on the 4.4 bug.

Comment 3 Seth Jennings 2020-08-10 16:06:16 UTC
Ryan is on leave

Comment 4 Seth Jennings 2020-08-10 20:03:56 UTC
fixed in https://github.com/openshift/origin/pull/25266

Comment 7 Ke Wang 2020-09-09 10:39:39 UTC
- Tried upgrade 4.2.z to 4.3 nightly build,

IPI install on AWS, upgrade from 4.2.36 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.36    True        False         6m22s   Cluster version is 4.2.36

$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE   VERSION                INTERNAL-IP    EXTERNAL-IP   OS-IMAGE  KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-131-241.us-east-2.compute.internal   Ready    master   19m   v1.14.6-152-g117ba1f   10.0.131.241   <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-143-227.us-east-2.compute.internal   Ready    worker   13m   v1.14.6-152-g117ba1f   10.0.143.227   <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-157-23.us-east-2.compute.internal    Ready    master   19m   v1.14.6-152-g117ba1f   10.0.157.23    <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-159-130.us-east-2.compute.internal   Ready    worker   13m   v1.14.6-152-g117ba1f   10.0.159.130   <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-160-238.us-east-2.compute.internal   Ready    master   19m   v1.14.6-152-g117ba1f   10.0.160.238   <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8
ip-10-0-164-122.us-east-2.compute.internal   Ready    worker   13m   v1.14.6-152-g117ba1f   10.0.164.122   <none>        Red Hat Enterprise Linux CoreOS 42.81.20200525.0 (Ootpa)   4.18.0-147.8.1.el8_1.x86_64   cri-o://1.14.12-10.dev.rhaos4.2.git313d784.el8_2

$ cat newapp.sh 
#!/usr/bin/env bash
newapp(){
oc new-app ./ruby-hello-world/ --strategy=docker
oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex
oc new-app https://github.com/sclorg/cakephp-ex
oc new-app https://github.com/sclorg/nodejs-ex
}
for i in {1..10}; do
  oc new-project test$i
  if [ $i -eq 9 -o $i -eq 10 ];then
    oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder
  fi
newapp
done

Using above script to create some apps.

$ oc get pods -A | grep ^test
test1         cakephp-ex-1-build                          0/1     Completed   0          23m
test1         cakephp-ex-6b884dd687-jdtdx                 1/1     Running     0          19m
test1         nodejs-ex-1-build                           0/1     Completed   0          22m
test1         nodejs-ex-7bdf958f7f-f4dl5                  1/1     Running     0          18m
test1         ruby-ex-1-build                             0/1     Completed   0          23m
test1         ruby-ex-7577b4dd55-fpdtt                    1/1     Running     0          20m
test1         ruby-hello-world-1-build                    0/1     Completed   0          23m
test1         ruby-hello-world-59589497-bbxxf             1/1     Running     0          21m
test10        cakephp-ex-1-build                          0/1     Completed   0          15m
test10        cakephp-ex-6d77b74995-5jfkt                 1/1     Running     0          5m7s
test10        nodejs-ex-1-build                           0/1     OOMKilled   0          15m
test10        nodejs-ex-7698c47fd6-zggf2                  1/1     Running     0          9m8s
test10        ruby-ex-1-build                             0/1     Completed   0          15m
test10        ruby-ex-7bc9777745-zhzhf                    1/1     Running     0          5m51s
test10        ruby-hello-world-1-build                    0/1     Completed   0          15m
test10        ruby-hello-world-54bc4b48df-zmzft           1/1     Running     0          5m2s
test2         cakephp-ex-1-build                          0/1     Completed   0          22m
test2         cakephp-ex-5dd565f6b6-wb67m                 1/1     Running     0          17m
test2         nodejs-ex-1-build                           0/1     Completed   0          21m
test2         nodejs-ex-6cc88874bb-tsqqn                  1/1     Running     0          14m
test2         ruby-ex-1-build                             0/1     Completed   0          22m
test2         ruby-ex-747bcdb76c-j4pkp                    1/1     Running     0          18m
test2         ruby-hello-world-1-build                    0/1     Completed   0          22m
test2         ruby-hello-world-6f7f4ccf98-6cbrw           1/1     Running     0          16m
test3         cakephp-ex-1-build                          0/1     Completed   0          21m
test3         cakephp-ex-657f7bf957-wnds5                 1/1     Running     0          13m
test3         nodejs-ex-1-build                           0/1     OOMKilled   0          21m
test3         nodejs-ex-8b86c7d86-862ck                   1/1     Running     0          16m
test3         ruby-ex-1-build                             0/1     Completed   0          21m
test3         ruby-ex-7b77499c6-54d7h                     1/1     Running     0          13m
test3         ruby-hello-world-1-build                    0/1     Completed   0          21m
test3         ruby-hello-world-848d6fffcc-fr2k8           1/1     Running     0          14m
test4         cakephp-ex-1-build                          0/1     Completed   0          20m
test4         cakephp-ex-76f579894c-s6x7l                 1/1     Running     0          11m
test4         nodejs-ex-1-build                           0/1     Completed   0          20m
test4         nodejs-ex-757fb7f8c8-l46dj                  1/1     Running     0          11m
test4         ruby-ex-1-build                             0/1     Completed   0          20m
test4         ruby-ex-54bf97bbcb-sp49w                    1/1     Running     0          11m
test4         ruby-hello-world-1-build                    0/1     Completed   0          20m
test4         ruby-hello-world-7c9bdcfdf5-gnjpt           1/1     Running     0          12m
test5         cakephp-ex-1-build                          0/1     Completed   0          19m
test5         cakephp-ex-57c877986b-6r9fp                 1/1     Running     0          11m
test5         nodejs-ex-1-build                           0/1     Completed   0          19m
test5         nodejs-ex-c7c5b667b-srm26                   1/1     Running     0          11m
test5         ruby-ex-1-build                             0/1     Completed   0          19m
test5         ruby-ex-8684b74769-5x62c                    1/1     Running     0          11m
test5         ruby-hello-world-1-build                    0/1     Completed   0          20m
test5         ruby-hello-world-7c8d887577-zb2m6           1/1     Running     0          11m
test6         cakephp-ex-1-build                          0/1     Completed   0          18m
test6         cakephp-ex-698c7989c8-t7cqm                 1/1     Running     0          9m15s
test6         nodejs-ex-1-build                           0/1     Completed   0          18m
test6         nodejs-ex-86dfcb8c48-qpn65                  1/1     Running     0          10m
test6         ruby-ex-1-build                             0/1     Completed   0          19m
test6         ruby-ex-769798c47-km4kc                     1/1     Running     0          11m
test6         ruby-hello-world-1-build                    0/1     Completed   0          19m
test6         ruby-hello-world-669fc65c-kgtqv             1/1     Running     0          7m34s
test7         cakephp-ex-1-build                          0/1     Completed   0          18m
test7         cakephp-ex-59c5b8d69c-w2pk4                 1/1     Running     0          6m10s
test7         nodejs-ex-1-build                           0/1     Completed   0          17m
test7         nodejs-ex-6bb9678bc4-r8zj7                  1/1     Running     0          9m21s
test7         ruby-ex-1-build                             0/1     Completed   0          18m
test7         ruby-ex-6c8c98f556-b88d8                    1/1     Running     0          10m
test7         ruby-hello-world-1-build                    0/1     Completed   0          18m
test7         ruby-hello-world-db8c8569b-zbr67            1/1     Running     0          6m12s
test8         cakephp-ex-1-build                          0/1     Completed   0          17m
test8         cakephp-ex-d6bd47599-cpqpc                  1/1     Running     0          9m46s
test8         nodejs-ex-1-build                           0/1     Completed   0          17m
test8         nodejs-ex-74b9c9b6b7-pwsxx                  1/1     Running     0          6m38s
test8         ruby-ex-1-build                             0/1     Completed   0          17m
test8         ruby-ex-867497b6dd-g7ksm                    1/1     Running     0          9m3s
test8         ruby-hello-world-1-build                    0/1     Completed   0          17m
test8         ruby-hello-world-79797cffff-9kf7p           1/1     Running     0          5m52s
test9         cakephp-ex-1-build                          0/1     Completed   0          16m
test9         cakephp-ex-857b798b57-cl77n                 1/1     Running     0          9m47s
test9         nodejs-ex-1-build                           0/1     Completed   0          16m
test9         nodejs-ex-67fcb884c9-bx48m                  1/1     Running     0          6m2s
test9         ruby-ex-1-build                             0/1     Completed   0          16m
test9         ruby-ex-55b8bf94c5-j6sk7                    1/1     Running     0          6m41s
test9         ruby-hello-world-1-build                    0/1     Completed   0          16m
test9         ruby-hello-world-7bb5858cdd-rkj9t           1/1     Running     0          8m40s

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-09-09-043522   True        False         31m     Cluster version is 4.3.0-0.nightly-2020-09-09-043522

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
master   rendered-master-59c6331d1421bfabeffbcd20bf8df0de   True      False      False      3              3                   3                     0
worker   rendered-worker-26925141dedf82d1c1313502a244a20a   True      False      False      3              3                   3                     0

$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      127m
cloud-credential                           4.3.0-0.nightly-2020-09-09-043522   True        False         False      138m
cluster-autoscaler                         4.3.0-0.nightly-2020-09-09-043522   True        False         False      131m
console                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      36m
dns                                        4.3.0-0.nightly-2020-09-09-043522   True        False         False      137m
image-registry                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      45m
ingress                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      39m
insights                                   4.3.0-0.nightly-2020-09-09-043522   True        False         False      138m
kube-apiserver                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      135m
kube-controller-manager                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      135m
kube-scheduler                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      135m
machine-api                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      138m
machine-config                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      137m
marketplace                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      37m
monitoring                                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      45m
network                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      137m
node-tuning                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      38m
openshift-apiserver                        4.3.0-0.nightly-2020-09-09-043522   True        False         False      35m
openshift-controller-manager               4.3.0-0.nightly-2020-09-09-043522   True        False         False      136m
openshift-samples                          4.3.0-0.nightly-2020-09-09-043522   True        False         False      55m
operator-lifecycle-manager                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      134m
operator-lifecycle-manager-catalog         4.3.0-0.nightly-2020-09-09-043522   True        False         False      134m
operator-lifecycle-manager-packageserver   4.3.0-0.nightly-2020-09-09-043522   True        False         False      38m
service-ca                                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      138m
service-catalog-apiserver                  4.3.0-0.nightly-2020-09-09-043522   True        False         False      134m
service-catalog-controller-manager         4.3.0-0.nightly-2020-09-09-043522   True        False         False      134m
storage                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      69m

$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE    VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-131-241.us-east-2.compute.internal   Ready    master   140m   v1.16.2+417b9fd   10.0.131.241   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-143-227.us-east-2.compute.internal   Ready    worker   134m   v1.16.2+417b9fd   10.0.143.227   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-157-23.us-east-2.compute.internal    Ready    master   140m   v1.16.2+417b9fd   10.0.157.23    <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-159-130.us-east-2.compute.internal   Ready    worker   134m   v1.16.2+417b9fd   10.0.159.130   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-160-238.us-east-2.compute.internal   Ready    master   140m   v1.16.2+417b9fd   10.0.160.238   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-164-122.us-east-2.compute.internal   Ready    worker   134m   v1.16.2+417b9fd   10.0.164.122   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8

$ oc get pods -A | grep ^test
test1       cakephp-ex-6b884dd687-ncd9m         1/1     Running     0          46m
test1       nodejs-ex-7bdf958f7f-kh9lj          1/1     Running     0          46m
test1       ruby-ex-7577b4dd55-z6bqj            1/1     Running     0          40m
test1       ruby-hello-world-59589497-nqz9k     1/1     Running     0          40m
test10      cakephp-ex-6d77b74995-zxxpw         1/1     Running     0          46m
test10      nodejs-ex-7698c47fd6-4598c          1/1     Running     0          40m
test10      ruby-ex-7bc9777745-mh9b5            1/1     Running     0          46m
test10      ruby-hello-world-54bc4b48df-q9pbz   1/1     Running     0          40m
test2       cakephp-ex-5dd565f6b6-6qbnw         1/1     Running     0          40m
test2       nodejs-ex-6cc88874bb-9ptzz          1/1     Running     0          40m
test2       ruby-ex-747bcdb76c-8lp6r            1/1     Running     0          46m
test2       ruby-hello-world-6f7f4ccf98-mpdxt   1/1     Running     0          40m
test3       cakephp-ex-657f7bf957-2b25d         1/1     Running     0          40m
test3       nodejs-ex-8b86c7d86-xvjfd           1/1     Running     0          40m
test3       ruby-ex-7b77499c6-rwdbl             1/1     Running     0          46m
test3       ruby-hello-world-848d6fffcc-7zq6b   1/1     Running     0          40m
test4       cakephp-ex-76f579894c-lhxvg         1/1     Running     0          40m
test4       nodejs-ex-757fb7f8c8-k8hmm          1/1     Running     0          46m
test4       ruby-ex-54bf97bbcb-pddpj            1/1     Running     0          46m
test4       ruby-hello-world-7c9bdcfdf5-rbhv4   1/1     Running     0          46m
test5       cakephp-ex-57c877986b-78vdg         1/1     Running     0          40m
test5       nodejs-ex-c7c5b667b-7zkch           1/1     Running     0          46m
test5       ruby-ex-8684b74769-qqpv5            1/1     Running     0          40m
test5       ruby-hello-world-7c8d887577-c5l5c   1/1     Running     0          46m
test6       cakephp-ex-698c7989c8-gc9wd         1/1     Running     0          46m
test6       nodejs-ex-86dfcb8c48-kbn78          1/1     Running     0          46m
test6       ruby-ex-769798c47-598nw             1/1     Running     0          40m
test6       ruby-hello-world-669fc65c-km8mm     1/1     Running     0          46m
test7       cakephp-ex-59c5b8d69c-8jd7c         1/1     Running     0          46m
test7       nodejs-ex-6bb9678bc4-j5n5h          1/1     Running     0          40m
test7       ruby-ex-6c8c98f556-5ptbk            1/1     Running     0          40m
test7       ruby-hello-world-db8c8569b-4zk9l    1/1     Running     0          40m
test8       cakephp-ex-d6bd47599-j9slp          1/1     Running     0          40m
test8       nodejs-ex-74b9c9b6b7-vw5z8          1/1     Running     0          40m
test8       ruby-ex-867497b6dd-bpp2l            1/1     Running     0          46m
test8       ruby-hello-world-79797cffff-l4pmp   1/1     Running     0          46m
test9       cakephp-ex-857b798b57-jk8zl         1/1     Running     0          40m
test9       nodejs-ex-67fcb884c9-mk95p          1/1     Running     0          40m
test9       ruby-ex-55b8bf94c5-47zxj            1/1     Running     0          40m
test9       ruby-hello-world-7bb5858cdd-9dxxm   1/1     Running     0          46m

Comment 8 Ke Wang 2020-09-09 15:00:27 UTC
- Tried upgrade between 4.3.z.

IPI install on GCP, upgrade from 4.3.35 to 4.3 nightly 4.3.0-0.nightly-2020-09-09-043522 succeeded with builds pods getting terminated.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.35    True        False         71m     Cluster version is 4.3.35

$ oc get nodes -o wide
NAME                                             STATUS   ROLES    AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
kewang-p6sw8-m-0.c.openshift-qe.internal         Ready    master   93m   v1.16.2+7279a4a   10.0.0.5                    Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-1.c.openshift-qe.internal         Ready    master   94m   v1.16.2+7279a4a   10.0.0.3                    Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-2.c.openshift-qe.internal         Ready    master   93m   v1.16.2+7279a4a   10.0.0.4                    Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal   Ready    worker   80m   v1.16.2+7279a4a   10.0.32.2                   Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal   Ready    worker   80m   v1.16.2+7279a4a   10.0.32.3                   Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal   Ready    worker   80m   v1.16.2+7279a4a   10.0.32.4                   Red Hat Enterprise Linux CoreOS 43.82.202008311553.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8

$ cat newapp.sh 
#!/usr/bin/env bash
newapp(){
oc new-app ./ruby-hello-world/ --strategy=docker
oc new-app openshift/ruby:2.5~https://github.com/wangke19/ruby-ex
oc new-app https://github.com/sclorg/cakephp-ex
oc new-app https://github.com/sclorg/nodejs-ex
}
for i in {1..10}; do
  oc new-project test$i
  if [ $i -eq 9 -o $i -eq 10 ];then
    oc adm policy add-scc-to-user privileged system:serviceaccount:test$i:builder
  fi
newapp
done

Using above script to create some apps.

$ oc get pods -A | grep ^test
test1       cakephp-ex-1-build                     0/1     Completed   0          18m
test1       cakephp-ex-5669cf9595-9gd22            1/1     Running     0          16m
test1       nodejs-ex-1-build                      0/1     Completed   0          18m
test1       nodejs-ex-bc9dc8559-2xl2s              1/1     Running     0          17m
test1       ruby-ex-1-build                        0/1     Completed   0          19m
test1       ruby-ex-7857965989-6gnf8               1/1     Running     0          18m
test1       ruby-hello-world-1-build               0/1     Completed   0          19m
test1       ruby-hello-world-7797d99584-pjtmg      1/1     Running     0          17m
test10      cakephp-ex-1-build                     0/1     Completed   0          5m36s
test10      cakephp-ex-57db887d6c-4h5kv            1/1     Running     0          4m14s
test10      nodejs-ex-1-build                      0/1     Completed   0          5m13s
test10      nodejs-ex-67cd8b4844-6mpqb             1/1     Running     0          4m29s
test10      ruby-ex-1-build                        0/1     Completed   0          6m2s
test10      ruby-ex-7476bfb668-wjrxs               1/1     Running     0          5m18s
test10      ruby-hello-world-1-build               0/1     Completed   0          6m13s
test10      ruby-hello-world-b4595bd64-zvx66       1/1     Running     0          5m5s
test2       cakephp-ex-1-build                     0/1     Completed   0          17m
test2       cakephp-ex-dfd477ddd-c256r             1/1     Running     0          15m
test2       nodejs-ex-1-build                      0/1     Completed   0          16m
test2       nodejs-ex-656f966b5b-p8wmq             1/1     Running     0          15m
test2       ruby-ex-1-build                        0/1     Completed   0          17m
test2       ruby-ex-684bfc68df-ctmqj               1/1     Running     0          16m
test2       ruby-hello-world-1-build               0/1     Completed   0          17m
test2       ruby-hello-world-6c7d567c7d-87d6z      1/1     Running     0          16m
test3       cakephp-ex-1-build                     0/1     Completed   0          15m
test3       cakephp-ex-5958fc74-pdx9k              1/1     Running     0          13m
test3       nodejs-ex-1-build                      0/1     Completed   0          15m
test3       nodejs-ex-dbbcfc67f-n9j27              1/1     Running     0          14m
test3       ruby-ex-1-build                        0/1     Completed   0          16m
test3       ruby-ex-69b7787699-5npfp               1/1     Running     0          15m
test3       ruby-hello-world-1-build               0/1     Completed   0          16m
test3       ruby-hello-world-86995d466b-wbm5q      1/1     Running     0          14m
test4       cakephp-ex-1-build                     0/1     Completed   0          14m
test4       cakephp-ex-6dfd74d678-9zjwn            1/1     Running     0          12m
test4       nodejs-ex-1-build                      0/1     Completed   0          14m
test4       nodejs-ex-7546c5bc8-d8xjj              1/1     Running     0          12m
test4       ruby-ex-1-build                        0/1     Completed   0          15m
test4       ruby-ex-66cddc9d5-bbmgg                1/1     Running     0          14m
test4       ruby-hello-world-1-build               0/1     Completed   0          15m
test4       ruby-hello-world-6676457779-hwpxf      1/1     Running     0          13m
test5       cakephp-ex-1-build                     0/1     Completed   0          13m
test5       cakephp-ex-5486f85f84-dvl96            1/1     Running     0          11m
test5       nodejs-ex-1-build                      0/1     Completed   0          12m
test5       nodejs-ex-858cd79ff5-5r79z             1/1     Running     0          11m
test5       ruby-ex-1-build                        0/1     Completed   0          13m
test5       ruby-ex-f548f5f5f-pfhkt                1/1     Running     0          12m
test5       ruby-hello-world-1-build               0/1     Completed   0          13m
test5       ruby-hello-world-5dd5db7576-mdvpp      1/1     Running     0          12m
test6       cakephp-ex-1-build                     0/1     Completed   0          11m
test6       cakephp-ex-84cf68774f-62d84            1/1     Running     0          10m
test6       nodejs-ex-1-build                      0/1     Completed   0          11m
test6       nodejs-ex-79bb7fc4fb-g4cjl             1/1     Running     0          10m
test6       ruby-ex-1-build                        0/1     Completed   0          12m
test6       ruby-ex-544496f6dc-r962v               1/1     Running     0          11m
test6       ruby-hello-world-1-build               0/1     Completed   0          12m
test6       ruby-hello-world-6488bd9dd8-5xlzb      1/1     Running     0          11m
test7       cakephp-ex-1-build                     0/1     Completed   0          9m57s
test7       cakephp-ex-7cf8d866c9-dpdtf            1/1     Running     0          8m33s
test7       nodejs-ex-1-build                      0/1     Completed   0          9m32s
test7       nodejs-ex-844d6ccf64-h84l8             1/1     Running     0          8m39s
test7       ruby-ex-1-build                        0/1     Completed   0          10m
test7       ruby-ex-5544c8879-n6kgk                1/1     Running     0          9m43s
test7       ruby-hello-world-1-build               0/1     Completed   0          10m
test7       ruby-hello-world-74594b4675-t5mjf      1/1     Running     0          9m52s
test8       cakephp-ex-1-build                     0/1     Completed   0          8m29s
test8       cakephp-ex-7995f78697-c7v7d            1/1     Running     0          7m10s
test8       nodejs-ex-1-build                      0/1     Completed   0          8m8s
test8       nodejs-ex-7b5c94ff88-gt25v             1/1     Running     0          7m7s
test8       ruby-ex-1-build                        0/1     Completed   0          8m56s
test8       ruby-ex-59f8c8f748-rg4jz               1/1     Running     0          7m58s
test8       ruby-hello-world-1-build               0/1     Completed   0          9m8s
test8       ruby-hello-world-79d47b4d-trjgs        1/1     Running     0          7m50s
test9       cakephp-ex-1-build                     0/1     Completed   0          7m7s
test9       cakephp-ex-56945f9ff-mvp4b             1/1     Running     0          5m24s
test9       nodejs-ex-1-build                      0/1     Completed   0          6m42s
test9       nodejs-ex-64ff55cf59-8mrm8             1/1     Running     0          5m42s
test9       ruby-ex-1-build                        0/1     Completed   0          7m29s
test9       ruby-ex-7bb4bfdd8f-gktr9               1/1     Running     0          6m9s
test9       ruby-hello-world-1-build               0/1     Completed   0          7m40s
test9       ruby-hello-world-86d76db4dd-9xzkn      1/1     Running     0          6m41s

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2020-09-09-043522 --allow-explicit-upgrade --force

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-09-09-043522   True        False         3h5m    Cluster version is 4.3.0-0.nightly-2020-09-09-043522

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
master   rendered-master-cbcdc8e72e97bbe782402cf26c204fbe   True      False      False      3              3                   3                     0
worker   rendered-worker-eeda4a8846152a8c46b17dbf377c8184   True      False      False      3              3                   3                     0

$  oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h16m
cloud-credential                           4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h38m
cluster-autoscaler                         4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h26m
console                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h10m
dns                                        4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h36m
image-registry                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h23m
ingress                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h13m
insights                                   4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
kube-apiserver                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h35m
kube-controller-manager                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h34m
kube-scheduler                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h34m
machine-api                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
machine-config                             4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h7m
marketplace                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h9m
monitoring                                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h17m
network                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h36m
node-tuning                                4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h9m
openshift-apiserver                        4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h7m
openshift-controller-manager               4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h35m
openshift-samples                          4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h33m
operator-lifecycle-manager                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
operator-lifecycle-manager-catalog         4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
operator-lifecycle-manager-packageserver   4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h9m
service-ca                                 4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h36m
service-catalog-apiserver                  4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
service-catalog-controller-manager         4.3.0-0.nightly-2020-09-09-043522   True        False         False      5h27m
storage                                    4.3.0-0.nightly-2020-09-09-043522   True        False         False      3h33m

$ oc get nodes -o wide
NAME                                             STATUS   ROLES    AGE     VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
kewang-p6sw8-m-0.c.openshift-qe.internal         Ready    master   5h38m   v1.16.2+417b9fd   10.0.0.5                    Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-1.c.openshift-qe.internal         Ready    master   5h38m   v1.16.2+417b9fd   10.0.0.3                    Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-m-2.c.openshift-qe.internal         Ready    master   5h38m   v1.16.2+417b9fd   10.0.0.4                    Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-a-njvk2.c.openshift-qe.internal   Ready    worker   5h24m   v1.16.2+417b9fd   10.0.32.2                   Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-b-g75qq.c.openshift-qe.internal   Ready    worker   5h25m   v1.16.2+417b9fd   10.0.32.3                   Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-p6sw8-w-c-m2b4n.c.openshift-qe.internal   Ready    worker   5h24m   v1.16.2+417b9fd   10.0.32.4                   Red Hat Enterprise Linux CoreOS 43.82.202009080953.0 (Ootpa)   4.18.0-193.19.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8

$ oc get pods -A | grep ^test
test1     cakephp-ex-5669cf9595-rctd4          1/1     Running     0          3h14m
test1     nodejs-ex-bc9dc8559-h55fh            1/1     Running     0          3h14m
test1     ruby-ex-7857965989-9lk4t             1/1     Running     0          3h14m
test1     ruby-hello-world-7797d99584-4brgs    1/1     Running     0          3h14m
test10    cakephp-ex-57db887d6c-j8p2h          1/1     Running     0          3h16m
test10    nodejs-ex-67cd8b4844-4wk82           1/1     Running     0          3h14m
test10    ruby-ex-7476bfb668-7g5zw             1/1     Running     0          3h14m
test10    ruby-hello-world-b4595bd64-lkbnx     1/1     Running     0          3h14m
test2     cakephp-ex-dfd477ddd-m7n4v           1/1     Running     0          3h14m
test2     nodejs-ex-656f966b5b-bmzkf           1/1     Running     0          3h14m
test2     ruby-ex-684bfc68df-l6jbn             1/1     Running     0          3h14m
test2     ruby-hello-world-6c7d567c7d-mn7tx    1/1     Running     0          3h16m
test3     cakephp-ex-5958fc74-pqb5x            1/1     Running     0          3h16m
test3     nodejs-ex-dbbcfc67f-rlrvf            1/1     Running     0          3h14m
test3     ruby-ex-69b7787699-w2r8h             1/1     Running     0          3h16m
test3     ruby-hello-world-86995d466b-l6p4d    1/1     Running     0          3h16m
test4     cakephp-ex-6dfd74d678-2fbwc          1/1     Running     0          3h14m
test4     nodejs-ex-7546c5bc8-2pkzh            1/1     Running     0          3h14m
test4     ruby-ex-66cddc9d5-57n7c              1/1     Running     0          3h16m
test4     ruby-hello-world-6676457779-vhxqt    1/1     Running     0          3h16m
test5     cakephp-ex-5486f85f84-fph8x          1/1     Running     0          3h14m
test5     nodejs-ex-858cd79ff5-8g828           1/1     Running     0          3h16m
test5     ruby-ex-f548f5f5f-5f79v              1/1     Running     0          3h16m
test5     ruby-hello-world-5dd5db7576-ztfwq    1/1     Running     0          3h16m
test6     cakephp-ex-84cf68774f-dg67t          1/1     Running     0          3h14m
test6     nodejs-ex-79bb7fc4fb-zb9lx           1/1     Running     0          3h16m
test6     ruby-ex-544496f6dc-lfwjp             1/1     Running     0          3h14m
test6     ruby-hello-world-6488bd9dd8-tzq76    1/1     Running     0          3h14m
test7     cakephp-ex-7cf8d866c9-mc78k          1/1     Running     0          3h16m
test7     nodejs-ex-844d6ccf64-9zssg           1/1     Running     0          3h14m
test7     ruby-ex-5544c8879-qm459              1/1     Running     0          3h14m
test7     ruby-hello-world-74594b4675-5nc5d    1/1     Running     0          3h16m
test8     cakephp-ex-7995f78697-tjwbw          1/1     Running     0          3h16m
test8     nodejs-ex-7b5c94ff88-4z8rt           1/1     Running     0          3h16m
test8     ruby-ex-59f8c8f748-r9pnn             1/1     Running     0          3h16m
test8     ruby-hello-world-79d47b4d-98lxq      1/1     Running     0          3h14m
test9     cakephp-ex-56945f9ff-phpms           1/1     Running     0          3h14m
test9     nodejs-ex-64ff55cf59-jkrvs           1/1     Running     0          3h16m
test9     ruby-ex-7bb4bfdd8f-sh68t             1/1     Running     0          3h16m
test9     ruby-hello-world-86d76db4dd-gwqg9    1/1     Running     0          3h16m

Comment 10 errata-xmlrpc 2020-09-23 13:52:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.3.38 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3609