Bug 1855478 - Post-upgrade the MCO failing to evict "build" pods during drain
Summary: Post-upgrade the MCO failing to evict "build" pods during drain
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.4.z
Assignee: Stefan Schimanski
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On: 1855444
Blocks: 1855479
TreeView+ depends on / blocked
 
Reported: 2020-07-10 00:15 UTC by OpenShift BugZilla Robot
Modified: 2020-09-01 19:41 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-01 19:41:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25265 0 None closed [release-4.4] bug 1855478: UPSTREAM: 91648: Changes to ManagedFields is not mutation for GC 2020-12-10 07:39:59 UTC
Red Hat Product Errata RHBA-2020:3514 0 None None None 2020-09-01 19:41:54 UTC

Comment 1 W. Trevor King 2020-07-10 01:23:25 UTC
Depends on the 4.5 bug.

Comment 3 Seth Jennings 2020-08-10 16:05:53 UTC
Ryan is on leave

Comment 4 Seth Jennings 2020-08-10 20:03:13 UTC
fixed in https://github.com/openshift/origin/pull/25265

Comment 7 Ke Wang 2020-08-25 04:21:46 UTC
This bug is hard to reproduce, so run the following cases to verify it.

IPI install on GCP, upgrade from 4.4.18 to 4.4 nightly 4.4.0-0.nightly-2020-08-23-194129 succeeded with builds pods getting terminated.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.18    True        False         115m    Cluster version is 4.4.18


$ oc get nodes -o wide
NAME                                                      STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
kewang2444-p2kw8-master-0.c.openshift-qe.internal         Ready    master   149m   v1.17.1+20ba474   10.0.0.4                    Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8
kewang2444-p2kw8-master-1.c.openshift-qe.internal         Ready    master   144m   v1.17.1+20ba474   10.0.0.6                    Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8
kewang2444-p2kw8-master-2.c.openshift-qe.internal         Ready    master   149m   v1.17.1+20ba474   10.0.0.3                    Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8
kewang2444-p2kw8-worker-a-qksjb.c.openshift-qe.internal   Ready    worker   132m   v1.17.1+20ba474   10.0.32.3                   Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8
kewang2444-p2kw8-worker-b-kv99q.c.openshift-qe.internal   Ready    worker   132m   v1.17.1+20ba474   10.0.32.4                   Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8
kewang2444-p2kw8-worker-c-w8w5x.c.openshift-qe.internal   Ready    worker   132m   v1.17.1+20ba474   10.0.32.2                   Red Hat Enterprise Linux CoreOS 44.82.202008131530-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.4-24.rhaos4.4.git73658e6.el8


$  oc get pods -A | grep ^test
test1                                                   cakephp-example-1-build                                                      0/1     Completed   0          52m
test1                                                   cakephp-example-1-deploy                                                     0/1     Completed   0          51m
test1                                                   cakephp-example-1-jf25p                                                      1/1     Running     0          51m
test1                                                   nodejs-example-1-build                                                       0/1     Completed   0          82m
test1                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          82m
test1                                                   nodejs-example-1-xdh7l                                                       1/1     Running     0          82m
test1                                                   ruby-ex-1-build                                                              0/1     Completed   0          22m
test1                                                   ruby-ex-2-build                                                              0/1     Completed   0          22m
test1                                                   ruby-ex-8bd4c9987-j8r5v                                                      1/1     Running     0          21m
test1                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          48m
test1                                                   ruby-hello-world-2-build                                                     0/1     Completed   0          38m
test1                                                   ruby-hello-world-785949b87b-pkw94                                            1/1     Running     0          37m
test2                                                   nodejs-example-1-build                                                       0/1     Completed   0          12m
test2                                                   nodejs-example-1-cnpwv                                                       1/1     Running     0          11m
test2                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          11m
test2                                                   ruby-ex-1-build                                                              0/1     Completed   0          11m
test2                                                   ruby-ex-7b4b94c957-4kq76                                                     1/1     Running     0          10m
test2                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          11m
test2                                                   ruby-hello-world-844647774d-985hq                                            1/1     Running     0          10m
test3                                                   nodejs-example-1-build                                                       0/1     Completed   0          11m
test3                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          8m18s
test3                                                   nodejs-example-1-dr9tc                                                       1/1     Running     0          8m13s
test3                                                   ruby-ex-1-build                                                              0/1     Completed   0          11m
test3                                                   ruby-ex-fcfdb75f8-g8cjt                                                      1/1     Running     0          7m42s
test3                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          11m
test3                                                   ruby-hello-world-5f7b77b454-g9sqx                                            1/1     Running     0          6m9s
test4                                                   nodejs-example-1-build                                                       0/1     Completed   0          10m
test4                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          6m40s
test4                                                   nodejs-example-1-vqjfq                                                       1/1     Running     0          6m37s
test4                                                   ruby-ex-1-build                                                              0/1     Completed   0          10m
test4                                                   ruby-ex-6dbc586cd6-ppndj                                                     1/1     Running     0          9m34s
test4                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          10m
test4                                                   ruby-hello-world-757f7cd4bc-lm4d4                                            1/1     Running     0          5m52s
test5                                                   nodejs-example-1-build                                                       0/1     Completed   0          10m
test5                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          6m3s
test5                                                   nodejs-example-1-l4bkp                                                       1/1     Running     0          6m1s
test5                                                   ruby-ex-1-build                                                              0/1     Completed   0          9m41s
test5                                                   ruby-ex-8bbbbff84-8nrz7                                                      1/1     Running     0          8m29s
test5                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          9m49s
test5                                                   ruby-hello-world-c9fb65977-4tlp5                                             1/1     Running     0          5m26s
test6                                                   nodejs-example-1-build                                                       0/1     Completed   0          9m23s
test6                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          7m40s
test6                                                   nodejs-example-1-wfvth                                                       1/1     Running     0          7m38s
test6                                                   ruby-ex-1-build                                                              0/1     Completed   0          8m59s
test6                                                   ruby-ex-fccd9d9c4-9nzwj                                                      1/1     Running     0          5m55s
test6                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          9m9s
test6                                                   ruby-hello-world-859d5df784-zchmq                                            1/1     Running     0          7m27s
test7                                                   nodejs-example-1-4qmhp                                                       1/1     Running     0          6m56s
test7                                                   nodejs-example-1-build                                                       0/1     Completed   0          8m43s
test7                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          6m58s
test7                                                   ruby-ex-1-build                                                              0/1     Completed   0          8m12s
test7                                                   ruby-ex-869cf4b48b-b7vxj                                                     1/1     Running     0          5m33s
test7                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          8m30s
test7                                                   ruby-hello-world-679687d4f9-mzq76                                            1/1     Running     0          5m26s
test8                                                   nodejs-example-1-build                                                       0/1     Completed   0          7m51s
test8                                                   nodejs-example-1-deploy                                                      0/1     Completed   0          6m27s
test8                                                   nodejs-example-1-ts8r4                                                       1/1     Running     0          6m24s
test8                                                   ruby-ex-1-build                                                              0/1     Completed   0          7m30s
test8                                                   ruby-ex-678b56cbbc-9b7b4                                                     1/1     Running     0          6m26s
test8                                                   ruby-hello-world-1-build                                                     0/1     Completed   0          7m36s
test8                                                   ruby-hello-world-6bdc6f48ff-9gnmk                                            1/1     Running     0          5m13s


$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-08-23-194129 --allow-explicit-upgrade --force

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.18    True        True          11s     Working towards 4.4.0-0.nightly-2020-08-23-194129: 1% complete


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-23-194129   True        False         3h23m   Cluster version is 4.4.0-0.nightly-2020-08-23-194129

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-165809d19f1ceda06a28b2670245d285   True      False      False      3              3                   3                     0                      6h51m
worker   rendered-worker-31bdbc7f580511d6ab21054f30b3a390   True      False      False      3              3                   3                     0                      6h51m

$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h21m
cloud-credential                           4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h52m
cluster-autoscaler                         4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h35m
console                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h37m
csi-snapshot-controller                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h48m
dns                                        4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h48m
etcd                                       4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h45m
image-registry                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h33m
ingress                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h48m
insights                                   4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h36m
kube-apiserver                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h44m
kube-controller-manager                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h47m
kube-scheduler                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h47m
kube-storage-version-migrator              4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h48m
machine-api                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h36m
machine-config                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h28m
marketplace                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h36m
monitoring                                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h27m
network                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h50m
node-tuning                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      4h12m
openshift-apiserver                        4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h33m
openshift-controller-manager               4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h35m
openshift-samples                          4.4.0-0.nightly-2020-08-23-194129   True        False         False      4h11m
operator-lifecycle-manager                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h49m
operator-lifecycle-manager-catalog         4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h49m
operator-lifecycle-manager-packageserver   4.4.0-0.nightly-2020-08-23-194129   True        False         False      3h36m
service-ca                                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h50m
service-catalog-apiserver                  4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h50m
service-catalog-controller-manager         4.4.0-0.nightly-2020-08-23-194129   True        False         False      6h50m
storage                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      4h13m

$ oc get pods -A | grep ^test
test1                                                   cakephp-example-1-44xnb                                                      1/1     Running     0          3h54m
test1                                                   nodejs-example-1-x6n5d                                                       1/1     Running     0          3h54m
test1                                                   ruby-ex-8bd4c9987-8hrx4                                                      1/1     Running     0          3h51m
test1                                                   ruby-hello-world-785949b87b-9kkx6                                            1/1     Running     0          3h54m
test2                                                   nodejs-example-1-mztph                                                       1/1     Running     0          3h54m
test2                                                   ruby-ex-7b4b94c957-vqnwj                                                     1/1     Running     0          3h54m
test2                                                   ruby-hello-world-844647774d-76rvs                                            1/1     Running     0          3h54m
test3                                                   nodejs-example-1-d6d8v                                                       1/1     Running     0          3h54m
test3                                                   ruby-ex-fcfdb75f8-z97lf                                                      1/1     Running     0          3h54m
test3                                                   ruby-hello-world-5f7b77b454-cnk7t                                            1/1     Running     0          3h51m
test4                                                   nodejs-example-1-tc6p7                                                       1/1     Running     0          3h51m
test4                                                   ruby-ex-6dbc586cd6-p2wg5                                                     1/1     Running     0          3h54m
test4                                                   ruby-hello-world-757f7cd4bc-2bhjg                                            1/1     Running     0          3h51m
test5                                                   nodejs-example-1-d6ndr                                                       1/1     Running     0          3h51m
test5                                                   ruby-ex-8bbbbff84-fkzbh                                                      1/1     Running     0          3h54m
test5                                                   ruby-hello-world-c9fb65977-s4g7b                                             1/1     Running     0          3h54m
test6                                                   nodejs-example-1-6tvpp                                                       1/1     Running     0          3h54m
test6                                                   ruby-ex-fccd9d9c4-rz69v                                                      1/1     Running     0          3h54m
test6                                                   ruby-hello-world-859d5df784-nlh58                                            1/1     Running     0          3h54m
test7                                                   nodejs-example-1-7djj5                                                       1/1     Running     0          3h51m
test7                                                   ruby-ex-869cf4b48b-9mmzt                                                     1/1     Running     0          3h51m
test7                                                   ruby-hello-world-679687d4f9-zmstt                                            1/1     Running     0          3h51m
test8                                                   nodejs-example-1-d7gfz                                                       1/1     Running     0          3h51m
test8                                                   ruby-ex-678b56cbbc-n45l5                                                     1/1     Running     0          3h54m
test8                                                   ruby-hello-world-6bdc6f48ff-vr4zs                                            1/1     Running     0          3h51m

Comment 8 Ke Wang 2020-08-25 04:23:00 UTC
IPI install on GCP, upgrade from 4.3.33 to 4.4 nightly 4.4.0-0.nightly-2020-08-23-194129 succeeded with builds pods getting terminated.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.33    True        False         3h12m   Cluster version is 4.3.33

$ oc get nodes -o wide
NAME                                             STATUS   ROLES    AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
kewang-mn72b-m-0.c.openshift-qe.internal         Ready    master   68m   v1.16.2+295f6e6   10.0.0.3                    Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-m-1.c.openshift-qe.internal         Ready    master   68m   v1.16.2+295f6e6   10.0.0.6                    Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-m-2.c.openshift-qe.internal         Ready    master   68m   v1.16.2+295f6e6   10.0.0.4                    Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-w-a-p9jsz.c.openshift-qe.internal   Ready    worker   53m   v1.16.2+295f6e6   10.0.32.3                   Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-w-b-4w6q9.c.openshift-qe.internal   Ready    worker   53m   v1.16.2+295f6e6   10.0.32.4                   Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-w-c-g5kgv.c.openshift-qe.internal   Ready    worker   52m   v1.16.2+295f6e6   10.0.32.5                   Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
kewang-mn72b-w-f-vrfk4.c.openshift-qe.internal   Ready    worker   53m   v1.16.2+295f6e6   10.0.32.2                   Red Hat Enterprise Linux CoreOS 43.82.202008102153.0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8


$ oc adm policy add-scc-to-user privileged system:serviceaccount:test9:builder
$ oc adm policy add-scc-to-user privileged system:serviceaccount:test10:builder

$ oc new-app https://github.com/sclorg/cakephp-ex --name cakephp-ex-2 -n test9
$ oc new-app https://github.com/sclorg/cakephp-ex --name cakephp-ex-2 -n test10


$ oc get pods -A | grep ^test
test1                                                   cakephp-ex-1-build                                                  0/1     Completed   0          18m
test1                                                   cakephp-ex-69548d986f-whjh5                                         1/1     Running     0          16m
test1                                                   nodejs-ex-1-build                                                   0/1     Completed   0          17m
test1                                                   nodejs-ex-6fbd8c554-wqp6f                                           1/1     Running     0          16m
test1                                                   ruby-ex-1-build                                                     0/1     Completed   0          18m
test1                                                   ruby-ex-5b484dc8d4-d8tbs                                            1/1     Running     0          17m
test1                                                   ruby-hello-world-1-build                                            0/1     Completed   0          18m
test1                                                   ruby-hello-world-86b6b6c668-42r7v                                   1/1     Running     0          17m
test10                                                  cakephp-ex-1-build                                                  0/1     Completed   0          6m39s
test10                                                  cakephp-ex-2-1-build                                                0/1     Completed   0          114s
test10                                                  cakephp-ex-2-5b9b76cfc5-z5lns                                       1/1     Running     0          38s
test10                                                  cakephp-ex-6b94b76ff7-6pkzg                                         1/1     Running     0          5m15s
test10                                                  nodejs-ex-1-build                                                   0/1     Completed   0          6m15s
test10                                                  nodejs-ex-7cc4c77499-rglfr                                          1/1     Running     0          5m24s
test10                                                  ruby-ex-1-build                                                     0/1     Completed   0          7m10s
test10                                                  ruby-ex-fb4bf7648-frhff                                             1/1     Running     0          6m12s
test10                                                  ruby-hello-world-1-build                                            0/1     Completed   0          7m20s
test10                                                  ruby-hello-world-6fd8d85687-fzwj4                                   1/1     Running     0          5m56s
test2                                                   cakephp-ex-1-build                                                  0/1     Completed   0          16m
test2                                                   cakephp-ex-7987df7c67-lh8fg                                         1/1     Running     0          14m
test2                                                   nodejs-ex-1-build                                                   0/1     Completed   0          16m
test2                                                   nodejs-ex-7f6cc45667-w2mkf                                          1/1     Running     0          14m
test2                                                   ruby-ex-1-build                                                     0/1     Completed   0          17m
test2                                                   ruby-ex-556ff66dbc-jczks                                            1/1     Running     0          15m
test2                                                   ruby-hello-world-1-build                                            0/1     Completed   0          17m
test2                                                   ruby-hello-world-677fbbbf7-wzp9h                                    1/1     Running     0          15m
test3                                                   cakephp-ex-1-build                                                  0/1     Completed   0          15m
test3                                                   cakephp-ex-6b477d6d9b-npljs                                         1/1     Running     0          14m
test3                                                   nodejs-ex-1-build                                                   0/1     Completed   0          15m
test3                                                   nodejs-ex-7f75ffc7f5-9q4vv                                          1/1     Running     0          14m
test3                                                   ruby-ex-1-build                                                     0/1     Completed   0          16m
test3                                                   ruby-ex-5cff7b8cb8-9kmdm                                            1/1     Running     0          15m
test3                                                   ruby-hello-world-1-build                                            0/1     Completed   0          16m
test3                                                   ruby-hello-world-59899897c6-vxg5t                                   1/1     Running     0          14m
test4                                                   cakephp-ex-1-build                                                  0/1     Completed   0          14m
test4                                                   cakephp-ex-79c6644fbd-mcwpm                                         1/1     Running     0          12m
test4                                                   nodejs-ex-1-build                                                   0/1     Completed   0          14m
test4                                                   nodejs-ex-5d46cf5546-495wg                                          1/1     Running     0          13m
test4                                                   ruby-ex-1-build                                                     0/1     Completed   0          14m
test4                                                   ruby-ex-547745d4cf-q8xhz                                            1/1     Running     0          13m
test4                                                   ruby-hello-world-1-build                                            0/1     Completed   0          14m
test4                                                   ruby-hello-world-76747f4978-862n9                                   1/1     Running     0          13m
test5                                                   cakephp-ex-1-build                                                  0/1     Completed   0          13m
test5                                                   cakephp-ex-69694fb97d-8cdqx                                         1/1     Running     0          11m
test5                                                   nodejs-ex-1-build                                                   0/1     Completed   0          12m
test5                                                   nodejs-ex-98f587754-4kfds                                           1/1     Running     0          12m
test5                                                   ruby-ex-1-build                                                     0/1     Completed   0          13m
test5                                                   ruby-ex-55c67755b8-tp86t                                            1/1     Running     0          12m
test5                                                   ruby-hello-world-1-build                                            0/1     Completed   0          13m
test5                                                   ruby-hello-world-69757cbdf8-hgst5                                   1/1     Running     0          12m
test6                                                   cakephp-ex-1-build                                                  0/1     Completed   0          11m
test6                                                   cakephp-ex-6b9c5c45b8-psqcl                                         1/1     Running     0          9m37s
test6                                                   nodejs-ex-1-build                                                   0/1     Completed   0          11m
test6                                                   nodejs-ex-b5784587f-pvjd7                                           1/1     Running     0          9m39s
test6                                                   ruby-ex-1-build                                                     0/1     Completed   0          12m
test6                                                   ruby-ex-7796584c5b-zgm77                                            1/1     Running     0          11m
test6                                                   ruby-hello-world-1-build                                            0/1     Completed   0          12m
test6                                                   ruby-hello-world-6c7f698dcd-gllm5                                   1/1     Running     0          11m
test7                                                   cakephp-ex-1-build                                                  0/1     Completed   0          10m
test7                                                   cakephp-ex-5557d5fc6d-5mndc                                         1/1     Running     0          9m11s
test7                                                   nodejs-ex-1-build                                                   0/1     Completed   0          10m
test7                                                   nodejs-ex-75dd85675c-qx5pp                                          1/1     Running     0          9m1s
test7                                                   ruby-ex-1-build                                                     0/1     Completed   0          10m
test7                                                   ruby-ex-67767d4846-gd4bq                                            1/1     Running     0          9m19s
test7                                                   ruby-hello-world-1-build                                            0/1     Completed   0          11m
test7                                                   ruby-hello-world-868bc68667-7lp7j                                   1/1     Running     0          9m4s
test8                                                   cakephp-ex-1-build                                                  0/1     Completed   0          9m17s
test8                                                   cakephp-ex-668df4d665-hczsw                                         1/1     Running     0          7m43s
test8                                                   nodejs-ex-1-build                                                   0/1     Completed   0          8m53s
test8                                                   nodejs-ex-7545d946f6-rc68k                                          1/1     Running     0          8m4s
test8                                                   ruby-ex-1-build                                                     0/1     Completed   0          9m38s
test8                                                   ruby-ex-66b9f754d-qq5n6                                             1/1     Running     0          8m31s
test8                                                   ruby-hello-world-1-build                                            0/1     Completed   0          9m48s
test8                                                   ruby-hello-world-5b969974cc-sntfc                                   1/1     Running     0          8m32s
test9                                                   cakephp-ex-1-build                                                  0/1     Completed   0          8m1s
test9                                                   cakephp-ex-2-1-build                                                0/1     Completed   0          2m27s
test9                                                   cakephp-ex-2-c4f677c49-xmr8j                                        1/1     Running     0          64s
test9                                                   cakephp-ex-54c464c88b-g2mbp                                         1/1     Running     0          5m27s
test9                                                   nodejs-ex-1-build                                                   0/1     Completed   0          7m37s
test9                                                   nodejs-ex-687755cd8f-72gds                                          1/1     Running     0          6m46s
test9                                                   ruby-ex-1-build                                                     0/1     Completed   0          8m24s
test9                                                   ruby-ex-f67d4d495-nffbw                                             1/1     Running     0          7m27s
test9                                                   ruby-hello-world-1-build                                            0/1     Completed   0          8m33s
test9                                                   ruby-hello-world-76c4cb6f85-2fw55                                   1/1     Running     0          7m32s

$ oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-08-23-194129 --allow-explicit-upgrade --force

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-23-194129   True        False         67s     Cluster version is 4.4.0-0.nightly-2020-08-23-194129

$  oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-680059c127c8d118a8bfb3bda5f4b376   True      False      False      3              3                   3                     0                      156m
worker   rendered-worker-6dd5a27ef727e9b200429ffeaec9aac2   True      False      False      4              4                   4                     0                      156m

$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      135m
cloud-credential                           4.4.0-0.nightly-2020-08-23-194129   True        False         False      158m
cluster-autoscaler                         4.4.0-0.nightly-2020-08-23-194129   True        False         False      145m
console                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      39m
csi-snapshot-controller                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      46m
dns                                        4.4.0-0.nightly-2020-08-23-194129   True        False         False      155m
etcd                                       4.4.0-0.nightly-2020-08-23-194129   True        False         False      80m
image-registry                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      141m
ingress                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      33m
insights                                   4.4.0-0.nightly-2020-08-23-194129   True        False         False      146m
kube-apiserver                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      153m
kube-controller-manager                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      77m
kube-scheduler                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      78m
kube-storage-version-migrator              4.4.0-0.nightly-2020-08-23-194129   True        False         False      34m
machine-api                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      145m
machine-config                             4.4.0-0.nightly-2020-08-23-194129   True        False         False      155m
marketplace                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      38m
monitoring                                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      138m
network                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      157m
node-tuning                                4.4.0-0.nightly-2020-08-23-194129   True        False         False      42m
openshift-apiserver                        4.4.0-0.nightly-2020-08-23-194129   True        False         False      40m
openshift-controller-manager               4.4.0-0.nightly-2020-08-23-194129   True        False         False      156m
openshift-samples                          4.4.0-0.nightly-2020-08-23-194129   True        False         False      66m
operator-lifecycle-manager                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      146m
operator-lifecycle-manager-catalog         4.4.0-0.nightly-2020-08-23-194129   True        False         False      146m
operator-lifecycle-manager-packageserver   4.4.0-0.nightly-2020-08-23-194129   True        False         False      39m
service-ca                                 4.4.0-0.nightly-2020-08-23-194129   True        False         False      156m
service-catalog-apiserver                  4.4.0-0.nightly-2020-08-23-194129   True        False         False      152m
service-catalog-controller-manager         4.4.0-0.nightly-2020-08-23-194129   True        False         False      146m
storage                                    4.4.0-0.nightly-2020-08-23-194129   True        False         False      70m

$ oc get nodes -o wide
NAME                                             STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
kewang-mn72b-m-0.c.openshift-qe.internal         Ready    master   165m   v1.17.1+bb2b64d   10.0.0.3                    Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-m-1.c.openshift-qe.internal         Ready    master   165m   v1.17.1+bb2b64d   10.0.0.6                    Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-m-2.c.openshift-qe.internal         Ready    master   165m   v1.17.1+bb2b64d   10.0.0.4                    Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-w-a-p9jsz.c.openshift-qe.internal   Ready    worker   150m   v1.17.1+bb2b64d   10.0.32.3                   Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-w-b-4w6q9.c.openshift-qe.internal   Ready    worker   150m   v1.17.1+bb2b64d   10.0.32.4                   Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-w-c-g5kgv.c.openshift-qe.internal   Ready    worker   149m   v1.17.1+bb2b64d   10.0.32.5                   Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8
kewang-mn72b-w-f-vrfk4.c.openshift-qe.internal   Ready    worker   150m   v1.17.1+bb2b64d   10.0.32.2                   Red Hat Enterprise Linux CoreOS 44.82.202008220030-0 (Ootpa)   4.18.0-193.14.3.el8_2.x86_64   cri-o://1.17.5-3.rhaos4.4.git75b5183.el8

$ oc get pods -A | grep ^test
test1                                                   cakephp-ex-69548d986f-xwm85                                         1/1     Running     0          35m
test1                                                   nodejs-ex-6fbd8c554-kxdxk                                           1/1     Running     0          47m
test1                                                   ruby-ex-5b484dc8d4-tnq64                                            1/1     Running     0          47m
test1                                                   ruby-hello-world-86b6b6c668-n2pb2                                   1/1     Running     0          35m
test10                                                  cakephp-ex-2-5b9b76cfc5-rhqsz                                       1/1     Running     0          37m
test10                                                  cakephp-ex-6b94b76ff7-288rf                                         1/1     Running     0          37m
test10                                                  nodejs-ex-7cc4c77499-kvv5m                                          1/1     Running     0          35m
test10                                                  ruby-ex-fb4bf7648-9h24x                                             1/1     Running     0          37m
test10                                                  ruby-hello-world-6fd8d85687-4gd7z                                   1/1     Running     0          37m
test2                                                   cakephp-ex-7987df7c67-ng2v6                                         1/1     Running     0          35m
test2                                                   nodejs-ex-7f6cc45667-hf26s                                          1/1     Running     0          47m
test2                                                   ruby-ex-556ff66dbc-6zxwq                                            1/1     Running     0          47m
test2                                                   ruby-hello-world-677fbbbf7-2mnmq                                    1/1     Running     0          35m
test3                                                   cakephp-ex-6b477d6d9b-x4cgz                                         1/1     Running     0          35m
test3                                                   nodejs-ex-7f75ffc7f5-n7jrw                                          1/1     Running     0          47m
test3                                                   ruby-ex-5cff7b8cb8-c64lt                                            1/1     Running     0          35m
test3                                                   ruby-hello-world-59899897c6-p4w9x                                   1/1     Running     0          47m
test4                                                   cakephp-ex-79c6644fbd-sccc6                                         1/1     Running     0          47m
test4                                                   nodejs-ex-5d46cf5546-ggsqw                                          1/1     Running     0          37m
test4                                                   ruby-ex-547745d4cf-cjc57                                            1/1     Running     0          35m
test4                                                   ruby-hello-world-76747f4978-vdrs5                                   1/1     Running     0          37m
test5                                                   cakephp-ex-69694fb97d-6d98h                                         1/1     Running     0          37m
test5                                                   nodejs-ex-98f587754-d8vt6                                           1/1     Running     0          35m
test5                                                   ruby-ex-55c67755b8-bnrz7                                            1/1     Running     0          35m
test5                                                   ruby-hello-world-69757cbdf8-nkznf                                   1/1     Running     0          35m
test6                                                   cakephp-ex-6b9c5c45b8-rq7pw                                         1/1     Running     0          47m
test6                                                   nodejs-ex-b5784587f-zcp92                                           1/1     Running     0          47m
test6                                                   ruby-ex-7796584c5b-mv2g9                                            1/1     Running     0          37m
test6                                                   ruby-hello-world-6c7f698dcd-qb8j7                                   1/1     Running     0          37m
test7                                                   cakephp-ex-5557d5fc6d-ddzv7                                         1/1     Running     0          37m
test7                                                   nodejs-ex-75dd85675c-rtt6r                                          1/1     Running     0          37m
test7                                                   ruby-ex-67767d4846-xdwx2                                            1/1     Running     0          47m
test7                                                   ruby-hello-world-868bc68667-r8vbj                                   1/1     Running     0          35m
test8                                                   cakephp-ex-668df4d665-977js                                         1/1     Running     0          37m
test8                                                   nodejs-ex-7545d946f6-f564r                                          1/1     Running     0          35m
test8                                                   ruby-ex-66b9f754d-r7kck                                             1/1     Running     0          35m
test8                                                   ruby-hello-world-5b969974cc-fl7kg                                   1/1     Running     0          47m
test9                                                   cakephp-ex-2-c4f677c49-gzgzg                                        1/1     Running     0          37m
test9                                                   cakephp-ex-54c464c88b-8vj58                                         1/1     Running     0          37m
test9                                                   nodejs-ex-687755cd8f-kgl5t                                          1/1     Running     0          47m
test9                                                   ruby-ex-f67d4d495-sk9x5                                             1/1     Running     0          37m
test9                                                   ruby-hello-world-76c4cb6f85-nxbmp                                   1/1     Running     0          47m

Comment 10 errata-xmlrpc 2020-09-01 19:41:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.19 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3514


Note You need to log in before you can comment on or make changes to this bug.