1948443 – OCP 4.8 nightly still showing v1.20 even after 1.21 merge

Bug 1948443 - OCP 4.8 nightly still showing v1.20 even after 1.21 merge

Summary: OCP 4.8 nightly still showing v1.20 even after 1.21 merge

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	RHCOS
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Micah Abbott
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-04-12 07:58 UTC by Sunil Choudhary
Modified:	2021-07-27 22:59 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-07-27 22:59:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 22:59:46 UTC

Description Sunil Choudhary 2021-04-12 07:58:43 UTC

# oc get no
NAME                                         STATUS   ROLES           AGE    VERSION
ip-10-0-160-188.us-east-2.compute.internal   Ready    master,worker   119m   v1.20.0+5f82cdb

# oc version
Client Version: 4.8.0-0.nightly-2021-04-08-200632
Server Version: 4.8.0-0.nightly-2021-04-08-200632
Kubernetes Version: v1.21.0-rc.0+6d27558

Comment 1 Mike Fiedler 2021-04-12 13:05:25 UTC

Setting this to high severity.   Looks like kubelet has not been re-based yet as of feature freeze:

sh-4.4# kubelet --version
Kubernetes v1.20.0+5f82cdb


This blocks performance and scalability testing for 4.8.

Comment 4 Micah Abbott 2021-04-12 16:27:44 UTC

RHCOS builds have been impacted by an internal build cluster outage last week and now we are trying to land RHEL 8.4 Beta, which is complicating things.

We should have new RHCOS 4.8 builds soon.

Comment 5 Mike Fiedler 2021-04-12 17:18:37 UTC

crio is 1.21, just kubelet is 1.20

[root@ip-172-31-18-207 ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-12-091111   True        False         3h13m   Cluster version is 4.8.0-0.nightly-2021-04-12-091111

# oc debug node/mffiedler412b-t2zvk-worker-a-mhpfh.c.openshift-qe.internal -- chroot /host crio --version
Starting pod/mffiedler412b-t2zvk-worker-a-mhpfhcopenshift-qeinternal-debug ...
To use host binaries, run `chroot /host`
time="2021-04-12T17:15:51Z" level=info msg="Starting CRI-O, version: 1.21.0-53.rhaos4.8.gitec8716f.el8-dev, git: ()"
crio version 1.21.0-53.rhaos4.8.gitec8716f.el8-dev
Version:    1.21.0-53.rhaos4.8.gitec8716f.el8-dev
GoVersion:  go1.16.1
Compiler:   gc
Platform:   linux/amd64
Linkmode:   dynamic

Comment 6 Micah Abbott 2021-04-14 13:19:00 UTC

We finally got an RHCOS 4.8 build with the 1.21 kubelet made - 48.84.202104131355-0

But the promotion job has been failing due to an issue with the `aws` client used in the promote job.  Fix incoming - https://github.com/openshift/release/pull/17735

Comment 7 Micah Abbott 2021-04-14 19:38:22 UTC

We had successful promotion of the RHCOS build into 4.8.0-0.ci-2021-04-14-132500

Comment 9 Sunil Choudhary 2021-04-19 07:06:19 UTC

Verified on 4.8.0-0.nightly-2021-04-18-015132

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-18-015132   True        False         23h     Cluster version is 4.8.0-0.nightly-2021-04-18-015132

$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE   VERSION                INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION          CONTAINER-RUNTIME
ip-10-0-141-221.us-east-2.compute.internal   Ready    worker   23h   v1.21.0-rc.0+2993be8   10.0.141.221   <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
ip-10-0-155-3.us-east-2.compute.internal     Ready    master   24h   v1.21.0-rc.0+2993be8   10.0.155.3     <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
ip-10-0-164-32.us-east-2.compute.internal    Ready    master   24h   v1.21.0-rc.0+2993be8   10.0.164.32    <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
ip-10-0-179-155.us-east-2.compute.internal   Ready    worker   23h   v1.21.0-rc.0+2993be8   10.0.179.155   <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
ip-10-0-199-69.us-east-2.compute.internal    Ready    master   23h   v1.21.0-rc.0+2993be8   10.0.199.69    <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
ip-10-0-204-59.us-east-2.compute.internal    Ready    worker   23h   v1.21.0-rc.0+2993be8   10.0.204.59    <none>        Red Hat Enterprise Linux CoreOS 48.84.202104171300-0 (Ootpa)   4.18.0-293.el8.x86_64   cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8

sh-4.4# journalctl -u crio | grep -i "Starting CRI-O"
Apr 18 07:11:17 ip-10-0-141-221 crio[1324]: time="2021-04-18 07:11:17.230633239Z" level=info msg="Starting CRI-O, version: 1.21.0-74.rhaos4.8.gitbc1ef35.el8, git: ()"
Apr 18 12:03:48 ip-10-0-141-221 crio[1377]: time="2021-04-18 12:03:48.748597390Z" level=info msg="Starting CRI-O, version: 1.21.0-74.rhaos4.8.gitbc1ef35.el8, git: ()"

sh-4.4# crio version
INFO[2021-04-19 07:05:36.593562373Z] Starting CRI-O, version: 1.21.0-74.rhaos4.8.gitbc1ef35.el8, git: () 
Version:    1.21.0-74.rhaos4.8.gitbc1ef35.el8
GoVersion:  go1.16.1
Compiler:   gc
Platform:   linux/amd64
Linkmode:   dynamic
sh-4.4#

Comment 12 errata-xmlrpc 2021-07-27 22:59:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.