Bug 2011961

Summary:	[4.9] [tracker] Storage operator is not available after reboot cluster instances
Product:	OpenShift Container Platform	Reporter:	Benjamin Gilbert <bgilbert>
Component:	RHCOS	Assignee:	Micah Abbott <miabbott>
Status:	CLOSED ERRATA	QA Contact:	Michael Nguyen <mnguyen>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.9	CC:	aos-bugs, bgilbert, ccoleman, dhellmann, dornelas, ehashman, fbertina, jima, jligon, miabbott, mifiedle, mrussell, nstielau, pehunt, prkumar, rphillips, schoudha, smilner, tsze, wking, yliu1, yunjiang
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2011050	Environment:
Last Closed:	2021-10-18 17:52:32 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2009653, 2011050, 2011960
Bug Blocks:

Comment 1 Micah Abbott 2021-10-08 19:16:33 UTC

The 4.9 version of the BZ this is tracking - https://bugzilla.redhat.com/show_bug.cgi?id=2011050

Links to https://github.com/openshift/kubernetes/pull/1000

Which landed the merge commit 9d56495c8de8c2864d9a951c2da70318d1e41f3b

In the most recent RHCOS 4.9 build for x86_64 (49.84.202110081407-0), it contains `openshift-hyperkube-4.9.0-202110080828.p0.git.894a78b.assembly.stream.el8`

The short commit id in the NVR (894a78b) is newer than the commit that PR#1000 merged.

Additionally, an inspection of the `kubelet` itself on a running RHCOS node confirms the kubelet is from the newer commit:

```
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
* ostree://3093d4596a48e37c9926dc53240af084c077e3bf2063ef2a8d8a81421b6e9987
                   Version: 49.84.202110081407-0 (2021-10-08T14:10:58Z)
[core@cosa-devsh ~]$ /usr/bin/kubelet --version
Kubernetes v1.22.0-rc.0+894a78b
```

Comment 2 RHCOS Bug Bot 2021-10-08 19:21:21 UTC

This bug has been reported fixed in a new RHCOS build and is ready for QE verification.  To mark the bug verified, set the Verified field to Tested.  This bug will automatically move to MODIFIED once the fix has landed in a new bootimage.

Comment 3 RHCOS Bug Bot 2021-10-08 21:18:25 UTC

The fix for this bug has landed in a bootimage bump, as tracked in bug 2009653 (now in status MODIFIED).  Moving this bug to MODIFIED.

Comment 6 Yunfei Jiang 2021-10-09 04:04:15 UTC

Bug 2011050 has been resolved, move this bug to VERIFIED.


rhcos: 49.84.202110081407-0 (https://github.com/openshift/installer/commit/2a75f9bd56df413d726e5b9b98dbfbbc0884ff0f)
OCP: 4.9.0-rc.7

sh-4.4# rpm-ostree status
State: idle
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaaf9590ed870930313bb0275c6f87c012f64f6a5e784b119921b20369af783d
              CustomOrigin: Managed by machine-config-operator
                   Version: 49.84.202110081407-0 (2021-10-08T14:10:58Z)

  ostree://3093d4596a48e37c9926dc53240af084c077e3bf2063ef2a8d8a81421b6e9987
                   Version: 49.84.202110081407-0 (2021-10-08T14:10:58Z)
sh-4.4# kubelet --version
Kubernetes v1.22.0-rc.0+894a78b

Comment 8 errata-xmlrpc 2021-10-18 17:52:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759