Bug 1755000

Summary:	kubelet version skew causing CI flakes
Product:	OpenShift Container Platform	Reporter:	Michal Fojtik <mfojtik>
Component:	Storage	Assignee:	Hemant Kumar <hekumar>
Status:	CLOSED ERRATA	QA Contact:	Liang Xia <lxia>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.3.0	CC:	aos-bugs, aos-storage-staff, hekumar, jsafrane, rkrawitz, sdodson, wking
Target Milestone:	---
Target Release:	4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-13 21:25:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michal Fojtik 2019-09-24 14:01:26 UTC

Description of problem:

[k8s.io] Variable Expansion should allow substituting values in a volume subpath [sig-storage][NodeFeature:VolumeSubpathEnvExpansion] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI Volume expansion should expand volume by restarting pod if attach=off, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI Volume expansion should expand volume by restarting pod if attach=on, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI online volume expansion should expand volume without restarting pod if attach=off, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI online volume expansion should expand volume without restarting pod if attach=on, nodeExpansion=on [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI workload information using mock driver contain ephemeral=true when using inline volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] CSI mock volume CSI workload information using mock driver should be passed when podInfoOnMount=true [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it [Suite:openshift/conformance/parallel] [Suite:k8s]

These are very frequent now because the kubernetes API server and controller manager are on 1.16.0 but the kubelet is still 1.14.x.

These should be resolved automatically when the kubelet is updated.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 2 Jan Safranek 2019-09-24 14:08:16 UTC

Full logs: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_oc/102/pull-ci-openshift-oc-master-e2e-aws/308

It looks like external-attacher expects CSINode is filled:

CSINode ip-10-0-137-224.ec2.internal does not contain driver csi-mock-e2e-csi-mock-volumes-5403`

This will be fixed when RHCOS gets new kubelet, however, we should check that there is no bug in the attacher, it *should* work with older kubelet, so we don't break 3rd party CSI drivers during update.

Comment 7 Liang Xia 2019-10-22 02:19:27 UTC

All the tests mentioned in #comment 0 are passed in below jobs, 
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.3/1042
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.3/1041
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.3/1040

Job 1042 passed with 4.3.0-0.ci-2019-10-21-234502

Comment 9 errata-xmlrpc 2020-05-13 21:25:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062