Bug 1992244

Summary:	hostpath-provisioner Pods fail to pull hostpath-provisioner-csi image
Product:	Container Native Virtualization (CNV)	Reporter:	Denis Ollier <dollierp>
Component:	Storage	Assignee:	Alexander Wels <awels>
Status:	CLOSED DUPLICATE	QA Contact:	Natalie Gavrielov <ngavrilo>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.10.0	CC:	cnv-qe-bugs, mrashish, ngavrilo, yadu
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-14 09:50:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Denis Ollier 2021-08-10 19:51:15 UTC

Note
----

This issue can be observed only after fixing or applying a workaround for BZ#1992231.

Description of problem
----------------------

After creating a HostPathProvisioner CustomResource, hostpath-provisioner Pods are stuck in ImagePullBackOff state.

> kubectl -n openshift-cnv get pods -l k8s-app=hostpath-provisioner
> 
> NAME                         READY   STATUS             RESTARTS   AGE
> hostpath-provisioner-7dfb4   4/5     ImagePullBackOff   0          9m46s
> hostpath-provisioner-dwbgk   4/5     ImagePullBackOff   0          9m46s
> hostpath-provisioner-qr2w4   4/5     ImagePullBackOff   0          9m46s

They are failing to pull the hostpath-provisioner-csi image. 

>   Warning  Failed          69s (x2 over 80s)  kubelet            Error: ErrImagePull
>   Warning  Failed          69s (x2 over 80s)  kubelet            Failed to pull image "hostpath-provisioner-csi": rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/hostpath-provisioner-csi: errors:
> denied: requested access to the resource is denied
> unauthorized: authentication required
>   Warning  Failed   69s                kubelet  Error: ImagePullBackOff

Analysis
--------

Support for CSI has been added to hostpath-provisioner-operator by upstream PR https://github.com/kubevirt/hostpath-provisioner-operator/pull/113.

Downstream build chain need to be updated accordingly. 

Among others:
- hostpath-provisioner-csi image needs to be built downstream
- CSI_PROVISIONER_IMAGE variable needs to be set to point to this image.

Version
-------
OCP: 4.9.0-0.nightly
CNV: http://cnv-version-explorer.apps.cnv.engineering.redhat.com/BundleDetails?ver=v4.9.0-89

Comment 1 Denis Ollier 2021-08-12 11:49:22 UTC

Mitigated for CNV-4.9.0 by https://code.engineering.redhat.com/gerrit/c/hostpath-provisioner-operator/+/264197.

Included since http://cnv-version-explorer.apps.cnv.engineering.redhat.com/BundleDetails?ver=v4.9.0-103.

Lowering the severity since it will be fully addressed for CNV-v4.10.

Comment 2 Yan Du 2021-08-16 09:36:53 UTC

The feature should be fully addressed in CNV 4.10, so change the version to 4.10
If the issue can not be reproduced on 4.10, feel free to close it.

Comment 3 Maya Rashish 2021-10-13 09:17:38 UTC

the workaround is specific to 4.9, the work for 4.10 is not done yet.

Comment 4 Maya Rashish 2021-11-14 09:50:30 UTC

Prefer to track "HPP v4.10 fails to deploy downstream" with a single bug.
This bug describes an earlier failure mode that should be resolved by now, too.

*** This bug has been marked as a duplicate of bug 1992231 ***