Bug 1923243

Summary:	Error when deleting an instance of hostpath provisioner
Product:	Container Native Virtualization (CNV)	Reporter:	David Critch <dcritch>
Component:	Storage	Assignee:	Alex Kalenyuk <akalenyu>
Status:	CLOSED ERRATA	QA Contact:	Kevin Alon Goldblatt <kgoldbla>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.5.3	CC:	akalenyu, alitke, awels, cnv-qe-bugs, yadu, ycui
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	hostpath-provisioner-rhel8-operator v4.8.0-11	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 14:23:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Critch 2021-02-01 15:37:07 UTC

Description of problem:
I created an instance of the hostpath provisioner with an incorrect directory, so I went to delete it and create the proper one. The delete hung due to the hostpath SA not having permission to delete certain OpenShift objects.

Version-Release number of selected component (if applicable):
OpenShift Virtualization 2.5.3
OpenShift 4.6.13

How reproducible:
Always

Steps to Reproduce:
1. Create a CR for HostPathProvisioner [https://github.com/kubevirt/hostpath-provisioner-operator#custom-resource-cr]
2. Delete instance of HostPathProvisioner


Actual results:
The delete command hangs and never cleans up

The hostpath SA can not delete the following resources:
"msg":"Deleting  SecurityContextConstraint"  "Request.Namespace":""  "Request.Name":"kubevirt-hostpath-provisioner"
"msg":"Deleting  SecurityContextConstraint"  "Request.Namespace":""  "Request.Name":"kubevirt-hostpath-provisioner"
"msg":"Deleting  ClusterRoleBinding"         "Request.Namespace":""  "Request.Name":"kubevirt-hostpath-provisioner"
"msg":"Deleting  ClusterRole"                "Request.Namespace":""  "Request.Name":"kubevirt-hostpath-provisioner"



Expected results:
The hostpath provisioner resource is deleted and the pods deleted.

Additional info:
The delete gets stuck on permissions:

{"level":"info","ts":1612053034.3282533,"logger":"controller_hostpathprovisioner","msg":"Deleting SecurityContextConstraint","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","SecurityContextConstraints":"kubevirt-hostpath-provisioner"}
{"level":"error","ts":1612053034.330249,"logger":"controller_hostpathprovisioner","msg":"Unable to delete SecurityContextConstraints","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","error":"securitycontextconstraints.security.openshift.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"securitycontextconstraints\" in API group \"security.openshift.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:174\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1612053034.3302917,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"hostpathprovisioner-controller","request":"/kubevirt-hostpath-provisioner","error":"securitycontextconstraints.security.openshift.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"securitycontextconstraints\" in API group \"security.openshift.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}


I first manually deleted the SCC to clear it, then it moved on to a new error deleting the clusterrolebinding:

{"level":"error","ts":1612053280.09424,"logger":"controller_hostpathprovisioner","msg":"Unable to delete ClusterRoleBinding","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","error":"clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:181\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1612053280.0942893,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"hostpathprovisioner-controller","request":"/kubevirt-hostpath-provisioner","error":"clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}


At which point I temporarily gave the SA cluster-admin privileges so it could do its thing:

{"level":"info","ts":1612053607.7744071,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.77446,"logger":"controller_hostpathprovisioner","msg":"Deleting SecurityContextConstraint","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","SecurityContextConstraints":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.7745147,"logger":"controller_hostpathprovisioner","msg":"Deleting ClusterRoleBinding","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","ClusterRoleBinding":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.7793744,"logger":"controller_hostpathprovisioner","msg":"Deleting ClusterRole","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","ClusterRole":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.7933538,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.8156378,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"}
{"level":"info","ts":1612053607.8162339,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"}


I then revoked the cluster-admin privileges. The cleanup was then successful.

Comment 6 Adam Litke 2021-05-12 18:13:11 UTC

Alex do you have an update on this bug?  Did we decide to resolve it by allowing users to use a different CR name?

Comment 7 Alex Kalenyuk 2021-05-13 08:55:08 UTC

Yes, we decided to resolve it by allowing users to use a different CR name, attached the PR and moving to POST

Comment 8 Kevin Alon Goldblatt 2021-06-09 11:02:18 UTC

Verified with the following code:
------------------------------------------------------
 oc version
Client Version: 4.8.0-fc.5
Server Version: 4.8.0-fc.5
Kubernetes Version: v1.21.0-rc.0+88a3e8c
[cnv-qe-jenkins@stg03-kevin-zwzbq-executor kevin]$ oc get csv -n openshift-cnv
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v4.8.0   OpenShift Virtualization   4.8.0     kubevirt-hyperconverged-operator.v2.6.2   Succeeded


Verified with the following scenario:
------------------------------------------------------
1. Create a custom resource hpp with the name custom-hostpath-provisioner - successfully created and pods are running
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: custom-hostpath-provisioner
spec:
  pathConfig:
    path: "/var/hpvolumes"
  imagePullPolicy: IfNotPresent

oc get pods -A |grep host
openshift-cnv                                      hostpath-provisioner-85grn                                        1/1     Running             0          3s
openshift-cnv                                      hostpath-provisioner-hgbv7                                        0/1     ContainerCreating   0          3s
openshift-cnv                                      hostpath-provisioner-operator-85d8d846c4-bjshw                    1/1     Running             0          6d20h
openshift-cnv                                      hostpath-provisioner-vmg8p                                        0/1     ContainerCreating   0          3s


2. Delete the hpp resource 
oc delete hpp custom-hostpath-provisioner - successfully deleted and all the pods are deleted

oc delete hpp custom-hostpath-provisioner 
hostpathprovisioner.hostpathprovisioner.kubevirt.io "custom-hostpath-provisioner" deleted

[cnv-qe-jenkins@stg03-kevin-zwzbq-executor kevin]$ oc get pods -A |grep host
openshift-cnv                                      hostpath-provisioner-operator-85d8d846c4-bjshw                    1/1     Running       0          6d20h


Moving to VERIFIED!

Comment 11 errata-xmlrpc 2021-07-27 14:23:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920