Description of problem: I created an instance of the hostpath provisioner with an incorrect directory, so I went to delete it and create the proper one. The delete hung due to the hostpath SA not having permission to delete certain OpenShift objects. Version-Release number of selected component (if applicable): OpenShift Virtualization 2.5.3 OpenShift 4.6.13 How reproducible: Always Steps to Reproduce: 1. Create a CR for HostPathProvisioner [https://github.com/kubevirt/hostpath-provisioner-operator#custom-resource-cr] 2. Delete instance of HostPathProvisioner Actual results: The delete command hangs and never cleans up The hostpath SA can not delete the following resources: "msg":"Deleting SecurityContextConstraint" "Request.Namespace":"" "Request.Name":"kubevirt-hostpath-provisioner" "msg":"Deleting SecurityContextConstraint" "Request.Namespace":"" "Request.Name":"kubevirt-hostpath-provisioner" "msg":"Deleting ClusterRoleBinding" "Request.Namespace":"" "Request.Name":"kubevirt-hostpath-provisioner" "msg":"Deleting ClusterRole" "Request.Namespace":"" "Request.Name":"kubevirt-hostpath-provisioner" Expected results: The hostpath provisioner resource is deleted and the pods deleted. Additional info: The delete gets stuck on permissions: {"level":"info","ts":1612053034.3282533,"logger":"controller_hostpathprovisioner","msg":"Deleting SecurityContextConstraint","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","SecurityContextConstraints":"kubevirt-hostpath-provisioner"} {"level":"error","ts":1612053034.330249,"logger":"controller_hostpathprovisioner","msg":"Unable to delete SecurityContextConstraints","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","error":"securitycontextconstraints.security.openshift.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"securitycontextconstraints\" in API group \"security.openshift.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:174\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"error","ts":1612053034.3302917,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"hostpathprovisioner-controller","request":"/kubevirt-hostpath-provisioner","error":"securitycontextconstraints.security.openshift.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"securitycontextconstraints\" in API group \"security.openshift.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} I first manually deleted the SCC to clear it, then it moved on to a new error deleting the clusterrolebinding: {"level":"error","ts":1612053280.09424,"logger":"controller_hostpathprovisioner","msg":"Unable to delete ClusterRoleBinding","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","error":"clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner/controller.go:181\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"error","ts":1612053280.0942893,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"hostpathprovisioner-controller","request":"/kubevirt-hostpath-provisioner","error":"clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-hostpath-provisioner\" is forbidden: User \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" cannot delete resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope","stacktrace":"kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nkubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nkubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/kubevirt.io/hostpath-provisioner-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} At which point I temporarily gave the SA cluster-admin privileges so it could do its thing: {"level":"info","ts":1612053607.7744071,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.77446,"logger":"controller_hostpathprovisioner","msg":"Deleting SecurityContextConstraint","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","SecurityContextConstraints":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.7745147,"logger":"controller_hostpathprovisioner","msg":"Deleting ClusterRoleBinding","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","ClusterRoleBinding":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.7793744,"logger":"controller_hostpathprovisioner","msg":"Deleting ClusterRole","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner","ClusterRole":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.7933538,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.8156378,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"} {"level":"info","ts":1612053607.8162339,"logger":"controller_hostpathprovisioner","msg":"Reconciling HostPathProvisioner","Request.Namespace":"","Request.Name":"kubevirt-hostpath-provisioner"} I then revoked the cluster-admin privileges. The cleanup was then successful.
Alex do you have an update on this bug? Did we decide to resolve it by allowing users to use a different CR name?
Yes, we decided to resolve it by allowing users to use a different CR name, attached the PR and moving to POST
Verified with the following code: ------------------------------------------------------ oc version Client Version: 4.8.0-fc.5 Server Version: 4.8.0-fc.5 Kubernetes Version: v1.21.0-rc.0+88a3e8c [cnv-qe-jenkins@stg03-kevin-zwzbq-executor kevin]$ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.8.0 OpenShift Virtualization 4.8.0 kubevirt-hyperconverged-operator.v2.6.2 Succeeded Verified with the following scenario: ------------------------------------------------------ 1. Create a custom resource hpp with the name custom-hostpath-provisioner - successfully created and pods are running apiVersion: hostpathprovisioner.kubevirt.io/v1beta1 kind: HostPathProvisioner metadata: name: custom-hostpath-provisioner spec: pathConfig: path: "/var/hpvolumes" imagePullPolicy: IfNotPresent oc get pods -A |grep host openshift-cnv hostpath-provisioner-85grn 1/1 Running 0 3s openshift-cnv hostpath-provisioner-hgbv7 0/1 ContainerCreating 0 3s openshift-cnv hostpath-provisioner-operator-85d8d846c4-bjshw 1/1 Running 0 6d20h openshift-cnv hostpath-provisioner-vmg8p 0/1 ContainerCreating 0 3s 2. Delete the hpp resource oc delete hpp custom-hostpath-provisioner - successfully deleted and all the pods are deleted oc delete hpp custom-hostpath-provisioner hostpathprovisioner.hostpathprovisioner.kubevirt.io "custom-hostpath-provisioner" deleted [cnv-qe-jenkins@stg03-kevin-zwzbq-executor kevin]$ oc get pods -A |grep host openshift-cnv hostpath-provisioner-operator-85d8d846c4-bjshw 1/1 Running 0 6d20h Moving to VERIFIED!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2920