Description of problem: Create an HPP DV with cdi.kubevirt.io/storage.bind.immediate.requested: 'true' The DV remains in WaitForFirstConsumer Operator log indicates a failure to create ClusterRole Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" Version-Release number of selected component (if applicable): CNV 4.10.0 hostpath-provisioner-operator version is: v4.10.0-20 How reproducible: 100% Steps to Reproduce: 1. Create DV with cdi.kubevirt.io/storage.bind.immediate.requested: 'true' 2. 3. Actual results: The DV remains in WaitForFirstConsumer Expected results: DV import should start Additional info: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: annotations: cdi.kubevirt.io/storage.bind.immediate.requested: 'true' name: dv namespace: openshift-virtualization-os-images spec: contentType: kubevirt pvc: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi storageClassName: hostpath-provisioner volumeMode: Filesystem source: http: url: http://cnv-qe-server.rhevdev.lab.eng.rdu2.redhat.com/files/cnv-tests/cirros-images/cirros-0.4.0-x86_64-disk.qcow2 $ oc logs -n openshift-cnv hostpath-provisioner-operator-55dcf7bff-rcszg {"level":"info","ts":1635761841.125201,"logger":"cmd","msg":"Go Version: go1.15.14"} {"level":"info","ts":1635761841.1252306,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1635761841.1252406,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"} {"level":"info","ts":1635761841.1261609,"logger":"leader","msg":"Trying to become the leader."} I1101 10:17:22.218026 1 request.go:668] Waited for 1.026817136s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/cloudcredential.openshift.io/v1?timeout=32s {"level":"info","ts":1635761844.3838923,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1635761844.3883207,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1635761847.597953,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1635761847.598355,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1635761847.6993663,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.6994932,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.6995149,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.699533,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.6995492,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.699567,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1635761847.6995852,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting Controller"} {"level":"info","ts":1635761847.8016884,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting workers","worker count":1} {"level":"info","ts":1635762252.6834092,"logger":"controller_hostpathprovisioner","msg":"Reconciling CSI and legacy controller plugin","Request.Namespace":"","Request.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.683439,"logger":"controller_hostpathprovisioner","msg":"Adding deletion Finalizer","Request.Namespace":"","Request.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.6960826,"logger":"controller_hostpathprovisioner","msg":"Started deploying","Request.Namespace":"","Request.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.6974447,"logger":"controller_hostpathprovisioner","msg":"Creating a new DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Namespace":"openshift-cnv","Daemonset.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.720548,"logger":"KubeAPIWarningLogger","msg":"would violate \"latest\" version of \"baseline\" PodSecurity profile: hostPath volumes (volume \"pv-volume\")"} {"level":"info","ts":1635762252.7207282,"logger":"controller_hostpathprovisioner","msg":"Creating a new DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Namespace":"openshift-cnv","Daemonset.Name":"hostpath-provisioner-csi"} {"level":"info","ts":1635762252.728906,"logger":"KubeAPIWarningLogger","msg":"would violate \"latest\" version of \"baseline\" PodSecurity profile: hostPath volumes (volumes \"csi-data-dir\", \"socket-dir\", \"mountpoint-dir\", \"registration-dir\", \"plugins-dir\"), privileged (containers \"hostpath-provisioner\", \"node-driver-registrar\", \"csi-snapshotter\", \"csi-provisioner\" must not set securityContext.privileged=true)"} {"level":"info","ts":1635762252.7291372,"logger":"controller_hostpathprovisioner","msg":"Creating a new Service Account","Request.Namespace":"","Request.Name":"hostpath-provisioner","ServiceAccount.Namespace":"openshift-cnv","ServiceAccount.Name":"hostpath-provisioner-admin"} {"level":"info","ts":1635762252.7331536,"logger":"controller_hostpathprovisioner","msg":"Creating a new Service Account","Request.Namespace":"","Request.Name":"hostpath-provisioner","ServiceAccount.Namespace":"openshift-cnv","ServiceAccount.Name":"hostpath-provisioner-admin-csi"} {"level":"info","ts":1635762252.737785,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.7964823,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner-admin-csi"} {"level":"error","ts":1635762252.8410783,"logger":"controller_hostpathprovisioner","msg":"Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:openshift-cnv\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents\"], Verbs:[\"create\" \"get\" \"list\" \"watch\" \"update\" \"delete\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents/status\"], Verbs:[\"update\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshots\"], Verbs:[\"get\"]}","stacktrace":"kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/remote-source/app/pkg/controller/hostpathprovisioner/controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1635762252.8412113,"logger":"controller_hostpathprovisioner","msg":"Reconciling CSI and legacy controller plugin","Request.Namespace":"","Request.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.8476086,"logger":"controller_hostpathprovisioner","msg":"Started deploying","Request.Namespace":"","Request.Name":"hostpath-provisioner"} {"level":"info","ts":1635762252.8499458,"logger":"controller_hostpathprovisioner","msg":"DIFF","Request.Namespace":"","Request.Name":"hostpath-provisioner","obj":{"apiVersion":"apps/v1","kind":"DaemonSet","namespace":"openshift-cnv","name":"hostpath-provisioner-csi"},"patch":"[{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/4/terminationMessagePolicy\"},{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/4/terminationMessagePath\"}]"} {"level":"info","ts":1635762252.849972,"logger":"controller_hostpathprovisioner","msg":"Updating DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Name":"hostpath-provisioner-csi"} {"level":"info","ts":1635762252.857567,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner-admin-csi"} {"level":"error","ts":1635762252.8875806,"logger":"controller_hostpathprovisioner","msg":"Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:openshift-cnv\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents\"], Verbs:[\"create\" \"get\" \"list\" \"watch\" \"update\" \"delete\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents/status\"], Verbs:[\"update\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshots\"], Verbs:[\"get\"]}","stacktrace":"kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/remote-source/app/pkg/controller/hostpathprovisioner/controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"}
Looks like the csv generator didn't provide the correct permissions for the operator to create the cluster role. The linked PR fixes that.
Test on CNV-4.10.0-431, hostpath provisioner v4.10.0-38, issue have been fixed. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pending 2m12s datavolume-controller PVC dv Pending Normal ImportScheduled 2m10s datavolume-controller Import into dv scheduled Normal Bound 2m10s datavolume-controller PVC dv Bound Normal ImportInProgress 92s datavolume-controller Import into dv in progress Normal ImportSucceeded 89s datavolume-controller Successfully imported into PVC dv $ oc get dv NAME PHASE PROGRESS RESTARTS AGE dv Succeeded 100.0% 95s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947