Bug 2019053 - DV with immediate bind remains in WaitForFirstConsumer
Summary: DV with immediate bind remains in WaitForFirstConsumer
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.10.0
Assignee: Alexander Wels
QA Contact: Yan Du
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-01 14:23 UTC by Ruth Netser
Modified: 2022-03-16 15:56 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-16 15:56:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt hostpath-provisioner-operator pull 175 0 None open Allow usage of PVCTemplate as storage pool source. 2021-12-01 19:53:39 UTC
Red Hat Product Errata RHSA-2022:0947 0 None None None 2022-03-16 15:56:49 UTC

Description Ruth Netser 2021-11-01 14:23:42 UTC
Description of problem:
Create an HPP DV with cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
The DV remains in WaitForFirstConsumer
Operator log indicates a failure to create ClusterRole

Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\"


Version-Release number of selected component (if applicable):
CNV 4.10.0
hostpath-provisioner-operator version is: v4.10.0-20


How reproducible:
100%

Steps to Reproduce:
1. Create DV with cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
2.
3.

Actual results:
The DV remains in WaitForFirstConsumer


Expected results:
DV import should start

Additional info:


apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
  name: dv
  namespace: openshift-virtualization-os-images
spec:
  contentType: kubevirt
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 3Gi
    storageClassName: hostpath-provisioner
    volumeMode: Filesystem
  source:
    http:
      url: http://cnv-qe-server.rhevdev.lab.eng.rdu2.redhat.com/files/cnv-tests/cirros-images/cirros-0.4.0-x86_64-disk.qcow2


$ oc logs -n openshift-cnv hostpath-provisioner-operator-55dcf7bff-rcszg 
{"level":"info","ts":1635761841.125201,"logger":"cmd","msg":"Go Version: go1.15.14"}
{"level":"info","ts":1635761841.1252306,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1635761841.1252406,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
{"level":"info","ts":1635761841.1261609,"logger":"leader","msg":"Trying to become the leader."}
I1101 10:17:22.218026       1 request.go:668] Waited for 1.026817136s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/cloudcredential.openshift.io/v1?timeout=32s
{"level":"info","ts":1635761844.3838923,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1635761844.3883207,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1635761847.597953,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1635761847.598355,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1635761847.6993663,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.6994932,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.6995149,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.699533,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.6995492,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.699567,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1635761847.6995852,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting Controller"}
{"level":"info","ts":1635761847.8016884,"logger":"controller-runtime.manager.controller.hostpathprovisioner-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1635762252.6834092,"logger":"controller_hostpathprovisioner","msg":"Reconciling CSI and legacy controller plugin","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.683439,"logger":"controller_hostpathprovisioner","msg":"Adding deletion Finalizer","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.6960826,"logger":"controller_hostpathprovisioner","msg":"Started deploying","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.6974447,"logger":"controller_hostpathprovisioner","msg":"Creating a new DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Namespace":"openshift-cnv","Daemonset.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.720548,"logger":"KubeAPIWarningLogger","msg":"would violate \"latest\" version of \"baseline\" PodSecurity profile: hostPath volumes (volume \"pv-volume\")"}
{"level":"info","ts":1635762252.7207282,"logger":"controller_hostpathprovisioner","msg":"Creating a new DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Namespace":"openshift-cnv","Daemonset.Name":"hostpath-provisioner-csi"}
{"level":"info","ts":1635762252.728906,"logger":"KubeAPIWarningLogger","msg":"would violate \"latest\" version of \"baseline\" PodSecurity profile: hostPath volumes (volumes \"csi-data-dir\", \"socket-dir\", \"mountpoint-dir\", \"registration-dir\", \"plugins-dir\"), privileged (containers \"hostpath-provisioner\", \"node-driver-registrar\", \"csi-snapshotter\", \"csi-provisioner\" must not set securityContext.privileged=true)"}
{"level":"info","ts":1635762252.7291372,"logger":"controller_hostpathprovisioner","msg":"Creating a new Service Account","Request.Namespace":"","Request.Name":"hostpath-provisioner","ServiceAccount.Namespace":"openshift-cnv","ServiceAccount.Name":"hostpath-provisioner-admin"}
{"level":"info","ts":1635762252.7331536,"logger":"controller_hostpathprovisioner","msg":"Creating a new Service Account","Request.Namespace":"","Request.Name":"hostpath-provisioner","ServiceAccount.Namespace":"openshift-cnv","ServiceAccount.Name":"hostpath-provisioner-admin-csi"}
{"level":"info","ts":1635762252.737785,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.7964823,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner-admin-csi"}
{"level":"error","ts":1635762252.8410783,"logger":"controller_hostpathprovisioner","msg":"Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:openshift-cnv\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents\"], Verbs:[\"create\" \"get\" \"list\" \"watch\" \"update\" \"delete\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents/status\"], Verbs:[\"update\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshots\"], Verbs:[\"get\"]}","stacktrace":"kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/remote-source/app/pkg/controller/hostpathprovisioner/controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"}
{"level":"info","ts":1635762252.8412113,"logger":"controller_hostpathprovisioner","msg":"Reconciling CSI and legacy controller plugin","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.8476086,"logger":"controller_hostpathprovisioner","msg":"Started deploying","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
{"level":"info","ts":1635762252.8499458,"logger":"controller_hostpathprovisioner","msg":"DIFF","Request.Namespace":"","Request.Name":"hostpath-provisioner","obj":{"apiVersion":"apps/v1","kind":"DaemonSet","namespace":"openshift-cnv","name":"hostpath-provisioner-csi"},"patch":"[{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/4/terminationMessagePolicy\"},{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/4/terminationMessagePath\"}]"}
{"level":"info","ts":1635762252.849972,"logger":"controller_hostpathprovisioner","msg":"Updating DaemonSet","Request.Namespace":"","Request.Name":"hostpath-provisioner","DaemonSet.Name":"hostpath-provisioner-csi"}
{"level":"info","ts":1635762252.857567,"logger":"controller_hostpathprovisioner.Provisioner RBAC","msg":"Creating a new ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","ClusterRole.Name":"hostpath-provisioner-admin-csi"}
{"level":"error","ts":1635762252.8875806,"logger":"controller_hostpathprovisioner","msg":"Unable to create ClusterRole","Request.Namespace":"","Request.Name":"hostpath-provisioner","error":"clusterroles.rbac.authorization.k8s.io \"hostpath-provisioner-admin-csi\" is forbidden: user \"system:serviceaccount:openshift-cnv:hostpath-provisioner-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:openshift-cnv\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents\"], Verbs:[\"create\" \"get\" \"list\" \"watch\" \"update\" \"delete\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshotcontents/status\"], Verbs:[\"update\" \"patch\"]}\n{APIGroups:[\"snapshot.storage.k8s.io\"], Resources:[\"volumesnapshots\"], Verbs:[\"get\"]}","stacktrace":"kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile\n\t/remote-source/app/pkg/controller/hostpathprovisioner/controller.go:280\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"}

Comment 1 Alexander Wels 2021-12-01 19:53:40 UTC
Looks like the csv generator didn't provide the correct permissions for the operator to create the cluster role. The linked PR fixes that.

Comment 2 Yan Du 2021-12-08 03:17:45 UTC
Test on CNV-4.10.0-431, hostpath provisioner v4.10.0-38, issue have been fixed.

Events:
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  Pending           2m12s  datavolume-controller  PVC dv Pending
  Normal  ImportScheduled   2m10s  datavolume-controller  Import into dv scheduled
  Normal  Bound             2m10s  datavolume-controller  PVC dv Bound
  Normal  ImportInProgress  92s    datavolume-controller  Import into dv in progress
  Normal  ImportSucceeded   89s    datavolume-controller  Successfully imported into PVC dv

$ oc get dv
NAME   PHASE       PROGRESS   RESTARTS   AGE
dv     Succeeded   100.0%                95s

Comment 7 errata-xmlrpc 2022-03-16 15:56:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947


Note You need to log in before you can comment on or make changes to this bug.