Bug 2041530 - HPP CSI CR can't be deleted if it's a combination of a basic storage pool, and a pvcTemplate
Summary: HPP CSI CR can't be deleted if it's a combination of a basic storage pool, an...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.10.0
Assignee: Alexander Wels
QA Contact: Jenia Peimer
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-17 15:48 UTC by Jenia Peimer
Modified: 2022-03-16 16:06 UTC (History)
3 users (show)

Fixed In Version: cnv v4.10, hco v4.10.0-595, hostpath-provisioner-operator v4.10.0-56
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-16 16:06:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt hostpath-provisioner-operator pull 211 0 None Merged Fix issue with basic and pvcTemplate causing npe when removing hpp cr 2022-01-23 09:17:31 UTC
Github kubevirt hostpath-provisioner-operator pull 212 0 None Merged [release-v0.12] Fix issue with basic and pvcTemplate causing npe when removing hpp cr 2022-01-17 20:20:41 UTC
Red Hat Product Errata RHSA-2022:0947 0 None None None 2022-03-16 16:06:39 UTC

Description Jenia Peimer 2022-01-17 15:48:37 UTC
Description of problem:
HostPathProvisioner CSI custom resource can't be deleted if it's a combination of a basic storage pool, and a pvcTemplate. This causes the hostpath-provisioner-operator to CrashLoopBackOff.

We are NOT facing this issue in HPP CR with just a basic storage pool, or in HPP CR with just a pvcTemplate.

Version-Release number of selected component (if applicable):
4.10

How reproducible:
Always

Steps to Reproduce:
1. Get a cnv 4.10 cluster
2. Deploy an HPP CR with both basic storage pool and pvcTemplate
3. Delete HPP CR

Actual results:
HPP CR is not deleted, hostpath-provisioner-operator - CrashLoopBackOff.

Expected results:
HPP CR deleted.

Additional info:

HPP CR YAML:

$ cat hpp-cr-basic-pvc-block.yaml
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  storagePools:
      - name: local-basic
        path: "/var/local-basic"
      - name: local-pvc-template
        pvcTemplate:
          volumeMode: Block
          storageClassName: ocs-storagecluster-ceph-rbd
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 40Gi
        path: "/var/hpvolumes"
  workload:
   nodeSelector:
     kubernetes.io/os: linux



$ oc delete HostPathProvisioner hostpath-provisioner
hostpathprovisioner.hostpathprovisioner.kubevirt.io "hostpath-provisioner" deleted
^C
$
$ oc get pods -A | grep hostpath
openshift-cnv                                      hostpath-provisioner-csi-2z2jq                                    4/4     Running            0               8m33s
openshift-cnv                                      hostpath-provisioner-csi-cmlxz                                    4/4     Running            0               8m33s
openshift-cnv                                      hostpath-provisioner-csi-tggh6                                    4/4     Running            0               8m33s
openshift-cnv                                      hostpath-provisioner-operator-79c988456c-5mnqf                    0/1     CrashLoopBackOff   5 (64s ago)     21m


$ oc logs -n openshift-cnv hostpath-provisioner-operator-79c988456c-5mnqf -p
{"level":"info","ts":1642427183.4980357,"logger":"cmd","msg":"Go Version: go1.16.6"}
{"level":"info","ts":1642427183.498063,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1642427183.4980674,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
I0117 13:46:24.549470       1 request.go:668] Waited for 1.032877849s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s
{"level":"info","ts":1642427187.3572679,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1642427187.3657196,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1642427187.3664596,"logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, admission.Defaulter interface is not implemented","GVK":"hostpathprovisioner.kubevirt.io/v1beta1, Kind=HostPathProvisioner"}
{"level":"info","ts":1642427187.366523,"logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"hostpathprovisioner.kubevirt.io/v1beta1, Kind=HostPathProvisioner","path":"/validate-hostpathprovisioner-kubevirt-io-v1beta1-hostpathprovisioner"}
{"level":"info","ts":1642427187.3666127,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-hostpathprovisioner-kubevirt-io-v1beta1-hostpathprovisioner"}
{"level":"info","ts":1642427187.3666918,"logger":"cmd","msg":"Starting the Cmd."}
I0117 13:46:27.366945       1 leaderelection.go:243] attempting to acquire leader lease openshift-cnv/hostpath-provisioner-operator-lock...
{"level":"info","ts":1642427187.366945,"msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1642427187.3670144,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1642427187.3749099,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1642427187.375009,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1642427187.3757684,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I0117 13:46:44.468614       1 leaderelection.go:253] successfully acquired lease openshift-cnv/hostpath-provisioner-operator-lock
{"level":"info","ts":1642427204.4702606,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703252,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703326,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.470341,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703543,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703605,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703777,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703848,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4703944,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704046,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704115,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704182,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704297,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704354,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.4704466,"logger":"controller.hostpathprovisioner-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1642427204.470458,"logger":"controller.hostpathprovisioner-controller","msg":"Starting Controller"}
{"level":"info","ts":1642427204.5743701,"logger":"controller.hostpathprovisioner-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1642427204.574589,"logger":"controller_hostpathprovisioner","msg":"Detected CSI CR, reconciling CSI only","Request.Namespace":"","Request.Name":"hostpath-provisioner"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x155adb2]
 
goroutine 672 [running]:
kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).storagePoolDeploymentByNode(0xc00030c480, 0x1abed68, 0xc001355d80, 0xc00072f600, 0xc00123f6b0, 0xc000046030, 0xd, 0xc0006f2000, 0xc0006f2000)
        /remote-source/app/pkg/controller/hostpathprovisioner/storagepool.go:330 +0x252
kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).cleanDeployments(0xc00030c480, 0x1abed68, 0xc001355d80, 0xc00072f600, 0xc000046030, 0xd, 0x14, 0x1ad8350)
        /remote-source/app/pkg/controller/hostpathprovisioner/storagepool.go:116 +0x1ee
kubevirt.io/hostpath-provisioner-operator/pkg/controller/hostpathprovisioner.(*ReconcileHostPathProvisioner).Reconcile(0xc00030c480, 0x1ab61f8, 0xc00055b1a0, 0x0, 0x0, 0xc0005b10b0, 0x14, 0xc00055b1a0, 0xc00055b110, 0xc00123fdb0, ...)
        /remote-source/app/pkg/controller/hostpathprovisioner/controller.go:308 +0x40b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0001cab40, 0x1ab61f8, 0xc00055b110, 0x0, 0x0, 0xc0005b10b0, 0x14, 0xc00055b100, 0x0, 0x0, ...)
        /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114 +0x247
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cab40, 0x1ab6150, 0xc0007bc000, 0x1728440, 0xc0002c8180)
        /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311 +0x305
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cab40, 0x1ab6150, 0xc0007bc000, 0xc00093a700)
        /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2(0xc000673c60, 0xc0001cab40, 0x1ab6150, 0xc0007bc000)
        /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x6b
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x425

Comment 1 Jenia Peimer 2022-01-18 18:48:32 UTC
Verified on CNV v4.10, HCO v4.10.0-595, hostpath-provisioner-operator v4.10.0-56

$ oc get HostPathProvisioner
NAME                   AGE
hostpath-provisioner   4s

$ oc get pods -A | grep hostpath
openshift-cnv      hostpath-provisioner-csi-5mdj5                    4/4     Running   0  16s
openshift-cnv      hostpath-provisioner-csi-7hhvf                    4/4     Running   0  16s
openshift-cnv      hostpath-provisioner-csi-wvw65                    4/4     Running   0  16s
openshift-cnv      hostpath-provisioner-operator-64b5dbf66c-9fjnl    1/1     Running   0  3h26m

$ oc get pods -A | grep hpp
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-8sbqc   1/1     Running     0   36s
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-jcrbh   1/1     Running     0   36s
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-vgczp   1/1     Running     0   36s

$ oc get deployments -A | grep hpp
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-97lsq   1/1     1  1   75s
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-szklx   1/1     1  1   75s
openshift-cnv      hpp-pool-local-pvc-template-c01-jp410-fr10-zwx7t-worker-0-z6rkq   1/1     1  1   75s

$ oc delete HostPathProvisioner hostpath-provisioner 
hostpathprovisioner.hostpathprovisioner.kubevirt.io "hostpath-provisioner" deleted

$ oc get HostPathProvisioner
No resources found

$ oc get deployments -A | grep hpp 
$ oc get pods -A | grep hpp
$ oc get pods -A | grep hostpath
openshift-cnv    hostpath-provisioner-operator-64b5dbf66c-9fjnl      1/1     Running     0      3h28m

Comment 6 errata-xmlrpc 2022-03-16 16:06:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947


Note You need to log in before you can comment on or make changes to this bug.