Bug 1917315 - localvolumeset-local-provisoner-xxx pods are not killed after upgrading from 4.6 to 4.7
Summary: localvolumeset-local-provisoner-xxx pods are not killed after upgrading from ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.7.0
Assignee: Rohan CJ
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-18 10:00 UTC by Chao Yang
Modified: 2021-02-24 15:54 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:53:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift local-storage-operator pull 201 0 None closed Bug 1917315: delete localvolumeset-local-provisoner daemonset and fix PV name hashing 2021-02-19 11:34:08 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:54:11 UTC

Description Chao Yang 2021-01-18 10:00:16 UTC
Description of problem:
localvolumeset-local-provisoner-xxx pods are not killed after upgrading from 4.6 to 4.7

Version-Release number of selected component (if applicable):
local-storage-operator.4.7.0-202101160343.p0
4.7.0-0.nightly-2021-01-17-211555

How reproducible:
Always

Steps to Reproduce:
1.Deploy LocalStorageOperator 4.6
2.Create localvolumeset
3.Update sub channel from 4.6 to 4.7
oc get csv
NAME                                           DISPLAY         VERSION                 REPLACES                                       PHASE
local-storage-operator.4.7.0-202101160343.p0   Local Storage   4.7.0-202101160343.p0   local-storage-operator.4.6.0-202012161211.p0   Succeeded

4.oc get pods
NAME                                      READY   STATUS    RESTARTS   AGE
diskmaker-manager-44hgj                   1/1     Running   0          35m
diskmaker-manager-rgrfr                   1/1     Running   0          35m
diskmaker-manager-zh6zg                   1/1     Running   0          34m
local-storage-operator-65c547b864-79cmf   1/1     Running   0          35m
localvolumeset-local-provisioner-cpmcz    1/1     Running   0          55m
localvolumeset-local-provisioner-vdq9d    1/1     Running   0          55m
localvolumeset-local-provisioner-x876g    1/1     Running   0          55m

5.Attach volumes to node, then two pv are provisioned for one volume.
oc get pv 
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
local-pv-8e75ec0a   3Gi        RWO            Delete           Available           lvs                     13m
local-pv-ffcf8f0d   3Gi        RWO            Delete           Available           lvs                     13m
oc get pv local-pv-8e75ec0a -o json | jq .spec
{
  "accessModes": [
    "ReadWriteOnce"
  ],
  "capacity": {
    "storage": "3Gi"
  },
  "local": {
    "path": "/mnt/local-storage/lvs/nvme-Amazon_Elastic_Block_Store_vol0d20c598aa513418f"
  },

oc get pv local-pv-ffcf8f0d -o json | jq .spec
{
  "accessModes": [
    "ReadWriteOnce"
  ],
  "capacity": {
    "storage": "3Gi"
  },
  "local": {
    "path": "/mnt/local-storage/lvs/nvme-Amazon_Elastic_Block_Store_vol0d20c598aa513418f"
  },


Actual results:
pods localvolumeset-local-provisioner-xxx not killed and two pv are provisioned for one volume.

Expected results:
Pods localvolumeset-local-provisioner-xxx should be killed

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Hemant Kumar 2021-01-18 17:31:23 UTC
This looks like a pretty serious regression and could cause problems. I am going to mark this as a blocker bug.

Comment 2 Rohan CJ 2021-01-19 07:32:05 UTC
I see how this would happen. Working on it!

Comment 4 Qin Ping 2021-02-05 09:58:50 UTC
Verified with: local-storage-operator.4.7.0-202102041842.p0

$ oc get csv -n openshift-local-storage
NAME                                           DISPLAY         VERSION                 REPLACES   PHASE
local-storage-operator.4.6.0-202102031949.p0   Local Storage   4.6.0-202102031949.p0              Succeeded

$ oc get pod -n openshift-local-storage
NAME                                     READY   STATUS    RESTARTS   AGE
diskmaker-manager-465b8                  1/1     Running   0          3m54s
diskmaker-manager-tdpxs                  1/1     Running   0          3m54s
diskmaker-manager-v9slm                  1/1     Running   0          72s
local-storage-operator-57fd6597d-fbwwn   1/1     Running   0          8m41s
localvolumeset-local-provisioner-f6t5p   1/1     Running   0          63s
localvolumeset-local-provisioner-fvdfw   1/1     Running   0          3m54s
localvolumeset-local-provisioner-pb9v5   1/1     Running   0          3m54s

$ oc get csv -n openshift-local-storage
NAME                                           DISPLAY         VERSION                 REPLACES                                       PHASE
local-storage-operator.4.7.0-202102041842.p0   Local Storage   4.7.0-202102041842.p0   local-storage-operator.4.6.0-202102031949.p0   Succeeded

$ oc get pod -n openshift-local-storage
NAME                                      READY   STATUS    RESTARTS   AGE
diskmaker-manager-5lgqc                   1/1     Running   0          66s
diskmaker-manager-c8jmd                   1/1     Running   0          44s
diskmaker-manager-pg9rk                   1/1     Running   0          95s
local-storage-operator-866bdfb465-k2z6s   1/1     Running   0          2m45s

$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                STORAGECLASS   REASON   AGE
local-pv-7f182123                          5Gi        RWO            Delete           Available                        lvs-test                61s
pvc-72195945-f003-451e-831d-3913d224fe11   5Gi        RWO            Delete           Bound       default/test-pvc-5   gp2-test                4m28s

Comment 7 errata-xmlrpc 2021-02-24 15:53:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.