Bug 1917315
| Summary: | localvolumeset-local-provisoner-xxx pods are not killed after upgrading from 4.6 to 4.7 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> |
| Component: | Storage | Assignee: | Rohan CJ <rojoseph> |
| Storage sub component: | Local Storage Operator | QA Contact: | Qin Ping <piqin> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | aos-bugs, hekumar, piqin, pmuller, sapillai |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 15:53:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This looks like a pretty serious regression and could cause problems. I am going to mark this as a blocker bug. I see how this would happen. Working on it! Verified with: local-storage-operator.4.7.0-202102041842.p0 $ oc get csv -n openshift-local-storage NAME DISPLAY VERSION REPLACES PHASE local-storage-operator.4.6.0-202102031949.p0 Local Storage 4.6.0-202102031949.p0 Succeeded $ oc get pod -n openshift-local-storage NAME READY STATUS RESTARTS AGE diskmaker-manager-465b8 1/1 Running 0 3m54s diskmaker-manager-tdpxs 1/1 Running 0 3m54s diskmaker-manager-v9slm 1/1 Running 0 72s local-storage-operator-57fd6597d-fbwwn 1/1 Running 0 8m41s localvolumeset-local-provisioner-f6t5p 1/1 Running 0 63s localvolumeset-local-provisioner-fvdfw 1/1 Running 0 3m54s localvolumeset-local-provisioner-pb9v5 1/1 Running 0 3m54s $ oc get csv -n openshift-local-storage NAME DISPLAY VERSION REPLACES PHASE local-storage-operator.4.7.0-202102041842.p0 Local Storage 4.7.0-202102041842.p0 local-storage-operator.4.6.0-202102031949.p0 Succeeded $ oc get pod -n openshift-local-storage NAME READY STATUS RESTARTS AGE diskmaker-manager-5lgqc 1/1 Running 0 66s diskmaker-manager-c8jmd 1/1 Running 0 44s diskmaker-manager-pg9rk 1/1 Running 0 95s local-storage-operator-866bdfb465-k2z6s 1/1 Running 0 2m45s $ oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-7f182123 5Gi RWO Delete Available lvs-test 61s pvc-72195945-f003-451e-831d-3913d224fe11 5Gi RWO Delete Bound default/test-pvc-5 gp2-test 4m28s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |
Description of problem: localvolumeset-local-provisoner-xxx pods are not killed after upgrading from 4.6 to 4.7 Version-Release number of selected component (if applicable): local-storage-operator.4.7.0-202101160343.p0 4.7.0-0.nightly-2021-01-17-211555 How reproducible: Always Steps to Reproduce: 1.Deploy LocalStorageOperator 4.6 2.Create localvolumeset 3.Update sub channel from 4.6 to 4.7 oc get csv NAME DISPLAY VERSION REPLACES PHASE local-storage-operator.4.7.0-202101160343.p0 Local Storage 4.7.0-202101160343.p0 local-storage-operator.4.6.0-202012161211.p0 Succeeded 4.oc get pods NAME READY STATUS RESTARTS AGE diskmaker-manager-44hgj 1/1 Running 0 35m diskmaker-manager-rgrfr 1/1 Running 0 35m diskmaker-manager-zh6zg 1/1 Running 0 34m local-storage-operator-65c547b864-79cmf 1/1 Running 0 35m localvolumeset-local-provisioner-cpmcz 1/1 Running 0 55m localvolumeset-local-provisioner-vdq9d 1/1 Running 0 55m localvolumeset-local-provisioner-x876g 1/1 Running 0 55m 5.Attach volumes to node, then two pv are provisioned for one volume. oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-8e75ec0a 3Gi RWO Delete Available lvs 13m local-pv-ffcf8f0d 3Gi RWO Delete Available lvs 13m oc get pv local-pv-8e75ec0a -o json | jq .spec { "accessModes": [ "ReadWriteOnce" ], "capacity": { "storage": "3Gi" }, "local": { "path": "/mnt/local-storage/lvs/nvme-Amazon_Elastic_Block_Store_vol0d20c598aa513418f" }, oc get pv local-pv-ffcf8f0d -o json | jq .spec { "accessModes": [ "ReadWriteOnce" ], "capacity": { "storage": "3Gi" }, "local": { "path": "/mnt/local-storage/lvs/nvme-Amazon_Elastic_Block_Store_vol0d20c598aa513418f" }, Actual results: pods localvolumeset-local-provisioner-xxx not killed and two pv are provisioned for one volume. Expected results: Pods localvolumeset-local-provisioner-xxx should be killed Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: