Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2104258

Summary: Multiple PV are missing after upgrade from OCP from 4.7.43 to 4.8.39
Product: OpenShift Container Platform Reporter: Swati Mulje <smulje>
Component: StorageAssignee: Hemant Kumar <hekumar>
Storage sub component: Local Storage Operator QA Contact: Chao Yang <chaoyang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: hekumar, pewang, wduan
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-09 12:52:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2105453    
Bug Blocks:    

Description Swati Mulje 2022-07-05 19:25:51 UTC
Description of problem:

In order to upgrade the cluster, CU has uninstalled the application using helm. The only thing helm is doing is removing the claim (PVC). It is expected that PVC will be removed but not PV. 

but PV is missing after the cluster upgrade. 

Version-Release number of selected component (if applicable):
OCP 4.8.39


How reproducible:

CU has done test: upgraded cluster 4.7.43 - > 4.8.39 and fresh install 4.8.39. Behaviour is different

vlan145 - upgraded cluster 4.7.43 - > 4.8.39
vlan136 - fresh install 4.8.39

Here is the upgraded cluster

(bastion-145):(admin) 09:39:48 UTC BASTION-1.5.2
# oc get pvc
NAME                  STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS           AGE
logsapp-fluentd-0     Bound    local-pv-a73d74e3   4Gi        RWO            local-application-sc   11h
logsapp-fluentd-1     Bound    local-pv-93b0af9e   4Gi        RWO            local-application-sc   11h
logsapp-fluentd-2     Bound    local-pv-234d2ec9   4Gi        RWO            local-application-sc   11h
logsinfra-fluentd-0   Bound    local-pv-6dd19f3    4Gi        RWO            local-platform-sc      11h
logsinfra-fluentd-1   Bound    local-pv-9db1b17c   4Gi        RWO            local-platform-sc      11h
logsinfra-fluentd-2   Bound    local-pv-3f79c065   4Gi        RWO            local-platform-sc      11h

(bastion-145):(admin) 09:39:48 UTC BASTION-1.5.2
# oc delete pvc --all
persistentvolumeclaim "logsapp-fluentd-0" deleted
persistentvolumeclaim "logsapp-fluentd-1" deleted
persistentvolumeclaim "logsapp-fluentd-2" deleted
persistentvolumeclaim "logsinfra-fluentd-0" deleted
persistentvolumeclaim "logsinfra-fluentd-1" deleted
persistentvolumeclaim "logsinfra-fluentd-2" deleted

(bastion-145):(admin) 09:41:10 UTC BASTION-1.5.2
# oc get pv | grep -E  'local-pv-a73d74e3|local-pv-93b0af9e|local-pv-234d2ec9|local-pv-6dd19f3|local-pv-9db1b17c|local-pv-3f79c065'
local-pv-234d2ec9                          4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-2                                   local-application-sc            11h
local-pv-3f79c065                          4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-2                                 local-platform-sc               11h
local-pv-6dd19f3                           4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-0                                 local-platform-sc               11h
local-pv-93b0af9e                          4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-1                                   local-application-sc            11h
local-pv-9db1b17c                          4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-1                                 local-platform-sc               11h
local-pv-a73d74e3                          4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-0                                   local-application-sc            11h

(bastion-145):(admin) 09:42:27 UTC BASTION-1.5.2
# oc get pv | grep -E  'local-pv-a73d74e3|local-pv-93b0af9e|local-pv-234d2ec9|local-pv-6dd19f3|local-pv-9db1b17c|local-pv-3f79c065'

(bastion-145):(admin) 09:42:32 UTC BASTION-1.5.2
#
Here is the fresh install


(bastion-136):(admin) 09:54:31 UTC BASTION-1.5.1
# oc get pvc
NAME                  STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS           AGE
logsapp-fluentd-0     Bound    local-pv-dde7a22b   4Gi        RWO            local-application-sc   10d
logsapp-fluentd-1     Bound    local-pv-b088f0ae   4Gi        RWO            local-application-sc   10d
logsapp-fluentd-2     Bound    local-pv-fdce9e3d   4Gi        RWO            local-application-sc   10d
logsinfra-fluentd-0   Bound    local-pv-db289f43   4Gi        RWO            local-platform-sc      10d
logsinfra-fluentd-1   Bound    local-pv-28857ff4   4Gi        RWO            local-platform-sc      10d
logsinfra-fluentd-2   Bound    local-pv-c7e77c39   4Gi        RWO            local-platform-sc      10d

(bastion-136):(admin) 09:54:46 UTC BASTION-1.5.1
# oc delete pvc --all
persistentvolumeclaim "logsapp-fluentd-0" deleted
persistentvolumeclaim "logsapp-fluentd-1" deleted
persistentvolumeclaim "logsapp-fluentd-2" deleted
persistentvolumeclaim "logsinfra-fluentd-0" deleted
persistentvolumeclaim "logsinfra-fluentd-1" deleted
persistentvolumeclaim "logsinfra-fluentd-2" deleted

(bastion-136):(admin) 09:54:54 UTC BASTION-1.5.1
# oc get pv | grep -E 'local-pv-dde7a22b|local-pv-b088f0ae|local-pv-fdce9e3d|local-pv-db289f43|local-pv-28857ff4|local-pv-c7e77c39'
local-pv-28857ff4   4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-1                                 local-platform-sc               10d
local-pv-b088f0ae   4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-1                                   local-application-sc            10d
local-pv-c7e77c39   4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-2                                 local-platform-sc               10d
local-pv-db289f43   4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-0                                 local-platform-sc               10d
local-pv-dde7a22b   4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-0                                   local-application-sc            10d
local-pv-fdce9e3d   4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-2                                   local-application-sc            10d

(bastion-136):(admin) 09:55:16 UTC BASTION-1.5.1
# oc get pv | grep -E 'local-pv-dde7a22b|local-pv-b088f0ae|local-pv-fdce9e3d|local-pv-db289f43|local-pv-28857ff4|local-pv-c7e77c39'
local-pv-28857ff4   4Gi        RWO            Delete           Available                                                                   local-platform-sc               11s
local-pv-b088f0ae   4Gi        RWO            Delete           Available                                                                   local-application-sc            11s
local-pv-c7e77c39   4Gi        RWO            Delete           Released    mcp-logging/logsinfra-fluentd-2                                 local-platform-sc               10d
local-pv-db289f43   4Gi        RWO            Delete           Available                                                                   local-platform-sc               13s
local-pv-dde7a22b   4Gi        RWO            Delete           Available                                                                   local-application-sc            13s
local-pv-fdce9e3d   4Gi        RWO            Delete           Released    mcp-logging/logsapp-fluentd-2                                   local-application-sc            10d

(bastion-136):(admin) 09:55:16 UTC BASTION-1.5.1
# 


1. In the first case PV's are completely gone from the cluster.
2. In the second case we can see PV's got wiped out and put back into Available status. Some are still being Released and then Available.

Workaround:
After removing below dir pv's are back.

On vlan145 fixed master1. By fixed I mean I executed the following commands on the node. 

rm /mnt/local-storage/local-platform-sc/mcp-logsinfra
rm /mnt/local-storage/local-application-sc/mcp-logsapp

Expected Result:
After deleting PVC , PV's into Released state and then available.

Actual result:
After deleting PVC, PV's are missing.

Comment 15 Chao Yang 2022-07-21 09:42:54 UTC
oc get csv
NAME                                        DISPLAY                            VERSION              REPLACES                                    PHASE
elasticsearch-operator.v5.5.0               OpenShift Elasticsearch Operator   5.5.0                                                            Succeeded
local-storage-operator.4.8.0-202207192115   Local Storage                      4.8.0-202207192115   local-storage-operator.4.7.0-202207192115   Succeeded

1.Create localvolume as below.
oc get localvolume/example -o json | jq .spec
{
  "logLevel": "Normal",
  "managementState": "Managed",
  "storageClassDevices": [
    {
      "devicePaths": [
        "/dev/disk/by-path/pci-0000:00:1e.0-nvme-1"
      ],
      "fsType": "ext4",
      "storageClassName": "foobar",
      "volumeMode": "Filesystem"
    }
  ]
}

2.Create pvc/pod,after pod is running then delete pod and pvc.
3.Upgrade lso from 4.7 nightly to 4.8 nightly
4.After upgrade, pv still avalable.
oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                  STORAGECLASS   REASON   AGE
local-pv-ebf3dfad                          3Gi        RWO            Delete           Available                          foobar                  2s

Comment 18 errata-xmlrpc 2022-08-09 12:52:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.47 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5889