Description of problem: Fstype is changed after deleting pv provisioned by localvolumeset instance Version-Release number of selected component (if applicable): 4.6.0-0.nightly-2020-09-24-030538 local-storage-operator.4.6.0-202009231847.p0 How reproducible: Always Steps to Reproduce: 1.Install LSO 2.Create localvolumediscovery 3.Create localvolumeset apiVersion: v1 items: - apiVersion: local.storage.openshift.io/v1alpha1 kind: LocalVolumeSet metadata: creationTimestamp: "2020-09-25T05:54:59Z" generation: 1 managedFields: - apiVersion: local.storage.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:spec: .: {} f:deviceInclusionSpec: .: {} f:deviceMechanicalProperties: {} f:deviceTypes: {} f:nodeSelector: .: {} f:nodeSelectorTerms: {} f:storageClassName: {} f:volumeMode: {} manager: Mozilla operation: Update time: "2020-09-25T05:54:59Z" - apiVersion: local.storage.openshift.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:spec: f:deviceInclusionSpec: f:minSize: {} f:status: .: {} f:conditions: {} f:observedGeneration: {} f:totalProvisionedDeviceCount: {} manager: local-storage-operator operation: Update time: "2020-09-25T06:11:26Z" name: lvs namespace: openshift-local-storage resourceVersion: "1132670" selfLink: /apis/local.storage.openshift.io/v1alpha1/namespaces/openshift-local-storage/localvolumesets/lvs uid: cfb012ad-89c6-42f5-98ba-529a8bbdbc2f spec: deviceInclusionSpec: deviceMechanicalProperties: - NonRotational deviceTypes: - disk - part minSize: 0Ti nodeSelector: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ip-10-0-143-32 - ip-10-0-190-41 - ip-10-0-212-186 storageClassName: lvs volumeMode: Filesystem status: conditions: - lastTransitionTime: "2020-09-25T05:54:59Z" message: 'DiskMaker: Available, LocalProvisioner: Available' status: "True" type: DaemonSetsAvailable - lastTransitionTime: "2020-09-25T05:55:00Z" message: Operator reconciled successfully. status: "True" type: Available observedGeneration: 1 totalProvisionedDeviceCount: 4 kind: List metadata: resourceVersion: "" selfLink: "" 3.PV is provisioned 4.Create pvc/pod using above pv oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc5 Bound local-pv-485b8629 10Gi RWO lvs 7m20s 5.Check localvolumediscoveryresult - deviceID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol024547723ded32e1d fstype: ext4 model: 'Amazon Elastic Block Store ' path: /dev/nvme1n1 property: NonRotational serial: vol024547723ded32e1d size: 10737418240 status: state: NotAvailable type: disk vendor: "" discoveredTimeStamp: "2020-09-25T06:03:56Z" 6.Delete pod/pvc 7.Waiting for the pv is available - deviceID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol024547723ded32e1d fstype: ext2 model: 'Amazon Elastic Block Store ' path: /dev/nvme1n1 property: NonRotational serial: vol024547723ded32e1d size: 10737418240 status: state: NotAvailable type: disk vendor: "" discoveredTimeStamp: "2020-09-25T06:13:56Z" Actual results: pv fstype is changed from ext4 to ext2 after deleting pod/pvc Expected results: pv fstype should not changed Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
is it happening with LocalVolumeSet only or with LocalVolume as well?
The root cause is that `lsblk -o +fstype` info is out of date. Running `udevadm trigger` before `lsblk` refreshes it, but is considered somewhat expensive. We could run it only on the discovery pod (every 5 minutes) because by the time we run into this the localvolume/localvolumeset controller has already finished its job (symlinking).
Moving out to 4.7 as per discussion.
1.Create pvc/pod - deviceID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b9b38a379841f8e0 fstype: ext4 model: 'Amazon Elastic Block Store ' path: /dev/nvme2n1 property: NonRotational serial: vol0b9b38a379841f8e0 size: 1073741824 status: state: NotAvailable type: disk vendor: "" discoveredTimeStamp: "2020-10-10T05:42:06Z" 2.Delete pod/pvc - deviceID: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0b9b38a379841f8e0 fstype: "" model: 'Amazon Elastic Block Store ' path: /dev/nvme2n1 property: NonRotational serial: vol0b9b38a379841f8e0 size: 1073741824 status: state: Available type: disk vendor: "" discoveredTimeStamp: "2020-10-10T05:52:06Z" oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2020-10-08-210814 True False 20h Cluster version is 4.6.0-0.nightly-2020-10-08-210814
When pv is in available, blkid will return empty for /dev/nvme2n1, so the fstype is empty also. @hekumar Can we accept this?
> When pv is in available, blkid will return empty for /dev/nvme2n1, so the fstype is empty also. > @hekumar Can we accept this? That seems fine. Deleting the PVC results in calling wipefs on volume and hence fstype is empty. And although there is a PV on it, currently we don't track unused PVs and mark backing devices as "Unavailable". We could *do* that for local-storage PVs I think, by checking for existing symlinks in `/mnt/local-storage` for the backing device. Is that what you were expecting Chao Yang?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633