Bug 1897050

can you please confirm if PVs were provisioned by LocalVolumeSet?

Yes , PVs were created

[root@nx121-ahv ~]# oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                       STORAGECLASS   REASON   AGE
local-pv-1e31f771   256Gi      RWO            Delete           Available                                                               localblock              39h
local-pv-2ac8dd7d   4Mi        RWO            Delete           Bound       openshift-storage/ocs-deviceset-localblock-1-data-0-7ttnl   localblock              39h
local-pv-494ea7c2   4Mi        RWO            Delete           Bound       openshift-storage/ocs-deviceset-localblock-0-data-0-565bg   localblock              39h
local-pv-8137c873   256Gi      RWO            Delete           Available                                                               localblock              39h
local-pv-97511c2c   4Mi        RWO            Delete           Bound       openshift-storage/ocs-deviceset-localblock-2-data-0-5kt9t   localblock              39h
local-pv-ec7f2b80   256Gi      RWO            Delete           Available                                                               localblock              39h

[root@nx121-ahv ~]# oc get pvc
NAME                                      STATUS    VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
db-noobaa-db-0                            Pending                                                 ocs-storagecluster-ceph-rbd   39h
ocs-deviceset-localblock-0-data-0-565bg   Bound     local-pv-494ea7c2   4Mi        RWO            localblock                    39h
ocs-deviceset-localblock-1-data-0-7ttnl   Bound     local-pv-2ac8dd7d   4Mi        RWO            localblock                    39h
ocs-deviceset-localblock-2-data-0-5kt9t   Bound     local-pv-97511c2c   4Mi        RWO            localblock                    39h

[root@nx121-ahv ~]# 

[root@nx121-ahv ~]# oc get sc
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                    kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  39h
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   39h
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  39h
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   39h

[root@nx121-ahv ~]#

Something is wrong with the PV. The capacity is only 4Mi for all of them.

Please share : 
1. localVolumeSet CRD 
2. logs from localvolumeset daemon. 
3. Local storage operator logs.

Created attachment 1728744 [details]
crd for local volumeset

Created attachment 1728745 [details]
Logs for local-storage operator

(In reply to Aaruni from comment #6)
> Created attachment 1728744 [details]
> crd for local volumeset

Sorry, I meant the CR.

Created attachment 1728746 [details]
logs for diskmaker-manager

Created attachment 1728747 [details]
logs for localvolumeset-local-provisioner

@Rohan

Looks like issue with LocalVolumeSet. PVs are being created from boot device. Lsblk output from one of the nodes

`[core@worker-0 ~]$ lsblk  --bytes  --pairs  --output  "NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,FSTYPE,SERIAL,KNAME,PARTLABEL"
NAME="vda" ROTA="1" TYPE="disk" SIZE="42949672960" MODEL="" VENDOR="0x1af4" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="vda" PARTLABEL=""
NAME="vda1" ROTA="1" TYPE="part" SIZE="4194304" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="vda1" PARTLABEL="PowerPC-PReP-boot"
NAME="vda2" ROTA="1" TYPE="part" SIZE="402653184" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="ext4" SERIAL="" KNAME="vda2" PARTLABEL="boot"
NAME="vda4" ROTA="1" TYPE="part" SIZE="42541760000" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="crypto_LUKS" SERIAL="" KNAME="vda4" PARTLABEL="luks_root"
NAME="vdb" ROTA="1" TYPE="disk" SIZE="512" MODEL="" VENDOR="0x1af4" RO="1" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="vdb" PARTLABEL=""
NAME="vdc" ROTA="1" TYPE="disk" SIZE="274877906944" MODEL="" VENDOR="0x1af4" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="vdc" PARTLABEL=""
NAME="coreos-luks-root-nocrypt" ROTA="1" TYPE="dm" SIZE="42524982784" MODEL="" VENDOR="" RO="0" RM="0" STATE="running" FSTYPE="xfs" SERIAL="" KNAME="dm-0" PARTLABEL=""
` 

Notice device `vda1`. The PARTLABEL is `PowerPC-PReP-boot`, which would fail this check -
https://github.com/openshift/local-storage-operator/blob/fe65d2700416ba62e6a9fde947f1ee24c9e5b2bb/pkg/diskmaker/controllers/lvset/matcher.go#L54


Snippet from diskmaker logs:

`{"level":"info","ts":1605038528.3909867,"logger":"localvolumeset-symlink-controller","msg":"symlinking","Request.Namespace":"openshift-local-storage","Request.Name":"localblock","Device.Name":"vda1"}
{"level":"info","ts":1605038528.393325,"logger":"localvolumeset-symlink-controller","msg":"symlinking","Request.Namespace":"openshift-local-storage","Request.Name":"localblock","Device.Name":"vda1","sourcePath":"/dev/vda1","targetPath":"/mnt/local-storage/localblock/vda1"}`

Will look into a fix. Workaround is to set a .spec.deviceInclusionSpec.minSize to something like 100 Gi

Also looking at your device layout, you could set:

  localvolumeset.spec.deviceInclusionSpec.deviceTypes = ['disk']

instead of 
   localvolumeset.spec.deviceInclusionSpec.deviceTypes = ['disk','part']

(In reply to Rohan CJ from comment #12)
> Will look into a fix. Workaround is to set a
> .spec.deviceInclusionSpec.minSize to something like 100 Gi

Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work

@Hemant, would it make sense to check for the prep-boot like we do for bios partlabel?

Also, is this a blocker?

The partition type explained: https://tldp.org/HOWTO/IBM7248-HOWTO/faq.html#134-The-PReP-boot-partition

Observer the similar behavior over AWS-UPI-RHEL-I3 where the worker node is of rhel-7 image 


for i in $(oc get nodes |grep worker |awk '{print$1}'); do oc debug node/$i -- chroot /host lsblk  --bytes  --pairs  --output  "NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,FSTYPE,SERIAL,KNAME,PARTLABEL" ; done
Creating debug namespace/openshift-debug-node-chzvr ...
Starting pod/ip-10-0-59-96us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME="nvme0n1" ROTA="0" TYPE="disk" SIZE="128849018880" MODEL="Amazon Elastic Block Store              " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="vol040999c2486aa0412" KNAME="nvme0n1" PARTLABEL=""
NAME="nvme0n1p1" ROTA="0" TYPE="part" SIZE="1048576" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="nvme0n1p1" PARTLABEL=""
NAME="nvme0n1p2" ROTA="0" TYPE="part" SIZE="128846904320" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="xfs" SERIAL="" KNAME="nvme0n1p2" PARTLABEL=""
NAME="nvme1n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWSB6958A5581996F3DB" KNAME="nvme1n1" PARTLABEL=""
NAME="nvme2n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWSBE89B4C66EAC4F7D2" KNAME="nvme2n1" PARTLABEL=""

Removing debug pod ...
Removing debug namespace/openshift-debug-node-chzvr ...
Creating debug namespace/openshift-debug-node-b6bmd ...
Starting pod/ip-10-0-74-22us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME="nvme0n1" ROTA="0" TYPE="disk" SIZE="128849018880" MODEL="Amazon Elastic Block Store              " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="vol04735699405048a19" KNAME="nvme0n1" PARTLABEL=""
NAME="nvme0n1p1" ROTA="0" TYPE="part" SIZE="1048576" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="nvme0n1p1" PARTLABEL=""
NAME="nvme0n1p2" ROTA="0" TYPE="part" SIZE="128846904320" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="xfs" SERIAL="" KNAME="nvme0n1p2" PARTLABEL=""
NAME="nvme1n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWS222216B00E36EEAF0" KNAME="nvme1n1" PARTLABEL=""
NAME="nvme2n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWS6A250182A01DBEA19" KNAME="nvme2n1" PARTLABEL=""

Removing debug pod ...
Removing debug namespace/openshift-debug-node-b6bmd ...
Creating debug namespace/openshift-debug-node-wgz46 ...
Starting pod/ip-10-0-81-9us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME="nvme0n1" ROTA="0" TYPE="disk" SIZE="128849018880" MODEL="Amazon Elastic Block Store              " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="vol0fe7c4eef64d04b9d" KNAME="nvme0n1" PARTLABEL=""
NAME="nvme0n1p1" ROTA="0" TYPE="part" SIZE="1048576" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="nvme0n1p1" PARTLABEL=""
NAME="nvme0n1p2" ROTA="0" TYPE="part" SIZE="128846904320" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="xfs" SERIAL="" KNAME="nvme0n1p2" PARTLABEL=""
NAME="nvme1n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWS63A2B04037BCC167C" KNAME="nvme1n1" PARTLABEL=""
NAME="nvme2n1" ROTA="0" TYPE="disk" SIZE="2500000000000" MODEL="Amazon EC2 NVMe Instance Storage        " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="AWS2709233678AE3C953" KNAME="nvme2n1" PARTLABEL=""

Removing debug pod ...
Removing debug namespace/openshift-debug-node-wgz46 ...


for i in $(oc get nodes |grep worker |awk '{print$1}'); do oc debug node/$i -- chroot /host lsblk ; done                                                                                   
Creating debug namespace/openshift-debug-node-rdwl4 ...
Starting pod/ip-10-0-59-96us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:2    0  120G  0 disk 
|-nvme0n1p1 259:3    0    1M  0 part 
`-nvme0n1p2 259:4    0  120G  0 part /
nvme1n1     259:0    0  2.3T  0 disk 
nvme2n1     259:1    0  2.3T  0 disk 

Removing debug pod ...
Removing debug namespace/openshift-debug-node-rdwl4 ...
Creating debug namespace/openshift-debug-node-d72nv ...
Starting pod/ip-10-0-74-22us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:2    0  120G  0 disk 
|-nvme0n1p1 259:3    0    1M  0 part 
`-nvme0n1p2 259:4    0  120G  0 part /
nvme1n1     259:0    0  2.3T  0 disk 
nvme2n1     259:1    0  2.3T  0 disk 

Removing debug pod ...
Removing debug namespace/openshift-debug-node-d72nv ...
Creating debug namespace/openshift-debug-node-vf5c7 ...
Starting pod/ip-10-0-81-9us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:2    0  120G  0 disk 
|-nvme0n1p1 259:3    0    1M  0 part 
`-nvme0n1p2 259:4    0  120G  0 part /
nvme1n1     259:0    0  2.3T  0 disk 
nvme2n1     259:1    0  2.3T  0 disk 

Removing debug pod ...
Removing debug namespace/openshift-debug-node-vf5c7 ...

So basically this is going to be whack-a-mole and whatever rules we put in the code is not going to be sufficient and disks that shouldn't be claimed will be claimed. First thing, we need to do is add a big warning to docs about using this feature - https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/persistent-storage-local.html#local-storage-discovery_persistent-storage-local . I filed a doc bug about warning users - https://bugzilla.redhat.com/show_bug.cgi?id=1899491


@Pratik what similar behaviour you are pointing to? Can you please clarify?


> Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work

Do we know why 100Gi did not work? We should debug that and find root cause of it.

> @Hemant, would it make sense to check for the prep-boot like we do for bios partlabel?

@Rohan I think things like this are bound to happen when using LocalVolumeSet feature. I am thinking we should rather document this properly and if there are any bugs with minSize filter, we should fix it. I am not sure adding more rules to LSO disk filtering will help. The default filtering rules can also make it impossible for someone to use a disk that is filtered out. If filtering is less strict then they can always add their own filters (such as minSize in this case).

(In reply to Hemant Kumar from comment #17)
> So basically this is going to be whack-a-mole and whatever rules we put in
> the code is not going to be sufficient and disks that shouldn't be claimed
> will be claimed. First thing, we need to do is add a big warning to docs
> about using this feature -
> https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/
> persistent-storage-local.html#local-storage-discovery_persistent-storage-
> local . I filed a doc bug about warning users -
> https://bugzilla.redhat.com/show_bug.cgi?id=1899491
> 
> 
> @Pratik what similar behaviour you are pointing to? Can you please clarify?
> 
> 
> > Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work
> 
> Do we know why 100Gi did not work? We should debug that and find root cause
> of it.
> 
> > @Hemant, would it make sense to check for the prep-boot like we do for bios partlabel?
> 
> @Rohan I think things like this are bound to happen when using
> LocalVolumeSet feature. I am thinking we should rather document this
> properly and if there are any bugs with minSize filter, we should fix it. I
> am not sure adding more rules to LSO disk filtering will help. The default
> filtering rules can also make it impossible for someone to use a disk that
> is filtered out. If filtering is less strict then they can always add their
> own filters (such as minSize in this case).


> Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work
> 
> Do we know why 100Gi did not work? We should debug that and find root cause
> of it.
The pvs were getting created from boot device . that's why no osd pods were there. Situation was exactly same.

(In reply to Aaruni from comment #18)
> (In reply to Hemant Kumar from comment #17)
> > So basically this is going to be whack-a-mole and whatever rules we put in
> > the code is not going to be sufficient and disks that shouldn't be claimed
> > will be claimed. First thing, we need to do is add a big warning to docs
> > about using this feature -
> > https://docs.openshift.com/container-platform/4.6/storage/persistent_storage/
> > persistent-storage-local.html#local-storage-discovery_persistent-storage-
> > local . I filed a doc bug about warning users -
> > https://bugzilla.redhat.com/show_bug.cgi?id=1899491
> > 
> > 
> > @Pratik what similar behaviour you are pointing to? Can you please clarify?
> > 
> > 
> > > Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work
> > 
> > Do we know why 100Gi did not work? We should debug that and find root cause
> > of it.
> > 
> > > @Hemant, would it make sense to check for the prep-boot like we do for bios partlabel?
> > 
> > @Rohan I think things like this are bound to happen when using
> > LocalVolumeSet feature. I am thinking we should rather document this
> > properly and if there are any bugs with minSize filter, we should fix it. I
> > am not sure adding more rules to LSO disk filtering will help. The default
> > filtering rules can also make it impossible for someone to use a disk that
> > is filtered out. If filtering is less strict then they can always add their
> > own filters (such as minSize in this case).
> 
> 
> > Now I am setting minSize equal to the size of my additional attached disk . In my case its 256Gi . As for me 100Gi didn't work
> > 
> > Do we know why 100Gi did not work? We should debug that and find root cause
> > of it.
> The pvs were getting created from boot device . that's why no osd pods were
> there. Situation was exactly same.

after adding minSize=100Gi the pvs that were getting in bound state had capacity equal to 4Mi

> @Pratik what similar behavior you are pointing to? Can you please clarify?

Here without using many minSize filters PV is getting created from the Boot disk which is of size 1 Mi Just wanted to notify the BZ that it also affecting AWS-rhel-i3 deployment. After applying the workaround of minSize of 100Gi it worked

(In reply to Aaruni from comment #19)
> 
> after adding minSize=100Gi the pvs that were getting in bound state had
> capacity equal to 4Mi


Could be the case that previous PV was not deleted or the 4Mi disk simlink was not deleted.

Removing `blocker?` since we're planning to fix this in docs, not code.

Would it make sense to have the minSize default to 1GiB and if someone wants smaller volumes they need to explicitly override it?

Notes on Testing:
- MinSize in LocalVolumeSet now has a default value of 1Gi.  If user doesn't provide minSize then any disks below 1Gi will be ignored for provisioning PVs. User can override minSize anytime.
- PVs won't be provisioned on disks with `partLabel` containing substring  `boot`, `BOOT`, 'bios` and `BIOS`

lsblk  --bytes  --pairs  --output  "NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,FSTYPE,SERIAL,KNAME,PARTLABEL"
NAME="nvme0n1" ROTA="0" TYPE="disk" SIZE="128849018880" MODEL="Amazon Elastic Block Store              " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="vol05eadde8fd667f4b2" KNAME="nvme0n1" PARTLABEL=""
NAME="nvme0n1p1" ROTA="0" TYPE="part" SIZE="1048576" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="" SERIAL="" KNAME="nvme0n1p1" PARTLABEL="BIOS-BOOT"
NAME="nvme0n1p2" ROTA="0" TYPE="part" SIZE="133169152" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="vfat" SERIAL="" KNAME="nvme0n1p2" PARTLABEL="EFI-SYSTEM"
NAME="nvme0n1p3" ROTA="0" TYPE="part" SIZE="402653184" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="ext4" SERIAL="" KNAME="nvme0n1p3" PARTLABEL="boot"
NAME="nvme0n1p4" ROTA="0" TYPE="part" SIZE="128311082496" MODEL="" VENDOR="" RO="0" RM="0" STATE="" FSTYPE="xfs" SERIAL="" KNAME="nvme0n1p4" PARTLABEL="root"
NAME="nvme1n1" ROTA="0" TYPE="disk" SIZE="1073741824" MODEL="Amazon Elastic Block Store              " VENDOR="" RO="0" RM="0" STATE="live" FSTYPE="" SERIAL="vol00c040c72b1fc0517" KNAME="nvme1n1" PARTLABEL="

Localvolumeset minSize=0
    "spec": {
      "deviceInclusionSpec": {
        "deviceTypes": [
          "disk"
        ],
        "minSize": "0Ti"
      },

PV is not provisioned for nvme0n1p3

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633