Description of problem:[Cephadm] 5.0 - Zap (erase) device is not clearing lvm data though output says zapping is successful.


Version-Release number of selected component (if applicable):

[root@magna122 ubuntu]# ./cephadm version
INFO:cephadm:Using recent ceph image registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-5.0-rhel-8-containers-candidate-62552-20200528181847
ceph version 15.2.2-1.el8cp (0c857e985a29d90501a285f242ea9c008df49eb8) octopus (stable)


How reproducible:
Steps to Reproduce:
1. Install a bootstrap cluster with cephadm and the dashboard service enabled.
2. # cephadm shell 
3. ceph -s reports health ok with 15 OSDs up
4. Perform failed/replaced OSDs 
5. From CLI, Perform the below
ceph orch osd rm 3 ( removed OSD ID#3 from host magna120)
-> OSD removed successfully
6. Remove one more OSD with replace option
ceph orch osd rm 4 --replace (removed OSD ID#4 from host magna120)
Status showing as destroyed in ceph osd tree
7. Now add new OSD which was removed at step5 with the below command
ceph orch daemon add osd magna120:/dev/sdb

Since the OSD has data we need to clean manually 
8. Perform clean using zap option

[ceph: root@magna122 /]# ceph orch device zap magna120 /dev/sdb
Error ENOENT: must pass --force to PERMANENTLY ERASE DEVICE DATA
see the above
[ceph: root@magna122 /]# ceph orch device zap magna120 /dev/sdb --force
INFO:cephadm:/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices.
INFO:cephadm:/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices.
INFO:cephadm:/bin/podman:stderr WARNING: The same type, major and minor should not be used for multiple devices.
INFO:cephadm:/bin/podman:stderr --> Zapping: /dev/sdb
INFO:cephadm:/bin/podman:stderr --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-2aba85ec-2810-47c0-928e-e1b370918b88/osd-data-eb3af14f-3ea2-401d-b9a5-eacff9eb2baf
INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-2aba85ec-2810-47c0-928e-e1b370918b88/osd-data-eb3af14f-3ea2-401d-b9a5-eacff9eb2baf bs=1M count=10 conv=fsync
INFO:cephadm:/bin/podman:stderr  stderr: 10+0 records in
INFO:cephadm:/bin/podman:stderr 10+0 records out
INFO:cephadm:/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.0847368 s, 124 MB/s
INFO:cephadm:/bin/podman:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-2aba85ec-2810-47c0-928e-e1b370918b88
INFO:cephadm:/bin/podman:stderr Running command: /usr/sbin/vgremove -v -f ceph-2aba85ec-2810-47c0-928e-e1b370918b88
INFO:cephadm:/bin/podman:stderr  stderr: Removing ceph--2aba85ec--2810--47c0--928e--e1b370918b88-osd--data--eb3af14f--3ea2--401d--b9a5--eacff9eb2baf (253:0)
INFO:cephadm:/bin/podman:stderr  stderr: Archiving volume group "ceph-2aba85ec-2810-47c0-928e-e1b370918b88" metadata (seqno 21).
INFO:cephadm:/bin/podman:stderr  stderr: Releasing logical volume "osd-data-eb3af14f-3ea2-401d-b9a5-eacff9eb2baf"
INFO:cephadm:/bin/podman:stderr  stderr: Creating volume group backup "/etc/lvm/backup/ceph-2aba85ec-2810-47c0-928e-e1b370918b88" (seqno 22).
INFO:cephadm:/bin/podman:stderr  stdout: Logical volume "osd-data-eb3af14f-3ea2-401d-b9a5-eacff9eb2baf" successfully removed
INFO:cephadm:/bin/podman:stderr  stderr: Removing physical volume "/dev/sdb" from volume group "ceph-2aba85ec-2810-47c0-928e-e1b370918b88"
INFO:cephadm:/bin/podman:stderr  stdout: Volume group "ceph-2aba85ec-2810-47c0-928e-e1b370918b88" successfully removed
INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync
INFO:cephadm:/bin/podman:stderr  stderr: 10+0 records in
INFO:cephadm:/bin/podman:stderr 10+0 records out
INFO:cephadm:/bin/podman:stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.097538 s, 108 MB/s
INFO:cephadm:/bin/podman:stderr --> Zapping successful for: <Raw Device: /dev/sdb>


Device is still listing under*********************************

[ceph: root@magna122 /]# ceph orch device ls --refresh
HOST      PATH      TYPE   SIZE  DEVICE                           AVAIL  REJECT REASONS                                          
magna023  /dev/sda  hdd    931G  Hitachi_HUA72201_JPW9K0N20D1S4E  False  locked                                                  
magna023  /dev/sdb  hdd    931G  Hitachi_HUA72201_JPW9K0N20D1ZHE  False  locked, LVM detected, Insufficient space (<5GB) on vgs  
magna023  /dev/sdc  hdd    931G  Hitachi_HUA72201_JPW9K0N20BS64E  False  locked, LVM detected, Insufficient space (<5GB) on vgs  
magna023  /dev/sdd  hdd    931G  Hitachi_HUA72201_JPW9K0N20D19WE  False  locked, LVM detected, Insufficient space (<5GB) on vgs  
magna057  /dev/sda  hdd    931G  Hitachi_HUA72201_JPW9J0N20BMXLC  False  locked                                                  
magna057  /dev/sdb  hdd    931G  Hitachi_HUA72201_JPW9J0N20B4RGC  False  LVM detected, Insufficient space (<5GB) on vgs, locked  
magna057  /dev/sdc  hdd    931G  Hitachi_HUA72201_JPW9J0N20BZGDC  False  LVM detected, Insufficient space (<5GB) on vgs, locked  
magna057  /dev/sdd  hdd    931G  Hitachi_HUA72201_JPW9J0N20BZETC  False  LVM detected, Insufficient space (<5GB) on vgs, locked  
magna067  /dev/sda  hdd    931G  Hitachi_HUA72201_JPW9J0N20A636C  False  locked                                                  
magna067  /dev/sdb  hdd    931G  Hitachi_HUA72201_JPW9J0N20BBBNC  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna067  /dev/sdc  hdd    931G  Hitachi_HUA72201_JPW9J0N20B535C  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna067  /dev/sdd  hdd    931G  Hitachi_HUA72201_JPW9J0N20B4W4C  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna120  /dev/sda  hdd    931G  Hitachi_HUA72201_JPW9M0N20BU8LE  False  locked                                                  
magna120  /dev/sdb  hdd    931G  Hitachi_HUA72201_JPW9M0N20BMN3E  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna120  /dev/sdc  hdd    931G  Hitachi_HUA72201_JPW9M0N20BRY0E  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna120  /dev/sdd  hdd    931G  Hitachi_HUA72201_JPW9M0N20BVPBE  False  Insufficient space (<5GB) on vgs, locked, LVM detected  
magna122  /dev/sda  hdd    931G  Hitachi_HUA72201_JPW9J0N20BZHWC  False  locked                                                  
magna122  /dev/sdb  hdd    931G  Hitachi_HUA72201_JPW9M0N20BWYME  False  LVM detected, locked, Insufficient space (<5GB) on vgs  
magna122  /dev/sdc  hdd    931G  Hitachi_HUA72201_JPW9M0N20BX1KE  False  LVM detected, locked, Insufficient space (<5GB) on vgs  
magna122  /dev/sdd  hdd    931G  Hitachi_HUA72201_JPW9M0N20BVUXE  False  LVM detected, locked, Insufficient space (<5GB) on vgs  


[root@magna120 ubuntu]# lvm pvscan *************reports lvm details after successful zap execution**************
  PV /dev/sdd   VG ceph-da82d88d-8312-49f4-9e07-7b05d6ea1d41   lvm2 [931.51 GiB / 0    free]
  PV /dev/sdc   VG ceph-2234e84b-705d-4207-9986-69991cf1467b   lvm2 [931.51 GiB / 0    free]
  PV /dev/sdb   VG ceph-558e8a62-ccd7-4ddc-b60c-3f47676cb34a   lvm2 [931.51 GiB / 0    free]
  Total: 3 [<2.73 TiB] / in use: 3 [<2.73 TiB] / in no VG: 0 [0   ]





cephadm host is magna122 root/q

Actual results: Zap is not wiping the data. output says zap is succesful


Expected results: Zap should wipe lvm data and device should not list in lvm pvscan command


Additional info:


ceph -s ->> OSDs rae removed from magna120 and magna067 to perform remove/replace OSDs feature hence its 12 OSDS now

[ceph: root@magna122 /]# ceph -s
  cluster:
    id:     d5873fce-a567-11ea-ba74-002590fc2528
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum magna122,magna057,magna023 (age 9d)
    mgr: magna057.dbrbnj(active, since 7d), standbys: magna067.voqbwd, magna122.bnvpqh, magna067.whiaht, magna067.eevcop
    mds:  1 up:standby
    osd: 12 osds: 12 up (since 67m), 12 in (since 67m)
    rgw: 2 daemons active (myorg.us-east-1.magna057.vpyehi, myorg.us-east-1.magna067.yrrvks)
 
  task status:
 
  data:
    pools:   6 pools, 137 pgs
    objects: 252 objects, 6.1 KiB
    usage:   13 GiB used, 11 TiB / 11 TiB avail
    pgs:     137 active+clean
 
[ceph: root@magna122 /]#