Bug 1896803

Summary: [cee/sd][ceph-volume] when running playbook add-osd.yml or site.yml ceph-volume does not create OSDs on new devices
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tomas Petr <tpetr>
Component: Ceph-VolumeAssignee: Guillaume Abrioux <gabrioux>
Status: CLOSED ERRATA QA Contact: Ameena Suhani S H <amsyedha>
Severity: medium Docs Contact: Ranjini M N <rmandyam>
Priority: medium    
Version: 4.1CC: agunn, amsyedha, asakthiv, aschoen, asoumpli, assingh, ceph-eng-bugs, ceph-qe-bugs, dsavinea, dwojewod, gabrioux, gjose, gmeno, gsitlani, jansingh, lithomas, mhackett, mmanjuna, mmuench, nthomas, quanghaidmn739, ridave, rmandyam, shzhou, torkil, tserlin, vashastr, vereddy, ykaul
Target Milestone: ---   
Target Release: 4.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-14.2.21-8.el8cp, ceph-14.2.21-8.el7cp Doc Type: Bug Fix
Doc Text:
.The `add-osd.yml` playbook does not fail anymore while creating new OSDs Previously, the `add-osd.yml` playbook would fail when new OSDs were added using `ceph-ansible`. This was due to the `ceph-volume lvm batch` limitation which does not allow addition of new OSDs in a non-interactive mode. With this release, the `--yes` and `--report` options are not passed to the command-line interface and the `add-osd.yml` playbook works as expected when creating new OSDs.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-05 07:53:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1966934    
Bug Blocks: 1890121, 2031070    

Description Tomas Petr 2020-11-11 15:11:53 UTC 
Description of problem:
On existing ceph node are deployed OSDs using ceph-ansible ceph-volume batch scenario.
When adding additional devices to the host_vars/node configuration file and running add-osd.yml or site.yml, the new devices are not being populated by OSDs:

Initial node deployment:
----
# cat /etc/ansible/host_vars/osds-3
devices:
  - /dev/vde

dedicated_devices:
  - /dev/vdb

[root@osds-3 ~]# lsblk
NAME                                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                                                                                                   253:0    0   20G  0 disk 
├─vda1                                                                                                                253:1    0    1M  0 part 
├─vda2                                                                                                                253:2    0  100M  0 part /boot/efi
└─vda3                                                                                                                253:3    0 19.9G  0 part /
vdb                                                                                                                   253:16   0   30G  0 disk 
└─ceph--block--dbs--199c644a--8657--46f3--8136--b6679cd68153-osd--block--db--7d8b9f07--e822--43da--866b--dc202c3740bd 252:1    0   30G  0 lvm  
vdc                                                                                                                   253:32   0   30G  0 disk 
vdd                                                                                                                   253:48   0   30G  0 disk 
vde                                                                                                                   253:64   0   30G  0 disk 
└─ceph--block--6171157b--47dc--426a--ba3f--5876bb49e42b-osd--block--4b812472--826e--41c4--ba00--c6c0eedc64ae          252:0    0   30G  0 lvm  


Adding new OSDs
----
# cat /etc/ansible/host_vars/osds-3
devices:
  - /dev/vde
  - /dev/vdd

dedicated_devices:
  - /dev/vdb
  - /dev/vdc


# ansible-playbook infrastructure-playbooks/add-osd.yml --limit osds-3
...
2020-11-11 03:15:58,084 p=47055 u=root |  PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
2020-11-11 03:15:58,085 p=47055 u=root |  osds-3                     : ok=184  changed=8    unreachable=0    failed=0    skipped=299  rescued=0    ignored=1   
2020-11-11 03:15:58,093 p=47055 u=root |  Wednesday 11 November 2020  03:15:58 -0500 (0:00:00.056)       0:04:41.749 **** 
2020-11-11 03:15:58,094 p=47055 u=root |  =============================================================================== 
....

[root@osds-3 ~]# lsblk
NAME                                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                                                                                                   253:0    0   20G  0 disk 
├─vda1                                                                                                                253:1    0    1M  0 part 
├─vda2                                                                                                                253:2    0  100M  0 part /boot/efi
└─vda3                                                                                                                253:3    0 19.9G  0 part /
vdb                                                                                                                   253:16   0   30G  0 disk 
└─ceph--block--dbs--199c644a--8657--46f3--8136--b6679cd68153-osd--block--db--7d8b9f07--e822--43da--866b--dc202c3740bd 252:1    0   30G  0 lvm  
vdc                                                                                                                   253:32   0   30G  0 disk 
vdd                                                                                                                   253:48   0   30G  0 disk 
vde                                                                                                                   253:64   0   30G  0 disk 
└─ceph--block--6171157b--47dc--426a--ba3f--5876bb49e42b-osd--block--4b812472--826e--41c4--ba00--c6c0eedc64ae          252:0    0   30G  0 lvm  
----

The playbook ends with OK, but no new devices are being deployed.

The task
2020-11-11 03:03:36,685 p=45877 u=root |  TASK [ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created] 

does not set any devices to be deployed


When we have in  host_vars/node configuration file only new devices, the OSDs will get deployed, but that brings different issue, that the 'osd memory target' value is calculated per number of OSDs in host_vars/node configuration file, so it will get calculated wrongly for the amount of OSDs the node will have in total, old+new.


Version-Release number of selected component (if applicable):
ansible-2.8.16-1.el8ae.noarch
ceph-ansible-4.0.31-1.el8cp.noarch
"ceph version 14.2.8-111.el8cp

How reproducible:
Always

Steps to Reproduce:
1. deploy node with x amount of devices 
2. add additional device into  host_vars/node configuration file, keep the old ones
3. try to deploy OSDs on those 

Actual results:


Expected results:


Additional info:
Comment 29 Vasishta 2021-06-09 14:07:41 UTC 
*** Bug 1966934 has been marked as a duplicate of this bug. ***
Comment 35 Anjana Suparna Sriram 2021-06-15 10:20:57 UTC 
*** Bug 1970031 has been marked as a duplicate of this bug. ***
Comment 47 errata-xmlrpc 2022-05-05 07:53:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 4.3 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1716
Comment 48 Kinder 2023-05-29 06:37:28 UTC 
Thanks to this article I can learn more. Expand my knowledge and abilities. Actually the article is very real. https://www.mybalancenow.page/