Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Created attachment 1739413 [details]
response from 'ceph orch device ls'

Created attachment 1739414 [details]
logged errors to 'ceph orch apply osd --all-available-devices'

@John Harrigan: Thank you for the report and for attach the important log files.

Well, it seems that we have a problem "detecting" the real available devices. 

This is the real bug: Devices with GPT tables appear as available devices using "ceph orch device ls"

Explanation of your problem:
We are not detecting GTP in the disk and this is why devices appears as available. When we try to use this devices, ceph-volume  does not allow it because it finds this GPT tables in the 
disks.

Just to confirm: Can you execute the command "sudo fdisk -l" in "e20-h17-6029r.alias.bos.scalelab.redhat.com" and provide the output?

Workaround for you:

Just clean completely the storage devices before trying to create again the devices. You can use the orchestrator command:

https://docs.ceph.com/en/latest/mgr/orchestrator/#erase-devices-zap-devices

or any kind of disk tool in order to "zap" the device.

Note: If you haven't deleted the "osd.all-available-devices" service. As soon as devices start to be completely clean they will be used to create OSDs automatically.

What is the meaning of 'ceph orch ls' AVAIL = True ?
In this doc, the section titled "DEPLOY OSDS" states the
conditions which determine if a storage device is considered
"Available". There is no mention of GPT Labels. 
  https://docs.ceph.com/en/latest/cephadm/install/

The detection of GPT Labels apparently isn't happening until
'ceph-volume' attempts to create the OSD (/bin/podman run --group-add=disk).  

For my system, I zapped the disks and then ceph orch brought them in
Here were the cmds used:

# ceph orch pause    ← pause cephadm background work
# yum install gdisk
# sgdisk -Z /dev/sda  ← remove GPT all ‘Available’ disks
# partprobe /dev/sda  ← repeat for all ‘Available’ disks

# ceph orch resume  ← OSDs get added

Since these device are now configured as OSDs, the fdisk output
from that sysetem would not reflect its state prior to OSD deployment.
I have attached the 'fdisk -l' from a similar system (same hardware,
same storage devices) which was freshly provisioned with RHEL 8.2

Created attachment 1739722 [details]
fdisk -l output

In cephadm the availability of the disks to create OSDs comes directly from the information provided by the command:
"ceph-volume inventory"
In each of the hosts in the cluster when you execute "ceph orch device ls"

In the documentation we have put the list of things that ceph volume is verifying in order to deem a device as available. To have a "disk label  of type gtp" is not one of them ... but lets continue investigating.

It seems that despite ceph-volume is reporting the devices as available .. there are not usable when we try to execute the "ceph-volume lvm batch" command

Can you copy the cephadm tool to the host where you say that it is a similar system and execute the following commands: (please attach in this bug the output obtained)

# fdisk -l

# cephadm ceph-volume inventory 

# cephadm ceph-volume lvm batch --no-auto /dev/vda /dev/vdb  <--- using a couple of the devices that appears with "Disklabel type: gpt" in the output of fdisk

Created attachment 1741211 [details]
fdisk, ceph volume inventory, cephadm ceph-volume lvm batch --no-auto

Thank you for the information, John:
I have moved the bug to "ceph-volume", and I have created an upstream bug (https://tracker.ceph.com/issues/48697) to fix this problem.

The real problem is:

Ceph-volume reports a device as available and the device cannot be used to create OSDs because it has  GPT headers.

Two possibilities.. report the device as no available for this cause (GPT headers), or fix the lvm batch command in order to make possible use this devices.

Extract of the relevant info:

[root@e20-h19-6029r tmp]# fdisk -l
...
Disk /dev/sda: 5.5 TiB, 6000606183424 bytes, 11719933952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: gpt
Disk identifier: 90557F98-DC56-45F8-8269-C5D64FB8F864
...

root@e20-h19-6029r tmp]# ./cephadm ceph-volume inventory 
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
INFO:cephadm:/usr/bin/podman:stdout 
INFO:cephadm:/usr/bin/podman:stdout Device Path               Size         rotates available Model name
INFO:cephadm:/usr/bin/podman:stdout /dev/nvme0n1              349.32 GB    False   True      INTEL SSDPED1K375GA
INFO:cephadm:/usr/bin/podman:stdout /dev/sda                  5.46 TB      True    True      SMC3108
....

Device Path               Size         rotates available Model name
/dev/nvme0n1              349.32 GB    False   True      INTEL SSDPED1K375GA
/dev/sda                  5.46 TB      True    True      SMC3108
...

[root@e20-h19-6029r tmp]# ./cephadm ceph-volume lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc
...
INFO:cephadm:/usr/bin/podman:stderr ceph-volume lvm batch: error: GPT headers found, they must be removed on: /dev/sda
Traceback (most recent call last):
  File "./cephadm", line 5202, in <module>
    r = args.func()
  File "./cephadm", line 1138, in _infer_fsid
    return func()
  File "./cephadm", line 1197, in _infer_image
    return func()
  File "./cephadm", line 3320, in command_ceph_volume
    out, err, code = call_throws(c.run_cmd(), verbose=True)
  File "./cephadm", line 876, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --net=host --ipc=host --privileged --group-add=disk -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=e20-h19-6029r.alias.bos.scalelab.redhat.com -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm --entrypoint /usr/sbin/ceph-volume docker.io/ceph/ceph:v15 lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc

Andrew, Would you please look into this and make a recommendation? What do you think about fixing this in the next 10 days?

(In reply to Christina Meno from comment #10)
> Andrew, Would you please look into this and make a recommendation? What do
> you think about fixing this in the next 10 days?

My recommendation would be to patch `ceph-volume inventory` to list devices with GPT headers as not available. I think that should be doable in the next 10 days, I'll work on an upstream patch.

Thanks,
Andrew

https://github.com/ceph/ceph/pull/40315 is in pacific now. We merged it upstream a few hours after I rebased downstream yesterday. Since I'm fixing bug 1939239 anyway today, I'm doing another rebase today for this one.

Hi Team,

I could Verify this from ceph-volume POV
1) Checked current state of devices
[ceph: root@pluto001 /]# ceph-volume inventory

Device Path               Size         rotates available Model name
/dev/sdb                  447.13 GB    False   True      SAMSUNG MZ7KM480
/dev/sdc                  447.13 GB    False   True      SAMSUNG MZ7KM480
/dev/sdd                  447.13 GB    False   True      SAMSUNG MZ7KM480
/dev/sda                  447.13 GB    False   False     SAMSUNG MZ7KM480


2) Created gpt label in /dev/sdd
[root@pluto001 ~]# parted /dev/sdd mklabel gpt
Information: You may need to update /etc/fstab.

Disk /dev/sdd: 447.1 GiB, 480103981056 bytes, 937703088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 46EF4769-8A3C-4A8D-AF3B-DDD9625F8B14


3) Checked ceph-volume inventory 
[root@pluto001 ~]# cephadm shell ceph-volume inventory      
....
Device Path               Size         rotates available Model name
/dev/sdb                  447.13 GB    False   True      SAMSUNG MZ7KM480
/dev/sdc                  447.13 GB    False   True      SAMSUNG MZ7KM480
/dev/sda                  447.13 GB    False   False     SAMSUNG MZ7KM480
/dev/sdd                  447.13 GB    False   False     SAMSUNG MZ7KM480

But ceph orch still listing /dev/sdd is available
4) [root@pluto001 ~]# cephadm shell ceph orch device ls
...
Hostname  Path      Type  Serial          Size   Health   Ident  Fault  Available  
pluto001  /dev/sdb  ssd   S2HSNX0H500733   480G  Unknown  N/A    N/A    Yes        
pluto001  /dev/sdc  ssd   S2HSNX0H500742   480G  Unknown  N/A    N/A    Yes        
pluto001  /dev/sdd  ssd   S2HSNX0H500743   480G  Unknown  N/A    N/A    Yes        

@jolmomar Can you please help me to proceed ? (I think we can move this BZ to VERIFIED state, we need to check with orchestrator ?)

I'm Sorry, Seemed like there was some latency
I tried cephadm shell ceph orch device ls after a minute and seemed it listed /dev/sdd as not available

@vashastr:

There is also the possibility of "force" the update, in the case where we are doing changes in devices ( removing, adding devices, or just use them for different things):

# ceph orch ls --refresh

Hi Team,

Ceph-volume's modifications have been verified, 

>> checked inventory, all devices were marked as available (Available - true)
>> Created gpt label using `parted /dev/sdd mklabel gpt` checked ceph-volume inventory, the device where gpt label was created was marked as unavailable (available - False)

I'm moving this BZ o VERIFIED state with a slight changes to summary of the BZ.

I observed that ceph orchestrator still included the new device to the list of all available devices and osd configuration did not happen 
I've filed https://bugzilla.redhat.com/show_bug.cgi?id=1962511 to track the same.

Regards,
Vasishta Shastry
QE, ceph

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294