Bug 1378090

I think to remember there was an issue with creating partitions on devicemapper devices but it's a vague rememberance. Are you able to create a partition table on the /dev/dm-0 device manually with sgdisk or parted ?

hi Loic,

    Yes, I am able to create partition table on the mapper device.

Also from the "osd create" command, the partitions are created:
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/mapper/cry3: 1000.2 GB, 1000202788864 bytes, 1953521072 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1     20482048   1953521038  921.8G  unknown         ceph data
 2         2048     20480000    9.8G  unknown         ceph journal


But still ceph disk throws a warning that the partition doesnt exist:
magna061][WARNIN] ceph-disk: Error: partition 1 for /dev/dm-0 does not appear to exist

Thanks,
Tejas

Hi Loic,

    We are able to  create partitions manually on encrypted disks.
However the partitions created by ceph-deploy on the encrypted disk are not seen , and its fails to create the xfs filesystem.

Thanks,
Tejas

We should first verify that the partition devices are created:

It's worth checking if

   ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/mapper/cry1

throws the same error as

   ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/dm-0

And also ls -ltr /dev/mapper/cry1* to verify if /dev/mapper/cry1p1 /dev/mapper/cry1p2 exist. In order to avoid device name parsing issues, I suggest naming the encrypted device cry instead of cry1 because the number could possibly be mistaken for a partition number. I'd have to check the code to be sure but let's be on the safe side.

You should not use ceph-deploy for these tests to reduce the number of layers.

@Tejas if you provide me with access to a machine where this can be reproduced, I will be able to figure it out by experimenting myself. Thanks !

TL;DR: I don't see how we can make it work with ceph-disk, there are too many issues. The last resort is to use ceph-osd --mkfs manually.

Using /dev/mapper/swift instead of /dev/dm-0 only marginally improves things.

[root@magna061 ~]# ceph-disk -v prepare /dev/mapper/swift
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
INFO:ceph-disk:Will colocate journal with data on /dev/mapper/swift
DEBUG:ceph-disk:Creating journal partition num 2 size 10000 on /dev/mapper/swift
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:10000M --change-name=2:ceph journal --partition-guid=2:19c3a473-2d4b-488c-bc9d-0e13e51c9914 --typecode=2:45b0969e-9b03-\
4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/mapper/swift
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/mapper/swift
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/mapper/swift
partx: /dev/mapper/swift: error adding partition 2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/19c3a473-2d4b-488c-bc9d-0e13e51c9914
DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/19c3a473-2d4b-488c-bc9d-0e13e51c9914
DEBUG:ceph-disk:Creating osd partition on /dev/mapper/swift
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:f44cc730-5726-4c69-a872-669c7541030e --typecode=1:89c57f98-2fe5-4dc0\
-89c1-f3ad0ceff2be -- /dev/mapper/swift
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
INFO:ceph-disk:calling partx on created device /dev/mapper/swift
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/mapper/swift
partx: /dev/mapper/swift: error adding partitions 1-2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
ceph-disk: Error: partition 1 for /dev/mapper/swift does not appear to exist
[root@magna061 ~]# sgdisk --print /dev/mapper/swift
Disk /dev/mapper/swift: 1953521072 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 5A07CDAE-970B-4550-BCE0-B863A9591D8E
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953521038
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1        20482048      1953521038   921.7 GiB   FFFF  ceph data
   2            2048        20480000   9.8 GiB     FFFF  ceph journal

The problem here is that partx -a does not handle the devicemapper devices and we should use partprobe instead.

[root@magna061 ~]# partprobe /dev/mapper/swift
[root@magna061 ~]# ls -l /dev/mapper/swift*
lrwxrwxrwx. 1 root root 7 Sep 22 16:24 /dev/mapper/swift -> ../dm-0
lrwxrwxrwx. 1 root root 7 Sep 22 16:24 /dev/mapper/swift1 -> ../dm-1
lrwxrwxrwx. 1 root root 7 Sep 22 16:24 /dev/mapper/swift2 -> ../dm-2

we now have the expected block devices. The preparation has been interrupted but let see how far we can get by monkey patching ceph-disk to use partprobe instead of partx.

(same as above after ceph-disk zap /dev/mapper/swift)
DEBUG:ceph-disk:Calling partprobe on created device /dev/mapper/swift
INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/mapper/swift
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
ceph-disk: Error: partition 1 for /dev/mapper/swift does not appear to exist

Now we hit another problem: to figure out the partition name ceph-disk uses /sys/block and expects the names to refect the partitions. However we have:

[root@magna061 ~]# ls -l /sys/block
total 0
lrwxrwxrwx. 1 root root 0 Sep 22 11:28 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx. 1 root root 0 Sep 22 16:35 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx. 1 root root 0 Sep 22 16:35 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx. 1 root root 0 Sep 22 11:10 sda -> ../devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda
lrwxrwxrwx. 1 root root 0 Sep 22 11:10 sdb -> ../devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb
lrwxrwxrwx. 1 root root 0 Sep 22 11:10 sdc -> ../devices/pci0000:00/0000:00:1f.2/ata3/host2/target2:0:0/2:0:0:0/block/sdc
lrwxrwxrwx. 1 root root 0 Sep 22 11:10 sdd -> ../devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sdd

and only the cannonical devicemapper names are shown here.

For the record these issues have been fixed in infernalis (the release after 0.94) but backporting the changes would be too much of a risk: they are extensive.

Should we try to find a way to do that manually using ceph-osd --mkfs ?

IMO if we can get the ceph-deploy --dmcrypt method to the customer without the need for a OSD reboot, that would be better than pursuing this:
https://bugzilla.redhat.com/show_bug.cgi?id=1377639

Harish can you also pitch in on this?

Thanks,
Tejas

#13: yes, it would be better, but it seems too late in the Hammer cycle to do this.

We will use the process outlined by Loic in #11 as our setup process for encrypted OSDs. Please test those steps, and document these in the corresponding Doc bug.

For the record https://bugzilla.redhat.com/show_bug.cgi?id=1377639 is closed and this should probably be closed too.