Description of problem: [Ubuntu] ceph-deploy-1.5.27.4-4redhat1 failing to add osd with dmcrypt flag Version-Release number of selected component (if applicable): Red Hat Ceph Storage 1.3.2 Version: 1.5.27.4-4redhat1 udevadm version 204 dpkg -s ceph | grep Version Version: 0.94.5-5redhat1trusty How reproducible: Not always but yes mostly when we add two OSDs one by one using same journal disk $ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdc:/dev/vde $ ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdd:/dev/vde Like Here Journal disk is same /dev/vde .
########### successful deployment of OSD ceph@ceph1:~/my-cluster$ ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdc:/dev/vde [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/my-cluster/cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.27.4): /usr/bin/ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdc:/dev/vde [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] disk : [('ceph7', '/dev/vdc', '/dev/vde')] [ceph_deploy.cli][INFO ] dmcrypt : True [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : True [ceph_deploy.cli][INFO ] subcommand : create [ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fea1a4a2bd8> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] fs_type : xfs [ceph_deploy.cli][INFO ] func : <function osd at 0x7fea1a4ebb90> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] zap_disk : False [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph7:/dev/vdc:/dev/vde [ceph7][DEBUG ] connection detected need for sudo [ceph7][DEBUG ] connected to host: ceph7 [ceph7][DEBUG ] detect platform information from remote host [ceph7][DEBUG ] detect machine type [ceph7][DEBUG ] find the location of an executable [ceph7][INFO ] Running command: sudo /sbin/initctl version [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty [ceph_deploy.osd][DEBUG ] Deploying osd to ceph7 [ceph7][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph7][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add [ceph_deploy.osd][DEBUG ] Preparing host ceph7 disk /dev/vdc journal /dev/vde activate True [ceph7][INFO ] Running command: sudo ceph-disk -v prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph --fs-type xfs -- /dev/vdc /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/parted --machine -- /dev/vde print [ceph7][WARNIN] INFO:ceph-disk:cannot read partition index; assume it isn't present [ceph7][WARNIN] (Error: Command '/sbin/parted' returned non-zero exit status 1) [ceph7][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data [ceph7][WARNIN] DEBUG:ceph-disk:Creating journal partition num 1 size 5120 on /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --new=1:0:+5120M --change-name=1:ceph journal --partition-guid=1:bfb1d2e3-ade4-4e7e-a24a-286815617aa3 --typecode=1:89c57f98-2fe5-4dc0-89c1-35865ceff2be --mbrtogpt -- /dev/vde [ceph7][DEBUG ] Creating new GPT entries. [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle [ceph7][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/bfb1d2e3-ade4-4e7e-a24a-286815617aa3 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/bfb1d2e3-ade4-4e7e-a24a-286815617aa3.luks.key luksFormat /dev/disk/by-partuuid/bfb1d2e3-ade4-4e7e-a24a-286815617aa3 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:45b0969e-9b03-4f30-b4c6-35865ceff106 -- /dev/vde [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/bfb1d2e3-ade4-4e7e-a24a-286815617aa3 [ceph7][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/vdc [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:3ca23d0c-2540-4bf4-a914-7ce16d666696 --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/vdc [ceph7][DEBUG ] Creating new GPT entries. [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on created device /dev/vdc [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vdc [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/3ca23d0c-2540-4bf4-a914-7ce16d666696.luks.key luksFormat /dev/vdc1 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/3ca23d0c-2540-4bf4-a914-7ce16d666696.luks.key luksOpen /dev/vdc1 3ca23d0c-2540-4bf4-a914-7ce16d666696 [ceph7][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/mapper/3ca23d0c-2540-4bf4-a914-7ce16d666696 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/mapper/3ca23d0c-2540-4bf4-a914-7ce16d666696 [ceph7][DEBUG ] meta-data=/dev/mapper/3ca23d0c-2540-4bf4-a914-7ce16d666696 isize=2048 agcount=4, agsize=1310527 blks [ceph7][DEBUG ] = sectsz=512 attr=2, projid32bit=0 [ceph7][DEBUG ] data = bsize=4096 blocks=5242107, imaxpct=25 [ceph7][DEBUG ] = sunit=0 swidth=0 blks [ceph7][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 [ceph7][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2 [ceph7][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1 [ceph7][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0 [ceph7][WARNIN] DEBUG:ceph-disk:Mounting /dev/mapper/3ca23d0c-2540-4bf4-a914-7ce16d666696 on /var/lib/ceph/tmp/mnt.fe5Dsy with options noatime,inode64 [ceph7][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/mapper/3ca23d0c-2540-4bf4-a914-7ce16d666696 /var/lib/ceph/tmp/mnt.fe5Dsy [ceph7][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.fe5Dsy [ceph7][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.fe5Dsy/journal -> /dev/mapper/bfb1d2e3-ade4-4e7e-a24a-286815617aa3 [ceph7][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.fe5Dsy/journal_dmcrypt -> /dev/disk/by-partuuid/bfb1d2e3-ade4-4e7e-a24a-286815617aa3 [ceph7][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.fe5Dsy [ceph7][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.fe5Dsy [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup remove 3ca23d0c-2540-4bf4-a914-7ce16d666696 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/vdc [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/vdc [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vdc [ceph7][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add [ceph7][INFO ] checking OSD status... [ceph7][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json [ceph_deploy.osd][DEBUG ] Host ceph7 is now ready for osd use.
########### Unsuccessful deployment of OSD ############### ceph@ceph1:~/my-cluster$ ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdd:/dev/vde [ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/my-cluster/cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.27.4): /usr/bin/ceph-deploy --overwrite-conf osd create --dmcrypt ceph7:/dev/vdd:/dev/vde [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] disk : [('ceph7', '/dev/vdd', '/dev/vde')] [ceph_deploy.cli][INFO ] dmcrypt : True [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : True [ceph_deploy.cli][INFO ] subcommand : create [ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe697c53bd8> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] fs_type : xfs [ceph_deploy.cli][INFO ] func : <function osd at 0x7fe697c9cb90> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] zap_disk : False [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph7:/dev/vdd:/dev/vde [ceph7][DEBUG ] connection detected need for sudo [ceph7][DEBUG ] connected to host: ceph7 [ceph7][DEBUG ] detect platform information from remote host [ceph7][DEBUG ] detect machine type [ceph7][DEBUG ] find the location of an executable [ceph7][INFO ] Running command: sudo /sbin/initctl version [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty [ceph_deploy.osd][DEBUG ] Deploying osd to ceph7 [ceph7][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph7][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add [ceph_deploy.osd][DEBUG ] Preparing host ceph7 disk /dev/vdd journal /dev/vde activate True [ceph7][INFO ] Running command: sudo ceph-disk -v prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys --cluster ceph --fs-type xfs -- /dev/vdd /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size [ceph7][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/parted --machine -- /dev/vde print [ceph7][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data [ceph7][WARNIN] DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:d7053fc0-69ed-4dfc-b189-d93114d24d38 --typecode=2:89c57f98-2fe5-4dc0-89c1-35865ceff2be --mbrtogpt -- /dev/vde [ceph7][DEBUG ] Warning: The kernel is still using the old partition table. [ceph7][DEBUG ] The new table will be used at the next reboot. [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vde [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle [ceph7][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/d7053fc0-69ed-4dfc-b189-d93114d24d38 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/d7053fc0-69ed-4dfc-b189-d93114d24d38.luks.key luksFormat /dev/disk/by-partuuid/d7053fc0-69ed-4dfc-b189-d93114d24d38 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=2:45b0969e-9b03-4f30-b4c6-35865ceff106 -- /dev/vde [ceph7][DEBUG ] Warning: The kernel is still using the old partition table. [ceph7][DEBUG ] The new table will be used at the next reboot. [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/d7053fc0-69ed-4dfc-b189-d93114d24d38 [ceph7][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/vdd [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:2f0d1bb9-3ca3-4705-a134-72a77a9df768 --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/vdd [ceph7][DEBUG ] Creating new GPT entries. [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on created device /dev/vdd [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vdd [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/2f0d1bb9-3ca3-4705-a134-72a77a9df768.luks.key luksFormat /dev/vdd1 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/2f0d1bb9-3ca3-4705-a134-72a77a9df768.luks.key luksOpen /dev/vdd1 2f0d1bb9-3ca3-4705-a134-72a77a9df768 [ceph7][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/mapper/2f0d1bb9-3ca3-4705-a134-72a77a9df768 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/mapper/2f0d1bb9-3ca3-4705-a134-72a77a9df768 [ceph7][DEBUG ] meta-data=/dev/mapper/2f0d1bb9-3ca3-4705-a134-72a77a9df768 isize=2048 agcount=4, agsize=1310527 blks [ceph7][DEBUG ] = sectsz=512 attr=2, projid32bit=0 [ceph7][DEBUG ] data = bsize=4096 blocks=5242107, imaxpct=25 [ceph7][DEBUG ] = sunit=0 swidth=0 blks [ceph7][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 [ceph7][DEBUG ] log =internal log bsize=4096 blocks=2560, version=2 [ceph7][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1 [ceph7][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0 [ceph7][WARNIN] DEBUG:ceph-disk:Mounting /dev/mapper/2f0d1bb9-3ca3-4705-a134-72a77a9df768 on /var/lib/ceph/tmp/mnt.dRsNsU with options noatime,inode64 [ceph7][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/mapper/2f0d1bb9-3ca3-4705-a134-72a77a9df768 /var/lib/ceph/tmp/mnt.dRsNsU [ceph7][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.dRsNsU [ceph7][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.dRsNsU/journal -> /dev/mapper/d7053fc0-69ed-4dfc-b189-d93114d24d38 [ceph7][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.dRsNsU/journal_dmcrypt -> /dev/disk/by-partuuid/d7053fc0-69ed-4dfc-b189-d93114d24d38 [ceph7][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.dRsNsU [ceph7][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.dRsNsU [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup remove 2f0d1bb9-3ca3-4705-a134-72a77a9df768 [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/vdd [ceph7][DEBUG ] The operation has completed successfully. [ceph7][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/vdd [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vdd [ceph7][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=add [ceph7][INFO ] checking OSD status... [ceph7][INFO ] Running command: sudo ceph --cluster=ceph osd stat --format=json [ceph_deploy.osd][DEBUG ] Host ceph7 is now ready for osd use. ####################################################### /dev/vdb1 15G 47M 15G 1% /var/lib/ceph/osd/ceph-2 /dev/dm-3 20G 46M 20G 1% /var/lib/ceph/osd/ceph-5 root@ceph3:~# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.08995 root default -2 0.02998 host ceph5 0 0.00999 osd.0 up 1.00000 1.00000 3 0.00999 osd.3 up 1.00000 1.00000 6 0.00999 osd.6 up 1.00000 1.00000 -3 0.02998 host ceph6 1 0.00999 osd.1 up 1.00000 1.00000 4 0.00999 osd.4 up 1.00000 1.00000 7 0.00999 osd.7 up 1.00000 1.00000 -4 0.02998 host ceph7 2 0.00999 osd.2 up 1.00000 1.00000 5 0.01999 osd.5 up 1.00000 1.00000 8 0 osd.8 down 0 1.00000 ** OSD.8 did not get activated **
######## As a workaround for now , ceph-disk activate can be used ############# 1. check the dm id for failed to activate the OSD disk in ceph-deploy log for example here it is : [ceph7][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:2f0d1bb9-3ca3-4705-a134-72a77a9df768 --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/vdd Like here it is : 2f0d1bb9-3ca3-4705-a134-72a77a9df768 2. Then go to OSD node and dmsetup ls command you will come to know the major number of this device $ dmsetup ls ceph7--vg-root (252:0) ceph7--vg-swap_1 (252:1) 2f0d1bb9-3ca3-4705-a134-72a77a9df768 (252:4) <================= bfb1d2e3-ade4-4e7e-a24a-286815617aa3 (252:2) d7053fc0-69ed-4dfc-b189-d93114d24d38 (252:5) 3ca23d0c-2540-4bf4-a914-7ce16d666696 (252:3) Like here it is : 4 3. Run ceph-disk activate $ ceph-disk activate /dev/dm-4 got monmap epoch 1 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device 2016-04-15 19:11:15.206517 7fb3a9f1f8c0 -1 journal read_header error decoding journal header HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device 2016-04-15 19:11:15.262500 7fb3a9f1f8c0 -1 filestore(/var/lib/ceph/tmp/mnt.4JAsbF) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2016-04-15 19:11:15.279878 7fb3a9f1f8c0 -1 created object store /var/lib/ceph/tmp/mnt.4JAsbF journal /var/lib/ceph/tmp/mnt.4JAsbF/journal for osd.8 fsid 4bb2a4a5-44f1-4fac-9ec4-19aef317dc17 2016-04-15 19:11:15.279923 7fb3a9f1f8c0 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.4JAsbF/keyring: can't open /var/lib/ceph/tmp/mnt.4JAsbF/keyring: (2) No such file or directory 2016-04-15 19:11:15.280066 7fb3a9f1f8c0 -1 created new key in keyring /var/lib/ceph/tmp/mnt.4JAsbF/keyring added key for osd.8 <====== it gives some error that tmp mount is not found but finally activates the OSD. ############################## # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.10994 root default -2 0.02998 host ceph5 0 0.00999 osd.0 up 1.00000 1.00000 3 0.00999 osd.3 up 1.00000 1.00000 6 0.00999 osd.6 up 1.00000 1.00000 -3 0.02998 host ceph6 1 0.00999 osd.1 up 1.00000 1.00000 4 0.00999 osd.4 up 1.00000 1.00000 7 0.00999 osd.7 up 1.00000 1.00000 -4 0.04997 host ceph7 2 0.00999 osd.2 up 1.00000 1.00000 5 0.01999 osd.5 up 1.00000 1.00000 8 0.01999 osd.8 up 1.00000 1.00000 root@ceph3:~# ceph -s cluster 4bb2a4a5-44f1-4fac-9ec4-19aef317dc17 health HEALTH_OK monmap e1: 3 mons at {ceph2=10.65.2.238:6789/0,ceph3=10.65.2.241:6789/0,ceph4=10.65.2.243:6789/0} election epoch 1004, quorum 0,1,2 ceph2,ceph3,ceph4 osdmap e740: 9 osds: 9 up, 9 in pgmap v21194: 192 pgs, 3 pools, 0 bytes data, 0 objects 472 MB used, 144 GB / 144 GB avail 192 active+clean $ df -h Filesystem Size Used Avail Use% Mounted on udev 989M 4.0K 989M 1% /dev tmpfs 200M 464K 200M 1% /run /dev/dm-0 18G 1.6G 15G 10% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 1000M 0 1000M 0% /run/shm none 100M 0 100M 0% /run/user /dev/vda1 236M 41M 183M 19% /boot /dev/vdb1 15G 48M 15G 1% /var/lib/ceph/osd/ceph-2 /dev/dm-3 20G 47M 20G 1% /var/lib/ceph/osd/ceph-5 /dev/dm-4 20G 46M 20G 1% /var/lib/ceph/osd/ceph-8
I *think* this is related to the extra dmcrypt calls that happens after partprobe has already been called. In the below extract, /dev/vde is being prepared, and partprobe and udevadm is called after it finishes: [ceph6][WARNIN] DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/vde [ceph6][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:fa8f2e0a-f3e7-42b2-96b6-579cd39f0a74 --typecode=2:89c57f98-2fe5-4dc0-89c1-35865ceff2be --mbrtogpt -- /dev/vde [ceph6][DEBUG ] Warning: The kernel is still using the old partition table. [ceph6][DEBUG ] The new table will be used at the next reboot. [ceph6][DEBUG ] The operation has completed successfully. [ceph6][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/vde [ceph6][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/vde [ceph6][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle But then the dmcrypt setup continues *without* calling partprobe: [ceph6][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/fa8f2e0a-f3e7-42b2-96b6-579cd39f0a74 [ceph6][WARNIN] INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/fa8f2e0a-f3e7-42b2-96b6-579cd39f0a74.luks.key luksFormat /dev/disk/by-partuuid/fa8f2e0a-f3e7-42b2-96b6-579cd39f0a74 [ceph6][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=2:45b0969e-9b03-4f30-b4c6-35865ceff106 -- /dev/vde [ceph6][DEBUG ] Warning: The kernel is still using the old partition table. [ceph6][DEBUG ] The new table will be used at the next reboot. [ceph6][DEBUG ] The operation has completed successfully. And finally it continues for the next device: [ceph6][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/fa8f2e0a-f3e7-42b2-96b6-579cd39f0a74 [ceph6][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/vdd This makes me think that partprobe/udevadm needs to be called after that as well. The partitions do get created but the changes require a reboot (maybe the ticket title should get updated). Re-assigning to loic since he might know better what can be done here to improve this with my findings.
With udevadm 204, there should be no unexpected indirect calls to partprobe. However, IIRC, ceph-deploy calls partprobe (there is a tracker issue about that) and that can cause misbehavior like this one.
(In reply to Loic Dachary from comment #30) > With udevadm 204, there should be no unexpected indirect calls to partprobe. > However, IIRC, ceph-deploy calls partprobe (there is a tracker issue about > that) and that can cause misbehavior like this one. ceph-deploy only does that when "zapping" a device, not when creating an OSD. The issue here is that when using dmcrypt ceph-disk is failing to call partprobe/partx correctly. As demonstrated by the log output ceph-deploy is not doing any calls to partprobe/partx or udevadm, it is all ceph-disk. The tracker issue referenced (for using partprobe/partx when zapping) is: http://tracker.ceph.com/issues/14099
Is there a ceph-disk reproducer for this problem ? I.e. not involving ceph-deploy at all ?
- I have tested it with `ceph-disk` and as discussed it with Alfredo it is at ceph-disk (patprobe) layer. - ceph-disk is failing to create second journal partition in same journal device which is used by another OSD. ~~~ Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. ~~~ ^^ With same warning. - I am adding the logs in next comment, file name is : ceph-disk_second_journal_partition_add_failure.txt - Then I tried doing the partprobe to journal device before going for second osd addition then also it got failed with same Warning : ~~~ Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. ~~~ - adding the logs in next comment , file name is : ceph-disk_second_journal_partition_add_failure_with_partprobe.txt - After reboot second journal partition is available and OSDs are up. - after_reboot_success.txt
Created attachment 1160660 [details] ceph disk second journal partition add failure
Created attachment 1160661 [details] ceph disk second journal partition add failure with partprobe
Created attachment 1160662 [details] after reboot
It looks good to me.
Thank you Bara and Loic. Looks perfect!