Bug 1269403

Summary: ceph-deploy error during OSD prepare while installing ceph 1.3.1 on RHEL 7.1
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: shilpa <smanjara>
Component: Ceph-InstallerAssignee: Travis Rhoden <trhoden>
Status: CLOSED NOTABUG QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 1.3.1CC: adeza, aschoen, ceph-eng-bugs, nthomas, sankarshan
Target Milestone: rc   
Target Release: 1.3.2   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-12 09:58:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description shilpa 2015-10-07 08:41:06 UTC
Description of problem:


ceph-deploy prepare command has an error: "partx: /dev/sdb: error adding partitions 1-2" which is flagged as a WARNING. 

See ceph health warnings. The OSD's are up though. 

#ceph health
HEALTH_WARN 64 pgs degraded; 64 pgs stuck degraded; 64 pgs stuck unclean; 64 pgs stuck undersized; 64 pgs undersized

Error snippet:

[magna075][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdb
[magna075][WARNIN] partx: /dev/sdb: error adding partitions 1-2



# ceph-deploy osd prepare magna075:/dev/sdb


[magna075][INFO  ] Running command: ceph-disk -v prepare --fs-type xfs --cluster ceph -- /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
[magna075][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[magna075][WARNIN] DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:5120M --change-name=2:ceph journal --partition-guid=2:d1a8fd80-ddb4-4673-b5dd-7d3763a44994 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
[magna075][DEBUG ] The operation has completed successfully.
[magna075][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdb
[magna075][WARNIN] partx: /dev/sdb: error adding partition 2
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/udevadm settle
[magna075][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/d1a8fd80-ddb4-4673-b5dd-7d3763a44994
[magna075][WARNIN] DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/d1a8fd80-ddb4-4673-b5dd-7d3763a44994
[magna075][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:0669b7f9-a33e-4689-928b-33d51c09f5c6 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/sdb
[magna075][DEBUG ] The operation has completed successfully.
[magna075][WARNIN] INFO:ceph-disk:calling partx on created device /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdb
[magna075][WARNIN] partx: /dev/sdb: error adding partitions 1-2
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/udevadm settle
[magna075][WARNIN] DEBUG:ceph-disk:Creating xfs fs on /dev/sdb1
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdb1
[magna075][DEBUG ] meta-data=/dev/sdb1              isize=2048   agcount=4, agsize=60719917 blks
[magna075][DEBUG ]          =                       sectsz=512   attr=2, projid32bit=1
[magna075][DEBUG ]          =                       crc=0        finobt=0
[magna075][DEBUG ] data     =                       bsize=4096   blocks=242879665, imaxpct=25
[magna075][DEBUG ]          =                       sunit=0      swidth=0 blks
[magna075][DEBUG ] naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
[magna075][DEBUG ] log      =internal log           bsize=4096   blocks=118593, version=2
[magna075][DEBUG ]          =                       sectsz=512   sunit=0 blks, lazy-count=1
[magna075][DEBUG ] realtime =none                   extsz=4096   blocks=0, rtextents=0
[magna075][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.nbe7yk with options noatime,inode64
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdb1 /var/lib/ceph/tmp/mnt.nbe7yk
[magna075][WARNIN] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.nbe7yk
[magna075][WARNIN] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.nbe7yk/journal -> /dev/disk/by-partuuid/d1a8fd80-ddb4-4673-b5dd-7d3763a44994
[magna075][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.nbe7yk
[magna075][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.nbe7yk
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdb
[magna075][DEBUG ] The operation has completed successfully.
[magna075][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb
[magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors
[magna075][WARNIN] INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdb
[magna075][WARNIN] partx: /dev/sdb: error adding partitions 1-2
[magna075][INFO  ] checking OSD status...
[magna075][INFO  ] Running command: ceph --cluster=ceph osd stat --format=json


#ceph osd tree

# ceph osd tree
ID WEIGHT  TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 1.79999 root default                                        
-2 0.89999     host magna062                                   
 0 0.89999         osd.0          up  1.00000          1.00000 
-3 0.89999     host magna075                                   
 1 0.89999         osd.1          up  1.00000          1.00000

Comment 2 Alfredo Deza 2015-10-07 11:34:47 UTC
This is known behavior from ceph-deploy and the output is already saying that partx will complain (form your output):

[magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display errors

In this case /dev/sdb2 is the one that is already known and indeed, it displays an error:

[magna075][WARNIN] partx: /dev/sdb: error adding partition 2

The origin of this behavior in RHEL is described in http://tracker.ceph.com/issues/7334

The process of deploying OSDs checks for their status and making sure they are all "up", "in", and not full or near full which the output seems to say is not the case.

I am not sure why the health warning but I am inclined to say this is not because there was a failure in deploying the OSD.

Comment 3 shilpa 2015-10-12 09:58:17 UTC
(In reply to Alfredo Deza from comment #2)
> This is known behavior from ceph-deploy and the output is already saying
> that partx will complain (form your output):
> 
> [magna075][WARNIN] INFO:ceph-disk:re-reading known partitions will display
> errors
> 
> In this case /dev/sdb2 is the one that is already known and indeed, it
> displays an error:
> 
> [magna075][WARNIN] partx: /dev/sdb: error adding partition 2
> 
> The origin of this behavior in RHEL is described in
> http://tracker.ceph.com/issues/7334
> 
> The process of deploying OSDs checks for their status and making sure they
> are all "up", "in", and not full or near full which the output seems to say
> is not the case.
> 
> I am not sure why the health warning but I am inclined to say this is not
> because there was a failure in deploying the OSD.

Understood. Thanks Alfredo. Just wanted to make sure health warning has nothing to do with this. Closing Not a bug.