Bug 1468840 - RHCS parameter setuser_match_path = /var/lib/ceph/$type/$cluster-$id in ceph.conf causes add of new OSD to fail
RHCS parameter setuser_match_path = /var/lib/ceph/$type/$cluster-$id in ceph....
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RADOS (Show other bugs)
All All
unspecified Severity low
: rc
: 2.5
Assigned To: Josh Durgin
Depends On:
  Show dependency treegraph
Reported: 2017-07-08 15:04 EDT by jquinn
Modified: 2017-10-21 12:55 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3121291 None None None 2017-07-24 08:16 EDT
Ceph Project Bug Tracker 19642 None None None 2017-07-08 15:04 EDT

  None (edit)
Description jquinn 2017-07-08 15:04:55 EDT
Description of problem: customer is upgrading from RHCS 1.3.x to RHCS 2.x, but they wish to avoid the chown that is needed on /var/lib/ceph/osd due to the time it takes to complete.  In RHCS 1.3.x, the OSD daemons run as root, while in RHCS 2.x they run as ceph user.  The directories for each of the OSD's (/var/lib/ceph/osd/ceph-x) is owned by root, so in order for these processes to stop/start/run they set setuser_match_path = /var/lib/ceph/$type/$cluster-$id in the ceph.conf under [osd].  This worked for the existing osd processes and they are running with no issues.  

The problem they are running into is that when ceph-disk (or ceph-ansible) runs the install of the OSD's it tries to install the processes with 'ceph' user and it fails with the below error. 

Error: journal specified but not allowed by osd backend

The reason for this error is because the setuser_match_path values is being used for existing osd's and it attempts to use it for the new osd creation as well. To work around this we moved the parameter to a [osd.x] section for each of the existing osd's.  Once the value was removed from the [osd] section then ceph-disk was able to add the osd without issue. 

Note: We had to add the [osd.x] sections with the parameters to the group_vars/all.yml under the ceph.conf override section so they did not get overwritten. 

We believe that this option should be used only when the /var/lib/ceph/osd/ceph-x directory exists rather than being used as a global parameter for old and new OSD's.  Maybe there should be a user_default = ceph type option that it takes when the directory does not exist.  

I did discuss with the customer that it's not supported to be running in this manner as we don't know any future unforseen issues that may arise having 2 different users running the daemons.  Their plan is to replace the OSD's as they fail and they will be using the ceph user with the new OSD.  If you know of any issues that already exist please let me know and I will forward that along. 

Version-Release number of selected component (if applicable):RHCS 3.x

How reproducible:every time

Steps to Reproduce:
1.Upgrade from 1.3.x to 2.x and do not perform the chown as part of the upgrade. 
2.set setuser_match_path = /var/lib/ceph/$type/$cluster-$id  under [osd] in the ceph.conf
3.Try to use ceph-disk or ceph-ansible to add a new osd. 

Actual results:Install fails with 
[ceph@ceph-osd1 osd]$ sudo ceph-disk --setuser ceph --setgroup ceph prepare --fs-type xfs /dev/vdb /dev/vda
ceph-disk: Error: journal specified but not allowed by osd backend
[ceph@ceph-osd1 osd]$ 

Expected results:Below is when the parameter is commented out in the ceph.conf, but without having it specified for each [osd.x] section the osd's will fail to restart. 

[ceph@ceph-osd1 lib]$ sudo ceph-disk --setuser ceph --setgroup ceph prepare --fs-type xfs /dev/vdb /dev/vda
prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/vdb1              isize=2048   agcount=4, agsize=655295 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2621179, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
[ceph@ceph-osd1 lib]$ 

Additional info:As a note, they were using a 'ceph' user to login and run the scripts using sudo.  I do not believe this affected anything in this situation, but wanted to be sure to note it.
Comment 5 Loic Dachary 2017-09-26 04:58:25 EDT
Moving to RADOS : the error is reported by ceph-disk but comes from ceph-osd

Note You need to log in before you can comment on or make changes to this bug.