Bug 1323232

What is the contents of ceph.conf on these nodes? Did you change the 'admin socket' setting there, or was it set up by ceph-ansible?

Does the /var/run/ceph/rbd-clients/ directory exist?

If it was set up automatically, we need to adjust the packaging to create that directory and label it with the proper selinux settings.

(In reply to Josh Durgin from comment #2)
> What is the contents of ceph.conf on these nodes? Did you change the 'admin
> socket' setting there, or was it set up by ceph-ansible?

Everything was setup by ceph-ansible. I didn't change anything in ceph.conf

> 
> Does the /var/run/ceph/rbd-clients/ directory exist?
[root@magna105 ~]# ll /var/run/ceph/rbd-clients/
ls: cannot access /var/run/ceph/rbd-clients/: No such file or directory

> 
> If it was set up automatically, we need to adjust the packaging to create
> that directory and label it with the proper selinux settings.


 ceph.conf
==============

# Please do not change this file directly since it is managed by Ansible and will be overwritten

[global]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
cephx require signatures = True # Kernel RBD does NOT support signatures!
cephx cluster require signatures = True
cephx service require signatures = False
fsid = e547f093-99ba-44e5-b63a-317efa3ecfb3
max open files = 131072
osd pool default pg num = 128
osd pool default pgp num = 128
osd pool default size = 2
osd pool default min size = 1
osd pool default crush rule = 0
# Disable in-memory logs
debug_lockdep = 0/0
debug_context = 0/0
debug_crush = 0/0
debug_buffer = 0/0
debug_timer = 0/0
debug_filer = 0/0
debug_objecter = 0/0
debug_rados = 0/0
debug_rbd = 0/0
debug_journaler = 0/0
debug_objectcatcher = 0/0
debug_client = 0/0
debug_osd = 0/0
debug_optracker = 0/0
debug_objclass = 0/0
debug_filestore = 0/0
debug_journal = 0/0
debug_ms = 0/0
debug_monc = 0/0
debug_tp = 0/0
debug_auth = 0/0
debug_finisher = 0/0
debug_heartbeatmap = 0/0
debug_perfcounter = 0/0
debug_asok = 0/0
debug_throttle = 0/0
debug_mon = 20/0
debug_paxos = 0/0
debug_rgw = 0/0

[client]
rbd cache = true
rbd cache writethrough until flush = true
rbd concurrent management ops = 20
admin socket = /var/run/ceph/rbd-clients/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor
log file = /var/log/rbd-clients/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor
rbd default map options = rw
rbd default features = 3 # sum features digits
rbd default format = 2

[mon]
mon osd down out interval = 600
mon osd min down reporters = 7
mon clock drift allowed = 0.15
mon clock drift warn backoff = 30
mon osd full ratio = 0.95
mon osd nearfull ratio = 0.85
mon osd report timeout = 300
mon pg warn max per osd = 0
mon osd allow primary affinity = true
mon pg warn max object skew = 10

[mon.magna105]
host = magna105
mon addr = 10.8.128.105
[mon.magna107]
host = magna107
mon addr = 10.8.128.107
[mon.magna108]
host = magna108
mon addr = 10.8.128.108

[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 10000
cluster_network = 10.8.128.0/21
public_network = 10.8.128.0/21
osd mon heartbeat interval = 30
# Performance tuning
filestore merge threshold = 40
filestore split multiple = 8
osd op threads = 8
filestore op threads = 8
filestore max sync interval = 5
osd max scrubs = 1
osd scrub begin hour = 0
osd scrub end hour = 24
# Recovery tuning
osd recovery max active = 5
osd max backfills = 2
osd recovery op priority = 2
osd recovery max chunk = 1048576
osd recovery threads = 1
osd objectstore = filestore
osd crush update on start = true
# Deep scrub impact
osd scrub sleep = 0.1
osd disk thread ioprio class = idle
osd disk thread ioprio priority = 0
osd scrub chunk max = 5
osd deep scrub stride = 1048576

Ok, those are a bunch of settings that we mostly don't want in ceph.conf, since the defaults in ceph itself are more appropriate for jewel.

Added a ceph-ansible ticket here: 

https://github.com/ceph/ceph-ansible/issues/693

and commented here for the selinux issue:

https://github.com/ceph/ceph-ansible/issues/687

There appear to be other ansible bzs assigned to gmeno, piling on.

Andrew, Would you please send status on where we are with this upstream?

A pull request for this has been opened upstream here: https://github.com/ceph/ceph-ansible/pull/694

Looks like that PR 694 needs a rebase

The PR for this has been merged upstream: https://github.com/ceph/ceph-ansible/pull/694

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2016:1754