ceph.conf file from the test run

[global]
fsid = 08c6a829-69fb-4db1-b5c8-9d137048ae03
mon_initial_members = clara007, clara012
mon_host = 10.8.129.7,10.8.129.12
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
mon_pg_warn_min_per_osd = 2

[osd]
debug_ms = 1
debug_journal = 20
osd_op_thread_timeout = 60
debug_osd = 20
debug_filestore = 20
osd_sloppy_crc = True
 
[mon]
osd_default_pool_size = 2
debug_mon = 1
debug_paxos = 20
debug_ms = 20
 
[client]
log_file = /var/log/ceph/ceph-$name.$pid.log

OK, based on the following lines in doc/rados/configuration/general-config-ref.rst


"""
``pid file``
:Description: The file in which the mon, osd or mds will write its
              PID.  For instance, ``/var/run/$cluster/$type.$id.pid``
              will create /var/run/ceph/mon.a.pid for the ``mon`` with
              id ``a`` running in the ``ceph`` cluster. The ``pid
              file`` is removed when the daemon stops gracefully. If 
              the process is not daemonized (i.e. runs with the ``-f``
              or ``-d`` option), the ``pid file`` is not created. 
:Type: String
:Required: No 
:Default: No 
"""


the pid file location is configurable and it is not created by default. Therefore, one of the tests must have specified it to be created in /var/run/<something>.pid instead of /var/run/ceph/<something>.pid which means that we should fix the test case, not the policy.

The test case can either continue to create the pid files in that location and afterwards, chcon to the ceph_var_run_t type or can put the pid files in /var/run/ceph/ directory where the labelling will be done automatically by kernel itself.

(In reply to Boris Ranto from comment #3)
> or can put the pid files in
> /var/run/ceph/ directory where the labelling will be done automatically by
> kernel itself.

... and I strongly recommend this latter option :)

Boris,

you can see the above conf file, there is no conf setting for monpid and by default the name of the cluster in rhc is 'ceph', so there is a bug in the default behaviour of the pid creation, no where else in the test the pid file location is changed.

Vasu,

I noticed that there is no mention of .pid file in the config but (one of) the tests must have told ceph to use a non-default pid location. It could be passed as a runtime value (--pid-file <path>), it could have been modified on fly, ...

I've set up a ceph cluster on hammer to be sure of that (I've previously played with this on infernalis) and the pid file is created in a valid location by default:

# ceph daemon mon.node1 config show | grep pid_file
    "pid_file": "\/var\/run\/ceph\/mon.node1.pid",

I have no mention of pid file in ceph.conf either and I did not change any default value, there. The same happened when I ran the osd daemon which created its pid file in /var/run/ceph/ just as well.

have you used all the versions same as one available downstream, ceph-deploy, ceph versions?

Mostly, I've used the downstream ceph-deploy and the current hammer upstream packages (for convenience). These are mostly the same packages as the downstream ones with few patches going either way. None of them should touch the default location for .pid files, though.

FYI: You can see the following lines in ceph_deploy/hosts/common.py that we ship downstream:

    # start the mon using the address
    pid_location = "/var/run/ceph/mon.%s.pid" % hostname
    remoto.process.run(
        distro.conn,
        [
            'ceph-mon',
            '-i',
            hostname,
            '--pid-file', pid_location,
            '--public-addr',
            args.address,
        ],
    )

I am confused as well, I still see few denails, going to look more closely on a live system

SELinux denials found on ubuntu.redhat.com: ['type=AVC msg=audit(1455259804.125:15157): avc: denied { create } for pid=9170 comm="ceph-osd" name="ceph-osd.3.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259847.600:15201): avc: denied { create } for pid=10971 comm="ceph-osd" name="ceph-osd.5.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259713.817:15043): avc: denied { create } for pid=7580 comm="ceph-mon" name="ceph-mon.clara002.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259825.773:15179): avc: denied { create } for pid=10047 comm="ceph-osd" name="ceph-osd.4.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455260219.597:15325): avc: denied { unlink } for pid=7580 comm="ceph-mon" name="ceph-mon.clara002.asok" dev="tmpfs" ino=370607 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455260219.601:15326): avc: denied { read } for pid=7580 comm="ceph-mon" name="mon.clara002.pid" dev="tmpfs" ino=370604 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file', 'type=AVC msg=audit(1455260219.601:15327): avc: denied { unlink } for pid=7580 comm="ceph-mon" name="mon.clara002.pid" dev="tmpfs" ino=370604 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file']

All of those point to a mis-configuration. The socket files (.asok) are labelled with var_run_t which means that they are located in /var/run/ instead of /var/run/ceph where they would get labelled properly. Again, the daemons must have been configured to start in that non-default way. The same goes for the pid file. We will be documenting where to store the files (or what to do if you want to store them elsewhere) in bz1305421.