Bug 1305978 - [RHCeph 1.3.2] mon.pid should be in /var/run/ceph instead of /var/run
[RHCeph 1.3.2] mon.pid should be in /var/run/ceph instead of /var/run
Status: CLOSED NOTABUG
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Ceph-Installer (Show other bugs)
1.3.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 1.3.2
Assigned To: Boris Ranto
ceph-qe-bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-09 12:35 EST by Vasu Kulkarni
Modified: 2017-12-12 19:26 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-11 13:27:52 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vasu Kulkarni 2016-02-09 12:35:20 EST
Description of problem:

From bz1303799

issues noticed by borris

1.) You do put mon.clara007.pid outside the /var/run/ceph directory (it looks like it is located directly in /var/run/ directory) which we should not do as we have designated /var/run/ceph/ directory for these files. If this is the default location, we should tune the policy to cover these files. We will need a format specification for that, though -- i.e. is it anything that matches {mon,osd,mds,radosgw}.*.pid or is there something else to it?


Version-Release number of selected component (if applicable):
1.3.2 - 0.94.5-8

How reproducible:
Always


Steps to Reproduce:
run tests with selinux enabled and the pid will be in /var/run

Actual results:
should be in /var/run/ceph
Comment 2 Vasu Kulkarni 2016-02-09 18:04:25 EST
ceph.conf file from the test run

[global]
fsid = 08c6a829-69fb-4db1-b5c8-9d137048ae03
mon_initial_members = clara007, clara012
mon_host = 10.8.129.7,10.8.129.12
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
mon_pg_warn_min_per_osd = 2

[osd]
debug_ms = 1
debug_journal = 20
osd_op_thread_timeout = 60
debug_osd = 20
debug_filestore = 20
osd_sloppy_crc = True
 
[mon]
osd_default_pool_size = 2
debug_mon = 1
debug_paxos = 20
debug_ms = 20
 
[client]
log_file = /var/log/ceph/ceph-$name.$pid.log
Comment 3 Boris Ranto 2016-02-11 05:51:45 EST
OK, based on the following lines in doc/rados/configuration/general-config-ref.rst


"""
``pid file``
:Description: The file in which the mon, osd or mds will write its
              PID.  For instance, ``/var/run/$cluster/$type.$id.pid``
              will create /var/run/ceph/mon.a.pid for the ``mon`` with
              id ``a`` running in the ``ceph`` cluster. The ``pid
              file`` is removed when the daemon stops gracefully. If 
              the process is not daemonized (i.e. runs with the ``-f``
              or ``-d`` option), the ``pid file`` is not created. 
:Type: String
:Required: No 
:Default: No 
"""


the pid file location is configurable and it is not created by default. Therefore, one of the tests must have specified it to be created in /var/run/<something>.pid instead of /var/run/ceph/<something>.pid which means that we should fix the test case, not the policy.

The test case can either continue to create the pid files in that location and afterwards, chcon to the ceph_var_run_t type or can put the pid files in /var/run/ceph/ directory where the labelling will be done automatically by kernel itself.
Comment 4 Ken Dreyer (Red Hat) 2016-02-11 10:29:49 EST
(In reply to Boris Ranto from comment #3)
> or can put the pid files in
> /var/run/ceph/ directory where the labelling will be done automatically by
> kernel itself.

... and I strongly recommend this latter option :)
Comment 5 Vasu Kulkarni 2016-02-11 11:49:02 EST
Boris,

you can see the above conf file, there is no conf setting for monpid and by default the name of the cluster in rhc is 'ceph', so there is a bug in the default behaviour of the pid creation, no where else in the test the pid file location is changed.
Comment 6 Boris Ranto 2016-02-11 13:27:52 EST
Vasu,

I noticed that there is no mention of .pid file in the config but (one of) the tests must have told ceph to use a non-default pid location. It could be passed as a runtime value (--pid-file <path>), it could have been modified on fly, ...

I've set up a ceph cluster on hammer to be sure of that (I've previously played with this on infernalis) and the pid file is created in a valid location by default:

# ceph daemon mon.node1 config show | grep pid_file
    "pid_file": "\/var\/run\/ceph\/mon.node1.pid",

I have no mention of pid file in ceph.conf either and I did not change any default value, there. The same happened when I ran the osd daemon which created its pid file in /var/run/ceph/ just as well.
Comment 7 Vasu Kulkarni 2016-02-11 13:31:05 EST
have you used all the versions same as one available downstream, ceph-deploy, ceph versions?
Comment 8 Boris Ranto 2016-02-11 14:03:37 EST
Mostly, I've used the downstream ceph-deploy and the current hammer upstream packages (for convenience). These are mostly the same packages as the downstream ones with few patches going either way. None of them should touch the default location for .pid files, though.

FYI: You can see the following lines in ceph_deploy/hosts/common.py that we ship downstream:

    # start the mon using the address
    pid_location = "/var/run/ceph/mon.%s.pid" % hostname
    remoto.process.run(
        distro.conn,
        [
            'ceph-mon',
            '-i',
            hostname,
            '--pid-file', pid_location,
            '--public-addr',
            args.address,
        ],
    )
Comment 9 Vasu Kulkarni 2016-02-12 15:19:57 EST
I am confused as well, I still see few denails, going to look more closely on a live system

SELinux denials found on ubuntu@clara002.ceph.redhat.com: ['type=AVC msg=audit(1455259804.125:15157): avc: denied { create } for pid=9170 comm="ceph-osd" name="ceph-osd.3.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259847.600:15201): avc: denied { create } for pid=10971 comm="ceph-osd" name="ceph-osd.5.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259713.817:15043): avc: denied { create } for pid=7580 comm="ceph-mon" name="ceph-mon.clara002.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455259825.773:15179): avc: denied { create } for pid=10047 comm="ceph-osd" name="ceph-osd.4.asok" scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455260219.597:15325): avc: denied { unlink } for pid=7580 comm="ceph-mon" name="ceph-mon.clara002.asok" dev="tmpfs" ino=370607 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file', 'type=AVC msg=audit(1455260219.601:15326): avc: denied { read } for pid=7580 comm="ceph-mon" name="mon.clara002.pid" dev="tmpfs" ino=370604 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file', 'type=AVC msg=audit(1455260219.601:15327): avc: denied { unlink } for pid=7580 comm="ceph-mon" name="mon.clara002.pid" dev="tmpfs" ino=370604 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file']
Comment 10 Boris Ranto 2016-02-15 18:16:49 EST
All of those point to a mis-configuration. The socket files (.asok) are labelled with var_run_t which means that they are located in /var/run/ instead of /var/run/ceph where they would get labelled properly. Again, the daemons must have been configured to start in that non-default way. The same goes for the pid file. We will be documenting where to store the files (or what to do if you want to store them elsewhere) in bz1305421.

Note You need to log in before you can comment on or make changes to this bug.