Bug 963332

Summary:	Fix mcollective-client.log permissions
Product:	OpenShift Container Platform	Reporter:	Jeremy Eder <jeder>
Component:	Node	Assignee:	John W. Lamb <jolamb>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	1.2.0	CC:	baulakh, bleanhar, erich, gpei, jeder, jialiu, jkeck, jolamb, libra-onpremise-devel, lmeyer, mwalker, pbrobinson, tiwillia
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Whenever mcollective-client.log is created, whether initially or due to log file rolling, it is created with the ownership of the current user. Consequence: When a root user invokes mcollective (either directly or via broker tools), there is a chance mcollective-client.log will be owned by root, preventing the broker (apache user) from writing to the log file and causing it to log errors in the httpd error_log. Fix: The openshift.sh/ks install script now configures the mco client to log to the console at log level warn. Deploy guide should be updated to do the same. Result: mco client calls on the broker now log to the console instead of directly to the client log file. When used from the broker application, this means warnings and errors will be directed to the application error log in /var/log/openshift/broker/httpd/error_log. When used via broker scripts (e.g. oo-admin-ctl-district) these would actually print out on the console. At log_level warn, though, this should only happen when there is a problem contacting the message server. If admins lower the log level to info or debug, they can expect these scripts to have some fairly verbose and confusing output, so this is discouraged.	Story Points:	---
Clone Of:
Clones:	1076525 (view as bug list)		Environment:
Last Closed:	2015-09-23 16:00:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1076525, 1077135

Description Jeremy Eder 2013-05-15 16:44:23 UTC

Related to:  https://bugzilla.redhat.com/show_bug.cgi?id=880385
and:  https://bugzilla.redhat.com/show_bug.cgi?id=849558

I don't believe a doc fix is the right way to go, I think we need to fix mcollective:

- add ownership capabilities to client.cfg
- allow mcollective to rely on logrotate.

See this:
http://markmail.org/message/5fpqyb5ncljjvthz

When mcollective rolls it's log file, the new file has the wrong ownership.  mcollective controls log rolling internally (doesn't rely on logrotate).

Not sure we can use logrotate to fix up permissions.

Notes: apparently mcollective has it's own file size limit control [1].  But it doesn't seem to be able to control ownership.

Our client.cfg doesn't mention size, so it defaults to 2MB (which is what happened on my system; it got rolled at 2MB, and the new log file was owned root:root).

[1] http://docs.puppetlabs.com/mcollective/reference/basic/configuration.html

Comment 2 Luke Meyer 2013-05-15 20:01:08 UTC

mcollective is a generalized tool from puppet. there's no reason for it to be aware that it needs to deal with conflicts in ownership of its log file.

perhaps, though, we could have the mcollective plugin on the broker modify the log file in its %post. or, just own the log file (perhaps have it use something other than the default).

Comment 3 Jeremy Eder 2013-05-15 20:03:53 UTC

if it's done in rpm post, what happens when the log file gets rolled ?  thats the actual issue. the new log file created after rolling gets root:root

Comment 4 Luke Meyer 2013-05-16 03:00:55 UTC

Actually, I would expect the log file after rolling to be owned by whoever was using it, which should be apache usually; but you have a point, when the mcollective client is being used by root, filling the log so it rolls would change the log file ownership. 

The original problem we were having was at install time - the first test would be root running "mco ping" and the log file created would belong to root. You're talking about something that is not a one-time fix.

I don't think there's a good way to have mcollective change ownership when it's writing that log file. The only thing I can think of is forcing all scripts that hook into the broker code to operate as the apache user when doing so. Yikes.

Comment 5 Brenton Leanhardt 2013-05-16 13:12:55 UTC

We have a few options:

* Monkey-patch things so our cli tools never write to mcollective-client.log
* Use mcollective's syslog logger for the client

I'm leaning towards syslog.  We might even be able to drop a file in /etc/rsyslog.d to keep it logging in /var/log/mcollective-client.log.

Comment 6 Brenton Leanhardt 2013-05-16 13:31:23 UTC

I tested by created a file called /etc/rsyslog.d/mcollective-client.conf with the following content:

user.*        /var/log/mcollective-client2.log

I then updated /etc/mcollective/client.conf with these settings:
logger_type = syslog
logfacility = user

The "user" facility is technically the default.  Unfortunately it doesn't appear that we can use custom facilities.  It seems "user" or "local0-7" are used for this sort of thing.

After touching /var/log/mcollective-client2.log as root both mco and the broker could write to that file with selinux enforcing.

Another interesting thing is that it picked up some gear logs like:

python: DEBUG: var entry /var/lib/openshift/5194dd726892df5361000001/php/env/OPENSHIFT_PHP_IDENT

Apparently a python script is called when creating php apps?

Whatever we do if we end up using syslog we'd likely have to also drop a file in /etc/logrotate.d.

Comment 7 Luke Meyer 2013-06-28 17:32:05 UTC

(In reply to Brenton Leanhardt from comment #6)
> Another interesting thing is that it picked up some gear logs like:
> 
> python: DEBUG: var entry
> /var/lib/openshift/5194dd726892df5361000001/php/env/OPENSHIFT_PHP_IDENT
> 
> Apparently a python script is called when creating php apps?

I think that might actually be part of the mcollective response getting dumped to DEBUG. Not sure.

Comment 10 Luke Meyer 2014-03-04 16:07:59 UTC

One other reasonable possibility for resolving this: don't log to a file.
http://docs.puppetlabs.com/mcollective/configure/client.html#logging

MCo allows logging to console or syslog. If logged to console, it goes to the console (from command line) or to broker httpd error_log. If logged to syslog then it ends up in /var/log/messages unless filtered elsewhere.

I would vote for console, with a log level of warn. There's rarely anything useful in this log anyway. Easy enough to doc, change the install script, make a kbase for the change. Possibly change automatically on upgrade to 2.1?

Comment 11 Brenton Leanhardt 2014-03-04 16:10:20 UTC

+100.  I agree this isn't the most useful log in the world and it's frustrating that it causes so many problems with different users writing to it.

Comment 13 Brenton Leanhardt 2014-03-18 13:50:48 UTC

*** Bug 1077135 has been marked as a duplicate of this bug. ***

Comment 14 John W. Lamb 2014-03-24 17:33:20 UTC

Should be addressed with merge of https://github.com/openshift/openshift-extras/pull/319

Comment 15 Gaoyun Pei 2014-03-25 07:17:04 UTC

After installing a ose-2.1 env using the installation script, check /opt/rh/ruby193/root/etc/mcollective/client.cfg 

The log related configuration has been changed to:

logger_type = console
loglevel = warn

When running oo-mco ping, there's no log appended to ruby193-mcollective-client.log, if change loglevel = debug, re-run oo-mco ping again, debug info will be printed out on terminal, not appended to log file.

Comment 18 Miciah Dashiel Butler Masters 2015-09-23 16:00:24 UTC

This is fixed in <https://github.com/openshift/openshift-extras/blob/enterprise-2.2/enterprise/install-scripts/generic/openshift.sh> and in the openshift.sh script that is included in the current oo-install archive, <https://install.openshift.com/ose-2.2/oo-install-ose-2.2-20150825-1235.tgz>.