Bug 1251903 - Include script for rotating logs when apache is managed via pacemaker
Include script for rotating logs when apache is managed via pacemaker
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: doc-High_Availability_Add-On_Reference (Show other bugs)
7.2
Unspecified Unspecified
high Severity medium
: rc
: 7.3
Assigned To: Steven J. Levine
ecs-bugs
: Documentation
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-10 05:22 EDT by Eduard Barrera
Modified: 2017-01-31 12:17 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-07 16:02:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eduard Barrera 2015-08-10 05:22:49 EDT
Description of problem:
When httpd is started by pcs resource script logrotate script is not able to rotate the logs.

httpd package provides /etc/logrotate.d/httpd

and it is the content:

And the content is the following:
~~~
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
       /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
    endscript
}

# /bin/systemctl reload httpd.service

Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.


# systemctl status httpd.service

httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: inactive (dead)

jun 17 12:11:10 localhost systemd[1]: Unit httpd.service cannot be reloaded because it is inactive.

It was workaround by changing /etc/logrotate.d/httpd with the following line:

 # /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd.pid" -k graceful > /dev/null 2>/dev/null || true

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. Start httpd as a pcs resource
2.
3.

Actual results:
httpd logs not rotated when logrotate is triggered

Expected results:
httpd logs rotated even it is started as a pcs resource
Comment 4 Andrew Beekhof 2015-08-12 02:02:33 EDT
So the apache team would prefer we removed or modified the packaged logrotate script (since the current one will still get run automatically) rather than use a less systemd specific method of restarting httpd?
Comment 5 Joe Orton 2015-08-12 05:40:27 EDT
It is hard to comment on what is the "right" way to solve the problem without understanding the requirements or why you're running httpd outside of systemd's control to start with. 

Is the httpd daemon expected to be the only httpd daemon running on the system?  Or is it expected that users can run httpd.service independent of the httpd which pacemaker controls?

From the ticket it looks like you are logging to the standard log directory, configured httpd via /etc/httpd/conf/httpd.conf, and have a pid file at /var/run/httpd.pid.  Why is the service not managed via systemd?

Ideally I would guess you should override the httpd.service in /etc/systemd/system with a systemd unit file which can interact with the httpd daemon run by pacemaker.
Comment 6 Andrew Beekhof 2015-08-13 02:38:43 EDT
(In reply to Joe Orton from comment #5)
> It is hard to comment on what is the "right" way to solve the problem
> without understanding the requirements or why you're running httpd outside
> of systemd's control to start with. 

So this is a cluster environment, systemd doesn't have the wider context to know if apache should be running on a given node.

Additionally, as you suspected below, there can be multiple httpd daemons with different personalities.  Each of these is a separate cluster resource with its own config file, of which it it is possible none are using the main system configuration that httpd.service uses.

> 
> Is the httpd daemon expected to be the only httpd daemon running on the
> system?  Or is it expected that users can run httpd.service independent of
> the httpd which pacemaker controls?

The latter.

> 
> From the ticket it looks like you are logging to the standard log directory,
> configured httpd via /etc/httpd/conf/httpd.conf, and have a pid file at
> /var/run/httpd.pid.  Why is the service not managed via systemd?

I'm not the author of this particular agent ("agent" is roughly analogous to an init script or unit file), but in general we try not to introduce non-standard locations just because something is running in a clustered context.

So it would be "normal" for agents to use the same default locations for logging, config and pid files.

> Ideally I would guess you should override the httpd.service in
> /etc/systemd/system with a systemd unit file which can interact with the
> httpd daemon run by pacemaker.

That is something we are investigating for other services when there can only be one copy of it on a node.  

However its not completely clear that this would solve all use cases for apache.  For example there is nothing that prevents an admin from wanting to start httpd.service /and/ a custom httpd.
Comment 7 Joe Orton 2015-08-13 07:26:06 EDT
Thanks for the explanation, Andrew.

I would say that if you are modifying /etc/httpd/conf/httpd.conf and running httpd using that config file, it is implicit that you are not expecting httpd.service to be usable outside of pacemaker on the same system, at least by default.

(That does not preclude users starting httpd with some other config, but the RHEL-supplied httpd.service will start an httpd using /etc/httpd/conf/httpd.conf and hence will want to bind to the same ports etc as the pacemaker-managed httpd).

If it is also not possible for you to manage httpd via systemd, it probably does make sense for you to either adjust or replace /etc/logrotate.d/httpd with a logrotate script which works appropriately for the pacemaker-managed httpd.  Making a copy of that logrotate script under a different name with the "postrotate" line suggested as in comment 1 would work, for example.

If you really don't want to adjust logrotate settings, or simply to get better systemd integration, it is possible to intercept the reload of httpd.service from using a separate unit file which has "ReloadPropagatedFrom=httpd.service" in [Unit].  Running "systemctl reload httpd.service" (as logrotate does) will reload such a service even if the "real" httpd.service is not running.
Comment 8 Ken Gaillot 2016-01-19 18:04:56 EST
Joe,

We are considering registering pacemaker as a systemd scope, and putting the services managed by pacemaker underneath that scope. Does that sound like a clean way to represent a cluster manager within systemd? If so, would that allow "systemctl reload httpd.service" to work?
Comment 9 Joe Orton 2016-01-20 04:04:22 EST
I'm not familiar with systemd scopes.  From a brief read through the docs, it doesn't seem obvious how it would solve the problem described here, you still either need some way to propagate the reload of httpd.service to the running httpd, or to add an additional logrotate config file specific to the pacemaker httpd.
Comment 10 Ken Gaillot 2016-01-20 18:05:56 EST
For completeness, here are a couple of other potential workarounds:

* If the cluster resource is configured with the systemd:httpd resource agent instead of ocf:heartbeat:apache, then systemd will see the service as active and reload it properly. (In this case, only one apache instance may be run, using the default locations, and the extra features of the OCF agent such as enhanced monitoring can't be used.)

* Apache can be configured to log to syslog instead of a file, in which case the logrotate script can be removed altogether. (Back in my sysadmin days, I liked this approach because system logs can be sent in real time to a remote server, and because a busy webserver doesn't need to be reloaded just to rotate the logs.)
Comment 11 Ken Gaillot 2016-01-20 18:08:08 EST
I don't think we can fix this in either pacemaker or the ocf:heartbeat:apache resource agent, at least not with the current apache and systemd interfaces.

Apache logging is highly configurable, and the system administrator may have configured a custom logrotate script. 

We could replace the logrotate script only if it is unchanged from the package default, but that would require us to keep track of the package default, and anyone with custom log rotation would still need to know about this issue and implement a workaround.

We could search and replace systemctl in the logrotate script, but this seems unreasonably intrusive and potentially error-prone.

And if someone has the logrotate script in a configuration management system, they will likely overwrite our changes anyway.

I don't see any way to integrate with systemd to make "systemctl reload httpd.service" work. From systemd's point of view, the service is inactive. ReloadPropagatedFrom does not intercept a reload, but duplicate it for a second service; the first service must still be active, and still gets reloaded. Overriding ExecReload for httpd also has the same issue, that httpd must be active from systemd's point of view. From what I can tell, putting httpd in a pacemaker scope would not make "systemctl reload httpd.service" work because httpd would not be a service.
Comment 12 Ken Gaillot 2016-01-20 18:23:09 EST
Unless anyone has a better idea, I'll reassign this as a documentation bug, so we can document the issue and possible workarounds.
Comment 13 Andrew Beekhof 2016-02-16 21:13:26 EST
(In reply to Ken Gaillot from comment #12)
> Unless anyone has a better idea, I'll reassign this as a documentation bug,
> so we can document the issue and possible workarounds.

Agreed. There is very little we can do, and even less we /should/ do (automatically editing packaged files is asking for trouble).

Given that /etc/logrotate.d/httpd has exclusively embraced systemd, the best we can do is document the admin's options - only they know the complete set of ways they'll be using apache.
Comment 14 Ken Gaillot 2016-02-17 16:38:27 EST
Reassigning to docs.

The issue here needs to be documented for anyone using the ocf:heartbeat:apache resource agent. When using that agent to manage apache in a Pacemaker cluster, the logrotate script supplied with apache will fail, because it tries to use "systemctl" to reload apache, which fails because apache is not being managed by systemd.

If only one instance of apache is in use, using the default configuration location, then the issue can be resolved by editing /etc/logrotate.d/httpd and changing the line:

/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

to:

/usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd.pid" -k graceful > /dev/null 2>/dev/null || true

If multiple instances of apache and/or non-default configuration locations are used, the line above can be modified to use one line per apache instance, with the non-default configuration file location and pid file location specified appropriately.

An alternative solution would be to configure apache to log via syslog, in which case the /etc/logrotate.d/httpd file should be removed. (I don't know whether our apache docs already describe such a configuration, but it's documented upstream.)
Comment 22 Steven J. Levine 2016-06-10 10:29:49 EDT
Ken:  There should be -- but this is the link I pointed to once before (when there were onoly 3 steps) so it might be cached. Try a reload and see if it works -- it definitely calls up a four-step process for me.

Steven
Comment 23 Ken Gaillot 2016-06-10 11:20:37 EDT
Oops, of course, that was it.

Yes, it looks perfect.
Comment 26 Trevor Hemsley 2017-01-31 11:56:48 EST
Have just run across this problem and am wondering what it is in comment 22 that "looks perfect" since comments 15 - 21 are invisible without a red hat on. My own immediate thought was that logrotate was just wrong and should be doing the old-style kill -USR1 $(cat /run/httpd.pid)
Comment 27 Ken Gaillot 2017-01-31 12:17:50 EST
(In reply to Trevor Hemsley from comment #26)
> Have just run across this problem and am wondering what it is in comment 22
> that "looks perfect" since comments 15 - 21 are invisible without a red hat
> on. My own immediate thought was that logrotate was just wrong and should be
> doing the old-style kill -USR1 $(cat /run/httpd.pid)

Sorry, it was a preliminary draft of what is now published:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Administration/s1-webserversetup-HAAA.html

Specifically, this:

   When you use the apache resource agent to manage Apache, it does not use systemd. Because of this, you must edit the logrotate script supplied with Apache so that it does not use systemctl to reload Apache.

   Remove the following line in the /etc/logrotate.d/httpd file.

   /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

   Replace the line you removed with the following line.

   /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd.pid" -k graceful > /dev/null 2>/dev/null || true

(Most private comments here involve internal links that aren't accessible from outside, and so would be useless. In this case, it was discussion of the pre-publication draft.)

Note You need to log in before you can comment on or make changes to this bug.