Bug 1251903

So the apache team would prefer we removed or modified the packaged logrotate script (since the current one will still get run automatically) rather than use a less systemd specific method of restarting httpd?

It is hard to comment on what is the "right" way to solve the problem without understanding the requirements or why you're running httpd outside of systemd's control to start with. 

Is the httpd daemon expected to be the only httpd daemon running on the system?  Or is it expected that users can run httpd.service independent of the httpd which pacemaker controls?

From the ticket it looks like you are logging to the standard log directory, configured httpd via /etc/httpd/conf/httpd.conf, and have a pid file at /var/run/httpd.pid.  Why is the service not managed via systemd?

Ideally I would guess you should override the httpd.service in /etc/systemd/system with a systemd unit file which can interact with the httpd daemon run by pacemaker.

(In reply to Joe Orton from comment #5)
> It is hard to comment on what is the "right" way to solve the problem
> without understanding the requirements or why you're running httpd outside
> of systemd's control to start with. 

So this is a cluster environment, systemd doesn't have the wider context to know if apache should be running on a given node.

Additionally, as you suspected below, there can be multiple httpd daemons with different personalities.  Each of these is a separate cluster resource with its own config file, of which it it is possible none are using the main system configuration that httpd.service uses.

> 
> Is the httpd daemon expected to be the only httpd daemon running on the
> system?  Or is it expected that users can run httpd.service independent of
> the httpd which pacemaker controls?

The latter.

> 
> From the ticket it looks like you are logging to the standard log directory,
> configured httpd via /etc/httpd/conf/httpd.conf, and have a pid file at
> /var/run/httpd.pid.  Why is the service not managed via systemd?

I'm not the author of this particular agent ("agent" is roughly analogous to an init script or unit file), but in general we try not to introduce non-standard locations just because something is running in a clustered context.

So it would be "normal" for agents to use the same default locations for logging, config and pid files.

> Ideally I would guess you should override the httpd.service in
> /etc/systemd/system with a systemd unit file which can interact with the
> httpd daemon run by pacemaker.

That is something we are investigating for other services when there can only be one copy of it on a node.  

However its not completely clear that this would solve all use cases for apache.  For example there is nothing that prevents an admin from wanting to start httpd.service /and/ a custom httpd.

Thanks for the explanation, Andrew.

I would say that if you are modifying /etc/httpd/conf/httpd.conf and running httpd using that config file, it is implicit that you are not expecting httpd.service to be usable outside of pacemaker on the same system, at least by default.

(That does not preclude users starting httpd with some other config, but the RHEL-supplied httpd.service will start an httpd using /etc/httpd/conf/httpd.conf and hence will want to bind to the same ports etc as the pacemaker-managed httpd).

If it is also not possible for you to manage httpd via systemd, it probably does make sense for you to either adjust or replace /etc/logrotate.d/httpd with a logrotate script which works appropriately for the pacemaker-managed httpd.  Making a copy of that logrotate script under a different name with the "postrotate" line suggested as in comment 1 would work, for example.

If you really don't want to adjust logrotate settings, or simply to get better systemd integration, it is possible to intercept the reload of httpd.service from using a separate unit file which has "ReloadPropagatedFrom=httpd.service" in [Unit].  Running "systemctl reload httpd.service" (as logrotate does) will reload such a service even if the "real" httpd.service is not running.

Joe,

We are considering registering pacemaker as a systemd scope, and putting the services managed by pacemaker underneath that scope. Does that sound like a clean way to represent a cluster manager within systemd? If so, would that allow "systemctl reload httpd.service" to work?

I'm not familiar with systemd scopes.  From a brief read through the docs, it doesn't seem obvious how it would solve the problem described here, you still either need some way to propagate the reload of httpd.service to the running httpd, or to add an additional logrotate config file specific to the pacemaker httpd.

For completeness, here are a couple of other potential workarounds:

* If the cluster resource is configured with the systemd:httpd resource agent instead of ocf:heartbeat:apache, then systemd will see the service as active and reload it properly. (In this case, only one apache instance may be run, using the default locations, and the extra features of the OCF agent such as enhanced monitoring can't be used.)

* Apache can be configured to log to syslog instead of a file, in which case the logrotate script can be removed altogether. (Back in my sysadmin days, I liked this approach because system logs can be sent in real time to a remote server, and because a busy webserver doesn't need to be reloaded just to rotate the logs.)

I don't think we can fix this in either pacemaker or the ocf:heartbeat:apache resource agent, at least not with the current apache and systemd interfaces.

Apache logging is highly configurable, and the system administrator may have configured a custom logrotate script. 

We could replace the logrotate script only if it is unchanged from the package default, but that would require us to keep track of the package default, and anyone with custom log rotation would still need to know about this issue and implement a workaround.

We could search and replace systemctl in the logrotate script, but this seems unreasonably intrusive and potentially error-prone.

And if someone has the logrotate script in a configuration management system, they will likely overwrite our changes anyway.

I don't see any way to integrate with systemd to make "systemctl reload httpd.service" work. From systemd's point of view, the service is inactive. ReloadPropagatedFrom does not intercept a reload, but duplicate it for a second service; the first service must still be active, and still gets reloaded. Overriding ExecReload for httpd also has the same issue, that httpd must be active from systemd's point of view. From what I can tell, putting httpd in a pacemaker scope would not make "systemctl reload httpd.service" work because httpd would not be a service.

Unless anyone has a better idea, I'll reassign this as a documentation bug, so we can document the issue and possible workarounds.

(In reply to Ken Gaillot from comment #12)
> Unless anyone has a better idea, I'll reassign this as a documentation bug,
> so we can document the issue and possible workarounds.

Agreed. There is very little we can do, and even less we /should/ do (automatically editing packaged files is asking for trouble).

Given that /etc/logrotate.d/httpd has exclusively embraced systemd, the best we can do is document the admin's options - only they know the complete set of ways they'll be using apache.

Reassigning to docs.

The issue here needs to be documented for anyone using the ocf:heartbeat:apache resource agent. When using that agent to manage apache in a Pacemaker cluster, the logrotate script supplied with apache will fail, because it tries to use "systemctl" to reload apache, which fails because apache is not being managed by systemd.

If only one instance of apache is in use, using the default configuration location, then the issue can be resolved by editing /etc/logrotate.d/httpd and changing the line:

/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

to:

/usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd.pid" -k graceful > /dev/null 2>/dev/null || true

If multiple instances of apache and/or non-default configuration locations are used, the line above can be modified to use one line per apache instance, with the non-default configuration file location and pid file location specified appropriately.

An alternative solution would be to configure apache to log via syslog, in which case the /etc/logrotate.d/httpd file should be removed. (I don't know whether our apache docs already describe such a configuration, but it's documented upstream.)

Ken:  There should be -- but this is the link I pointed to once before (when there were onoly 3 steps) so it might be cached. Try a reload and see if it works -- it definitely calls up a four-step process for me.

Steven

Oops, of course, that was it.

Yes, it looks perfect.

Have just run across this problem and am wondering what it is in comment 22 that "looks perfect" since comments 15 - 21 are invisible without a red hat on. My own immediate thought was that logrotate was just wrong and should be doing the old-style kill -USR1 $(cat /run/httpd.pid)

(In reply to Trevor Hemsley from comment #26)
> Have just run across this problem and am wondering what it is in comment 22
> that "looks perfect" since comments 15 - 21 are invisible without a red hat
> on. My own immediate thought was that logrotate was just wrong and should be
> doing the old-style kill -USR1 $(cat /run/httpd.pid)

Sorry, it was a preliminary draft of what is now published:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Administration/s1-webserversetup-HAAA.html

Specifically, this:

   When you use the apache resource agent to manage Apache, it does not use systemd. Because of this, you must edit the logrotate script supplied with Apache so that it does not use systemctl to reload Apache.

   Remove the following line in the /etc/logrotate.d/httpd file.

   /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

   Replace the line you removed with the following line.

   /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd.pid" -k graceful > /dev/null 2>/dev/null || true

(Most private comments here involve internal links that aren't accessible from outside, and so would be useless. In this case, it was discussion of the pre-publication draft.)