Description of problem: haproxy log should not be written in /var/log/messages In THT, we have following sections. ~~~ 258 host_prep_tasks: 259 - {get_attr: [HAProxyBase, role_data, host_prep_tasks]} 260 - name: create persistent directories 261 file: 262 path: "{{ item }}" 263 state: directory 264 with_items: 265 - /var/log/containers/haproxy 266 - /var/lib/haproxy 267 - name: haproxy logs readme 268 copy: 269 dest: /var/log/haproxy/readme.txt 270 content: | 271 Log files from the haproxy containers can be found under 272 /var/log/containers/haproxy. 273 ignore_errors: true ~~~ But actual result is that haproxy's log messages are written to /var/log/messages of controller nodes. Version-Release number of selected component (if applicable): Current openstack-tripleo-heat-templates How reproducible: Everytime of deployment. Steps to Reproduce: 1. Deploy overcloud according to Director Installation and Usage guide. 2. Check the directory and messages of haproxy. 3. Actual results: haproxy's log is written to /var/log/messages. Expected results: The log should be written under /var/log/containers/haproxy directory Additional info:
Hello Keigo-san, Sorry for the late answer... After looking at it more thoroughly, it's not clear what the intent of this bz is. You are referring to the non-HA version of the HAproxy service, which is not the one which is used for deploying overcloud. The file that is in charge of configuring HAProxy is t-h-t/docker/service/pacemaker/haproxy.yaml, and in this heat template we don't mention any haproxy log file explicitely. In fact, HAproxy has never log into log files in RHOSP, even before OSP12 and containers. It only logs in the journal. The reason for that is that unless other services, HAproxy natively lacks (on purpose) the ability to log into a file. It will essentially only log messages to a syslog with a specific facility type (e.g. local0), and the syslog may then be configured according to the needs. I had some experiment in order to force such messages to be written in a file and not in /var/log/messages, but this has at least two drawbacks: 1. the configuration for the HAproxy container would depends on rsyslog running on the host, to write to this new log file and rotate it. 2. even if we can prevent logging haproxy messages in /var/log/messages, those messages will still log into systemd's journal. That is because the rsyslog socket that HAproxy is configured to log into is created and consumed by systemd first, before being forwarded to the rsyslog daemon running on the host. So I have the feeling that this bz is there to point a discrepancy between what is stated in t-h-t and what is actually done, which is never write to a dedicated file for HAproxy (even for the non-HA case). Can you confirm that this was the intent of this bz? If so, I would suggest to fix the t-h-t template upstream and close that bz accordingly.
Hello Damien, Thank you for your work on this bugzilla. Is it possible to include a fix for this issue in RHOSP13z3? Best Regards, Keigo Noha
Hello Damien, Do you have any updates on this bugzilla? Best Regards, Keigo Noha
(In reply to Keigo Noha from comment #7) > Hello Damien, > > Do you have any updates on this bugzilla? > 1) So currently I've come up with a simple drop-in file in host's rsyslog config to create a haproxy file in /var/log/containers/haproxy.log, but I still need to validate that log rotation works as expected. 2) I have yet tried a longer term logging strategy that would not use the host's rsyslog and replace it with dedicated side-car containers for logging, so I can't report on that yet. I'll update shortly on 1), sorry for the delay > > Best Regards, > Keigo Noha
Hello Damien, Thank you for your work on this bugzilla. Do you have any updates related to plan 1? If we have a launchpad or upstream gerrit, please add them to this bugzilla. Best Regards, Keigo Noha
(In reply to Keigo Noha from comment #9) > Hello Damien, > > Thank you for your work on this bugzilla. > Do you have any updates related to plan 1? > If we have a launchpad or upstream gerrit, please add them to this bugzilla. > Hello Keigo So the current plan for plan 1 is more something along the lines of a KBase. The idea that I am experimenting with is: . create an additional rsyslog config file /etc/rsyslog.d/haproxy.conf on every controller. I am attaching the content of that file in the bz. . the rsyslog config diverts all haproxy logs from /var/log/messages and store them under /var/log/containers/haproxy/haproxy.log So far I've verified that haproxy logs don't show up anymore in /var/log/messages, and that they are correctly rotated in /var/log/containers/haproxy (you need [1] to fix a current bug in container log rotation). I still need to verify that logs from ha routers is not diverted into that file. Once I'll have that verified I'll propose a KBase that explain how setup haproxy file logging. Note that this doesn't preclude working on plan #2, it's only a longer term plan which has implication of container logging more generally. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1585516. > Best Regards, > Keigo Noha
Created attachment 1502696 [details] content of drop-in file /etc/rsyslog.d/haproxy.conf
Hello Damien, I'm sorry for the late. I modified the solution with your configuration, https://access.redhat.com/solutions/3662761. Thank you for sharing the configuration! Best Regards, Keigo Noha
Hello Damien-san, Thank you for your work on this bugzilla. A customer asks us which directory is preferred, /var/log/ or /var/log/containers/haproxy in the workaround. In your suggested workaround, we use /var/log/containers/haproxy. We will implement side-car container for collecting logs. And RHOSP13 locates all logs from containerized service under /var/log/containers/<service> directory. So, I think there is no difference from the point of functionality. But /var/log/containers/haproxy is desired because we should follow the way in RHOSP13. Do you have any concerns to my understandings? Best Regards, Keigo Noha
Hello Keigo-san, So yes there is no difference from a technical perspective. I would agree with you that /var/log/containers/haproxy makes more sense since we try to gather logs of all containerized services under the same location.
Hello Damien-san, Thank you for your quick response! I'll share this information with them. We'll wait for the implementation of side-car container. Best Regards, Keigo Noha
Also, worth noting is that work upstream is ongoing [1] to propose a similar fix by default for new deployments. [1] https://review.openstack.org/#/c/620601/
Hello Damien, Thank you for letting me know the upstream gerrit. The status of the gerrit is merged. Can we bakcport it into RHOSP13? Can we include it into z4 release? Best Regards, Keigo Noha
Hello, Small update: the following patch on openstack-selinux will probably be needed: https://github.com/redhat-openstack/openstack-selinux/pull/20 in addition of the rsyslog config part. @Damien: may I let you manage that, and see if backporting the t-h-t is enough? Cheers, C.
I've just created https://bugzilla.redhat.com/show_bug.cgi?id=1669146 to track the dependency on SELinux.
Instruction for testing the fix: 1. Deploy an HA overcloud 2. verify that HAProxy logs are now stored into a new dedicated file /var/log/containers/haproxy/haproxy.log 3. verify that there's no SELinux denials due to this new file access Notes: . HAProxy logs should still be stored in the journal, as systemd forwards its logs to rsyslog, which in turns stores haproxy entries into the new haproxy log file . HA routers (that spawn an HAProxy) won't store any log under /var/log/containers/hapoxy/haproxy.log
Verified, (undercloud) [stack@undercloud-0 ~]$ ansible controller -b -mshell -a'rpm -qa |grep puppet-tripleo' controller-0 | SUCCESS | rc=0 >> puppet-tripleo-8.3.6-15.el7ost.noarch controller-2 | SUCCESS | rc=0 >> puppet-tripleo-8.3.6-15.el7ost.noarch controller-1 | SUCCESS | rc=0 >> puppet-tripleo-8.3.6-15.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ ansible controller -b -mshell -a'head /var/log/containers/haproxy/haproxy.log' [WARNING]: Found both group and host with same name: undercloud controller-2 | SUCCESS | rc=0 >> Feb 28 09:08:49 controller-2 haproxy[11]: Proxy aodh started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy cinder started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy glance_api started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy gnocchi started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy haproxy.stats started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy heat_api started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy heat_cfn started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy horizon started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy keystone_admin started. Feb 28 09:08:49 controller-2 haproxy[11]: Proxy keystone_public started. controller-1 | SUCCESS | rc=0 >> Feb 28 09:08:49 controller-1 haproxy[11]: Proxy aodh started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy cinder started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy glance_api started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy gnocchi started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy haproxy.stats started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy heat_api started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy heat_cfn started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy horizon started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy keystone_admin started. Feb 28 09:08:49 controller-1 haproxy[11]: Proxy keystone_public started. controller-0 | SUCCESS | rc=0 >> Feb 28 09:08:49 controller-0 haproxy[11]: Proxy aodh started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy cinder started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy glance_api started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy gnocchi started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy haproxy.stats started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy heat_api started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy heat_cfn started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy horizon started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy keystone_admin started. Feb 28 09:08:49 controller-0 haproxy[11]: Proxy keystone_public started. (undercloud) [stack@undercloud-0 ~]$ ansible controller -b -mshell -a'grep haproxy /var/log/secure' [WARNING]: Found both group and host with same name: undercloud controller-2 | FAILED | rc=1 >> non-zero return code controller-0 | FAILED | rc=1 >> non-zero return code controller-1 | FAILED | rc=1 >> non-zero return code
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0448