Bug 2026290

Any update to share? As mentioned above, my customer needs to implement forwarding of the missing logs asap due to security requirements, so they have been out of compliance since they upgraded to 16.1.

For now we just need a workaround so that the missing logs are included on var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf.

I have a tested workaround that results in the following config in /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf:

=====
[heat-admin@controller-0 ~]$ egrep -iC3 '(haproxy.log)|(/var/log/message)' /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf
)
# haproxy_openstack.haproxy
input(type="imfile"
  file="/var/log/containers/haproxy.log"
  tag="openstack.haproxy"
  startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) "
)
--
)
# rsyslog_openstack.host.messages
input(type="imfile"
  file="/var/log/messages"
  tag="openstack.host.messages"
  startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) "
)
=====


Simply add the required configuration to your environment as ExtraConfig:

=====
[root@undercloud-0 stack]# cat virt/logging-environment-rsyslog.yaml 
## A Heat environment file which can be used to set up
## logging agents
parameter_defaults:
    ExtraConfig:
        tripleo_logging_sources_rsyslog:
            - tag: openstack.host.messages
              file: /var/log/messages
        tripleo_logging_sources_haproxy:
            - tag: openstack.haproxy
              file: /var/log/containers/haproxy.log
[...]
=====

It hasn't been tested beyond this point, but it looks it should do what you need. Take note of the tag for /var/log/messages, you might want to match this to your previous pre-upgrade configuration. I think this config will get you over your compliance hurdle and should be stable in the face of minor updates. 

We can work on a proper patch for the haproxy deployment template, since it seems to be conspicuously missing this config.

I will also talk to the team about what to do with the /var/log/messages request, but my hunch is that you'd need to file it as an RFE if this ExtraData solution is not adequate for some reason.

Hope this helps! LMK how it goes.

Just a minor change on the last line from this:

              file: /var/log/containers/haproxy.log

to this:

              file: /var/log/containers/haproxy/haproxy.log

After changing that, I can see the logs forwarded to my external logging server. Will now lower priority to this BZ and wait for confirmation from customer.

Thanks again for the quick fix.

One more change. This:

              file: /var/log/messages

should be:

              file: /var/log/host/messages


Finally, the default regex for HAProxy looks like this:

  startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) "

Is it possible to tune that regex so that it looks like the one that is applied to Pacemaker?:

  startmsg.regex="^[a-zA-Z]{3} [0-9]{2} [:0-9]{8}"

Yes, I think that is possible like so:

        tripleo_logging_sources_haproxy:
            - tag: openstack.haproxy
              file: /var/log/containers/haproxy.log
              startmsg.regex="^[a-zA-Z]{3} [0-9]{2} [:0-9]{8}"

Based on the example you mention, here: https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/pacemaker/pacemaker-baremetal-puppet.yaml#L99

Whoops,  s/=/: /  in that regex line. Sorry, copy pasted too fast

The patches have merged and been backported to to stable/train upstream. The will be released in OSP 16.2.2

(In reply to Chris Sibbitt from comment #3)
> I will also talk to the team about what to do with the /var/log/messages
> request, but my hunch is that you'd need to file it as an RFE if this
> ExtraData solution is not adequate for some reason.
> 

What is the feedback regarding /var/log/messages? I can see that the commit upstream includes haproxy only. Do I need to fill separate BZs for /var/log/messages and /var/log/containers/horizon/horizon.log?

If using ExtraData is not good enough for some reason, then you will need to file a Request For Enhancement as there is currently no support for forwarding of host logs in tripleo.

Is there an additional problem with horizon logging? Alex mentioned it in the first comment, but I understood from the wording that he was comparing the missing examples to a working example.  The options that were missing for haproxy are present for horizon: https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/horizon/horizon-container-puppet.yaml#L132

Since the patches are merged already, if there is indeed a problem with horizon logging, then yes, please open a second BZ with a clear evidence showing how to reproduce.

(In reply to Chris Sibbitt from comment #10)
> If using ExtraData is not good enough for some reason, then you will need to
> file a Request For Enhancement as there is currently no support for
> forwarding of host logs in tripleo.

Using ExtraConfig is fine, it works, but I don't know what's the criteria for including one log on the fixed version and not the other. Perhaps /var/log/messages is not included for a reason, and I just don't know it. 

Eventually, my customer is going to update to 16.2.2, and they will be able to remove the custom definition they added for /var/log/containers/haproxy.log, which is then included by default, but they won't be able to remove the custom definition for /var/log/messages, and maybe they should, unless there's a specific reason why this one is not included. If there's no reason, then I can fill another BZ, no problem.

(In reply to Chris Sibbitt from comment #11)
> Is there an additional problem with horizon logging? Alex mentioned it in
> the first comment, but I understood from the wording that he was comparing
> the missing examples to a working example.  The options that were missing
> for haproxy are present for horizon:
> https://github.com/openstack/tripleo-heat-templates/blob/stable/train/
> deployment/horizon/horizon-container-puppet.yaml#L132
> 
> Since the patches are merged already, if there is indeed a problem with
> horizon logging, then yes, please open a second BZ with a clear evidence
> showing how to reproduce.

Yeah, I think it's an issue. It's not added by default:

$ ansible -i inventory.yaml -b -m shell -a "egrep -w file /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf" overcloud-controller-0
overcloud-controller-0 | CHANGED | rc=0 >>
  file="/var/log/containers/cinder/cinder-scheduler.log"
  file="/var/log/containers/cinder/cinder-volume.log"
  file="/var/log/containers/glance/api.log"
  file="/var/log/containers/haproxy/haproxy.log"
  file="/var/log/containers/heat/heat_api_cfn.log"
  file="/var/log/containers/heat/heat_api.log"
  file="/var/log/containers/heat/heat-engine.log"
  file="/var/log/containers/keystone/keystone.log"
  file="/var/log/containers/neutron/server.log"
  file="/var/log/containers/nova/nova-api.log"
  file="/var/log/containers/nova/nova-conductor.log"
  file="/var/log/containers/nova/nova-metadata-api.log"
  file="/var/log/containers/nova/nova-scheduler.log"
  file="/var/log/containers/nova/nova-novncproxy.log"
  file="/var/log/containers/octavia/api.log"
  file="/var/log/containers/octavia/health-manager.log"
  file="/var/log/containers/octavia/housekeeping.log"
  file="/var/log/containers/octavia/worker.log"
  file="/var/log/host/pacemaker/pacemaker.log"
  file="/var/log/host/messages"

I will open a separate BZ for it.

>"unless there's a specific reason why this one [/var/log/host/messages] is not included"

I don't know of one offhand, but this is my first contact with logging in OSP. My gut says that the tripleo logging config is very service oriented, and the host's messages log is not associated with an OSP service so it is not included by default. As far as I can tell, this log has never been included in OSP 16, so it's not a "bug" that it is missing, it's an enhancement. I would want to check with those more familiar with the history if there is a specific reason it is excluded or if there is any other impact of including it by default.

>"I don't know what's the criteria for including one log on the fixed version and not the other"

HAProxy is an OSP service, and we should expect its logging to work the same way as other OSP services. This made it easy for me to justify this as a "bug fix" and immediately produce a patch for you and get it expediently backported to OSP 16 that adds the config for its log. If the existing Horizon config is broken, that can also be treated as a bug; I'm sorry if there was a miscommunication about that particular file. 

I think this will go more smoothly if you open one BZ per issue (Horizon vs host logging) as they may be seen differently by the team.

Thanks.

(In reply to Chris Sibbitt from comment #14)
> 
> I think this will go more smoothly if you open one BZ per issue (Horizon vs
> host logging) as they may be seen differently by the team.
> 

Done, Bug #2034793 (messages) and Bug #2034730 (Horizon)

haproxy logs are now collected. Regarding horizon and /var/log/messages separate bzs were opened.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack Platform 16.2 (openstack-tripleo-heat-templates) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0995