Description of problem: All customers require logging, which is currently (as of OSP 13) provided by fluentd. Due to ECL promoting use of rsyslog in RHEL 8, we have to consider a move to logging and forwarding using rsyslog to be compatible with ECL. Since ECL is moving to rsyslog support, this means that we will have to start supporting rsyslog in OSP 15, with parallel fluentd support (while indicating to customers that fluentd is being deprecated). This avoids OSP 16 (Long Life Release) having to support fluentd. During upgrade any fluentd logs not sent to central logging will need to be converted to rsyslog logs. We need to also look at the issue of double sending of logs once the upgrade is done. We need to take a crawl-walk-run approach to replacing rsyslog. The first step is do the same fluentd is doing and use the local file system. Containers send logs to var/log/containers/<service-name> files on host that they are running on, in fact some containers are bind mounted to the host. Like Fluentd, rsyslog should collects the files and dispatch to the servers specified during configuration. Requirements are 1. Rsyslog is configured via director during deployment to collect and forward container logs to be functionally equivalent to the fluentd installation. 2. Rsyslog log forwarding can be configured by the user in the same way as provided for fluentd Configuration changes will be required for the HEAT templates for installation of rsyslog. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Please,provide instructions for QA about testing.
*** Bug 1745578 has been marked as a duplicate of this bug. ***
I think this required info has already been provided, but just in case. As rsyslog was meant to replace fluentd, we need to do the same test as we do with fluentd.
Test instruction: Deploy overcloud with /usr/share/openstack-tripleo-heat-templates/logging-environment-rsyslog.yaml and point deployed rsyslog services to server side Elasticsearch cluster by RsyslogElasticsearchSetting configuration option in your env file. The content of the option should be something like: RsyslogElasticsearchSetting: Server="<IP>:<PORT>" You should start seeing logs in the ES after deploy.
Tests were resumed and we found another issues with the puppet. Seems like a bug in THT. Jan 10 09:30:10 controller-0 puppet-user[636927]: Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, has_key(): expects the first argument to be a hash, got ["file", "/var/log/containers/keystone/keystone.log"] which is of type Array (file: /etc/puppet/modules/tripleo/manifests/profile/base/logging/rsyslog/file_input.pp, line: 21, column: 12) (file: /etc/puppet/modules/tripleo/manifests/profile/base/logging/rsyslog.pp, line: 118) on node controller-0.redhat.local The BZ related to rsyslog was moved back to "Assign"
Hi Martin, I have edited the contents of the Doc Text field (below) which will appear in the RHOSP 16.0 GA Release Notes. Please review and make any necessary changes. Thanks for your help with this, --Greg PROPOSED DOC TEXT EDIT ---------------------- In Red Hat OpenStack Platform 16.0, a technology preview has been added to the Orchestration service (heat) for rsyslog changes: - Rsyslog is configured to collect and forward container logs to be functionally equivalent to the fluentd installation. - Administrators can configure rsyslog log forwarding in the same way as fluentd.
rsyslogs successfully connects to ES instance running on undercloud: ------------------------------------------------------------------------ (undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/_cat/indices yellow open .monitoring-es-6-2020.01.13 4YD7hwBjRT-cw0zqAXMqdw 1 1 3514 260 1.8mb 1.8mb yellow open controller-1.redhat.local-2020.01.13 w5MyclMvRW2WVq80HEb5dA 5 1 57408 0 24.4mb 24.4mb yellow open controller-2.redhat.local-2020.01.13 QPAh3o-vRk-h-e4h6GY6ig 5 1 54438 0 18.9mb 18.9mb yellow open compute-0.redhat.local-2020.01.13 ss2-cDulSKmw5xVMkd3UUA 5 1 1468 0 998kb 998kb yellow open controller-0.redhat.local-2020.01.13 c9om8QJSSASJTv1omYT5VA 5 1 59178 0 20.2mb 20.2mb yellow open compute-1.redhat.local-2020.01.13 inWh-IaxQruwaXTLykstDQ 5 1 1469 0 946.9kb 946.9kb (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/controller-0.redhat.local-2020.01.13/_search?size=5 {"took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":65898,"max_score":1.0,"hits":[{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Nb7LoG8BJTk5ddR4BDu2","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.977 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/030_contract_add_project_tags_table.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"} },{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Q77LoG8BJTk5ddR4BTsh","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/037_contract_remove_service_and_region_fk_for_registered_limit.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"} },{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Rr7LoG8BJTk5ddR4BTso","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/039_placeholder.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"} },{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Sb7LoG8BJTk5ddR4BTs5","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.981 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/040_placeholder.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"} },{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"VL7LoG8BJTk5ddR4BTtS","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.982 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/046_contract_old_password_data_to_password_hash_column.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"} }]}}(undercloud) [stack@undercloud-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283