Bug 1623152

Summary: [RFE] Configuration changes for TripleO HEAT templates for rsyslog
Product: Red Hat OpenStack Reporter: Anandeep Pannu <apannu>
Component: openstack-tripleo-heat-templatesAssignee: Martin Magr <mmagr>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: acanan, augol, ccopello, dcadzow, gregraka, jamsmith, jappleii, jbadiapa, mburns, mmagr, mvelavar, nlevinki, pkilambi, rmccabe, sclewis
Target Milestone: gaKeywords: FutureFeature, TechPreview, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.1.1-0.20190826090137.08a6b6c.el8ost puppet-tripleo-11.4.1-0.20200118215809.6f9bf6c.el8ost Doc Type: Technology Preview
Doc Text:
In Red Hat OpenStack Platform 16.0, a technology preview has been added to the Orchestration service (heat) for rsyslog changes: - Rsyslog is configured to collect and forward container logs to be functionally equivalent to the fluentd installation. - Administrators can configure rsyslog log forwarding in the same way as fluentd.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-06 14:37:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1684576, 1776497    
Bug Blocks: 1744125, 1779122    

Description Anandeep Pannu 2018-08-28 15:09:35 UTC
Description of problem:


All customers require logging, which is currently (as of OSP 13) provided by fluentd. Due to ECL promoting use of rsyslog in RHEL 8, we have to consider a move to logging and forwarding using rsyslog to be compatible with ECL.  Since ECL is moving to rsyslog support, this means that we will have to start supporting  rsyslog in OSP 15, with parallel fluentd support (while indicating to customers that fluentd is being deprecated). This avoids OSP 16 (Long Life Release) having to support fluentd.   During upgrade any fluentd logs not sent to central logging will need to be converted to rsyslog logs. We need to also look at the issue of double sending of logs once the upgrade is done. 


We need to take a crawl-walk-run approach to replacing rsyslog.  The first step is do the same fluentd is doing and use the local file system. Containers send logs to var/log/containers/<service-name> files on host that they are running on, in fact some containers are bind mounted to the host. Like Fluentd, rsyslog should collects the files and dispatch to the servers specified during configuration.  

Requirements are 
1. Rsyslog is configured via director during deployment to collect and forward container logs to be functionally equivalent to the fluentd installation.
2. Rsyslog log forwarding can be configured by the user in the same way as provided for fluentd

Configuration changes will be required for the HEAT templates for installation of rsyslog.




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 7 Leonid Natapov 2018-12-11 15:28:16 UTC
Please,provide instructions for QA about testing.

Comment 10 Martin Magr 2019-08-26 12:27:15 UTC
*** Bug 1745578 has been marked as a duplicate of this bug. ***

Comment 11 Juan Badia Payno 2019-10-23 07:02:42 UTC
I think this required info has already been provided, but just in case.
As rsyslog was meant to replace fluentd, we need to do the same test as we do with fluentd.

Comment 12 Martin Magr 2019-11-12 11:32:07 UTC
Test instruction:

Deploy overcloud with /usr/share/openstack-tripleo-heat-templates/logging-environment-rsyslog.yaml and point deployed rsyslog services to server side Elasticsearch cluster by RsyslogElasticsearchSetting configuration option in your env file. The content of the option should be something like:

  RsyslogElasticsearchSetting:
    Server="<IP>:<PORT>"

You should start seeing logs in the ES after deploy.

Comment 16 Leonid Natapov 2020-01-13 11:45:34 UTC
Tests were resumed and we found another issues with the puppet. Seems like a bug in THT.

Jan 10 09:30:10 controller-0 puppet-user[636927]: Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, has_key(): expects the first argument to be a hash, got ["file", "/var/log/containers/keystone/keystone.log"] which is of type Array (file: /etc/puppet/modules/tripleo/manifests/profile/base/logging/rsyslog/file_input.pp, line: 21, column: 12) (file: /etc/puppet/modules/tripleo/manifests/profile/base/logging/rsyslog.pp, line: 118) on node controller-0.redhat.local

The BZ related to rsyslog was moved back to "Assign"

Comment 23 Greg Rakauskas 2020-01-25 00:16:04 UTC
Hi Martin,

I have edited the contents of the Doc Text field (below) which will appear in
the RHOSP 16.0 GA Release Notes. Please review and make any necessary changes.

Thanks for your help with this,
--Greg


PROPOSED DOC TEXT EDIT
----------------------
In Red Hat OpenStack Platform 16.0, a technology preview has been added to the 
Orchestration service (heat) for rsyslog changes:

- Rsyslog is configured to collect and forward container logs to be functionally
  equivalent to the fluentd installation.

- Administrators can configure rsyslog log forwarding in the same way as 
  fluentd.

Comment 24 Leonid Natapov 2020-01-29 12:41:06 UTC
rsyslogs successfully connects to ES instance running on undercloud:
------------------------------------------------------------------------

(undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/_cat/indices
yellow open .monitoring-es-6-2020.01.13          4YD7hwBjRT-cw0zqAXMqdw 1 1  3514 260   1.8mb   1.8mb
yellow open controller-1.redhat.local-2020.01.13 w5MyclMvRW2WVq80HEb5dA 5 1 57408   0  24.4mb  24.4mb
yellow open controller-2.redhat.local-2020.01.13 QPAh3o-vRk-h-e4h6GY6ig 5 1 54438   0  18.9mb  18.9mb
yellow open compute-0.redhat.local-2020.01.13    ss2-cDulSKmw5xVMkd3UUA 5 1  1468   0   998kb   998kb
yellow open controller-0.redhat.local-2020.01.13 c9om8QJSSASJTv1omYT5VA 5 1 59178   0  20.2mb  20.2mb
yellow open compute-1.redhat.local-2020.01.13    inWh-IaxQruwaXTLykstDQ 5 1  1469   0 946.9kb 946.9kb
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/controller-0.redhat.local-2020.01.13/_search?size=5
{"took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":65898,"max_score":1.0,"hits":[{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Nb7LoG8BJTk5ddR4BDu2","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.977 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/030_contract_add_project_tags_table.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Q77LoG8BJTk5ddR4BTsh","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/037_contract_remove_service_and_region_fk_for_registered_limit.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Rr7LoG8BJTk5ddR4BTso","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/039_placeholder.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Sb7LoG8BJTk5ddR4BTs5","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.981 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/040_placeholder.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"VL7LoG8BJTk5ddR4BTtS","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.982 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/046_contract_old_password_data_to_password_hash_column.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"}
}]}}(undercloud) [stack@undercloud-0 ~]$

Comment 26 errata-xmlrpc 2020-02-06 14:37:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283