Bug 1623619

Summary:	[RFE] Rsyslog - test/configure rsyslog with ES cluster
Product:	Red Hat OpenStack	Reporter:	Anandeep Pannu <apannu>
Component:	openstack-tripleo-heat-templates	Assignee:	Martin Magr <mmagr>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Leonid Natapov <lnatapov>
Severity:	high	Docs Contact:
Priority:	high
Version:	15.0 (Stein)	CC:	augol, dcadzow, lmadsen, mburns, mmagr, oblaut, pkilambi
Target Milestone:	Upstream M2	Keywords:	FutureFeature, OtherQA, Triaged
Target Release:	16.0 (Train on RHEL 8.1)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-08-24 20:34:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1776497, 1777235
Bug Blocks:

Description Anandeep Pannu 2018-08-29 18:26:33 UTC

Description of problem:

Problem
All customers require logging, which is currently (as of OSP 13) provided by fluentd. Due to ECL promoting use of rsyslog in RHEL 8, we have to consider a move to logging and forwarding using rsyslog to be compatible with ECL.  Since ECL is moving to rsyslog support, this means that we will have to start supporting  rsyslog in OSP 15, with parallel fluentd support (while indicating to customers that fluentd is being deprecated). This avoids OSP 16 (Long Life Release) having to support fluentd.   During upgrade any fluentd logs not sent to central logging will need to be converted to rsyslog logs. We need to also look at the issue of double sending of logs once the upgrade is done. 

We need to take a crawl-walk-run approach to replacing rsyslog.  The first step is do the same fluentd is doing and use the local file system. Containers send logs to var/log/containers/<service-name> files on host that they are running on, in fact some containers are bind mounted to the host. Like Fluentd, rsyslog should collects the files and dispatch to the servers specified during configuration.  

We will need to configure and then test the compatibility of the logs sent by rsyslog with ECL (Engineering Common Logging) which was using fluentd logs in the past.  




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Anandeep Pannu 2018-09-10 19:43:48 UTC

As a clarification this RFE makes sure that
1. The logs work with ECL the same as the fluentd logs work today - no extra functionality is implied 
2. The QE / tests should be to make sure that the logs have the same functionality with the new component and that there is no regression in functionality.

Comment 6 Leonid Natapov 2018-12-11 15:28:58 UTC

Martin,please,provide instructions for QA about testing. Thanks !

Comment 12 Martin Magr 2020-01-14 12:45:44 UTC

I've been able to successfully connect rsyslogs to ES instance running on undercloud on Leonid's env:

(undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/_cat/indices
yellow open .monitoring-es-6-2020.01.13          4YD7hwBjRT-cw0zqAXMqdw 1 1  3514 260   1.8mb   1.8mb
yellow open controller-1.redhat.local-2020.01.13 w5MyclMvRW2WVq80HEb5dA 5 1 57408   0  24.4mb  24.4mb
yellow open controller-2.redhat.local-2020.01.13 QPAh3o-vRk-h-e4h6GY6ig 5 1 54438   0  18.9mb  18.9mb
yellow open compute-0.redhat.local-2020.01.13    ss2-cDulSKmw5xVMkd3UUA 5 1  1468   0   998kb   998kb
yellow open controller-0.redhat.local-2020.01.13 c9om8QJSSASJTv1omYT5VA 5 1 59178   0  20.2mb  20.2mb
yellow open compute-1.redhat.local-2020.01.13    inWh-IaxQruwaXTLykstDQ 5 1  1469   0 946.9kb 946.9kb
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ curl http://192.168.24.1:9200/controller-0.redhat.local-2020.01.13/_search?size=5
{"took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":65898,"max_score":1.0,"hits":[{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Nb7LoG8BJTk5ddR4BDu2","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.977 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/030_contract_add_project_tags_table.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Q77LoG8BJTk5ddR4BTsh","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/037_contract_remove_service_and_region_fk_for_registered_limit.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Rr7LoG8BJTk5ddR4BTso","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.980 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/039_placeholder.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"Sb7LoG8BJTk5ddR4BTs5","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.981 31 DEBUG migrate.versioning.script.base [-] Script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/040_placeholder.py loaded successfully __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:30"}
},{"_index":"controller-0.redhat.local-2020.01.13","_type":"events","_id":"VL7LoG8BJTk5ddR4BTtS","_score":1.0,"_source":{"@timestamp":"Jan 13 21:24:09", "host":"controller-0", "severity":"notice", "facility":"local0", "tag":"openstack.keystone", "source":"openstack.keystone", "message":"2020-01-13 21:23:45.982 31 DEBUG migrate.versioning.script.base [-] Loading script \/usr\/lib\/python3.6\/site-packages\/keystone\/common\/sql\/contract_repo\/versions\/046_contract_old_password_data_to_password_hash_column.py... __init__ \/usr\/lib\/python3.6\/site-packages\/migrate\/versioning\/script\/base.py:27"}
}]}}(undercloud) [stack@undercloud-0 ~]$

Comment 13 Leonid Natapov 2020-01-23 16:54:46 UTC

Moving back to ON_QA since it was not tested in official puddle

Comment 14 Leonid Natapov 2020-05-26 11:59:14 UTC

Verified.

Comment 15 Leif Madsen 2020-08-24 20:34:44 UTC

I can confirm testing and making this work as well. Closing as current release since there's no fixed in version, and this is against OSP16.0, so I don't expect this task to ever close automatically.