Bug 1728954

Summary:	Rsyslog prometheus exporter couldn't start.
Product:	OpenShift Container Platform	Reporter:	Qiaoling Tang <qitang>
Component:	Logging	Assignee:	Rich Megginson <rmeggins>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.2.0	CC:	aos-bugs, rmeggins
Target Milestone:	---
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 06:33:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Qiaoling Tang 2019-07-11 06:15:14 UTC

Description of problem:
Deploy logging using rsyslog as collector, set ENABLE_PROMETHEUS_ENDPOINT=true in ds/rsyslog, wait for about 10 minutes, try to curl rsyslog prometheus metrics, no result. The port 24231 is not started.

$ oc rsh rsyslog-ddmzb
Defaulting container name to rsyslog.
Use 'oc describe pod/rsyslog-ddmzb -n openshift-logging' to see all of the containers in this pod.
sh-4.2# env |grep -i prometheus
ENABLE_PROMETHEUS_ENDPOINT=true
sh-4.2# ps -eaf
UID         PID   PPID  C STIME TTY          TIME CMD
root          1      0  0 06:04 ?        00:00:00 /usr/bin/pod
root         13      0  1 06:04 ?        00:00:02 /usr/sbin/rsyslogd -i /var/run/rsyslogd.pid -f /etc/rsyslog/conf/rsyslog.conf -n
root         58      0  0 06:04 ?        00:00:00 /usr/sbin/crond -n
root         64      0  0 06:06 pts/0    00:00:00 /bin/sh
root         73     64  0 06:07 pts/0    00:00:00 ps -eaf
sh-4.2# lsof -i -P -n | grep LISTEN
sh-4.2# lsof -i -P -n              
COMMAND  PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
rsyslogd  13 root   26u  IPv4 11691926      0t0  TCP 10.131.0.64:53032->172.30.0.1:443 (ESTABLISHED)
sh-4.2# ls /etc/rsyslog.d/
00-global.conf		 55-elasticsearch.conf	  68-elasticsearch.conf        k8s_filename.rulebase	parse_json_skip_empty.rulebase
05-common-defaults.conf  60-mmk8s.conf		  99-logging.conf	       multiline-json.rulebase	prio_to_level.json
10-viaq-modules.conf	 65-viaq-formatting.conf  crio.rulebase		       normalize_level.json	rsyslog_prometheus_alerts.yaml
20-viaq-templates.conf	 67-mmexternal.conf	  k8s_container_name.rulebase  parse_json.rulebase
sh-4.2# 


Version-Release number of selected component (if applicable):
sh-4.2# ls /root/buildinfo/
Dockerfile-openshift-ose-base-v4.0-201907070546
Dockerfile-openshift-ose-logging-rsyslog-v4.2.0-201907082219
Dockerfile-openshift-ose-base-v4.2.0-201907071316
Dockerfile-ubi7-7.6-177.1561619828


How reproducible:
Always

Steps to Reproduce:
1.Deploy logging, set rsyslog as log collector
2. set ENABLE_PROMETHEUS_ENDPOINT=true
3.wait until all pods running, try to get rsyslog prometheus metrics

Actual results:
no rsyslog prometheus metrics exposed

Expected results:
should expose prometheus metrics

Additional info:

Comment 1 Rich Megginson 2019-07-11 22:54:53 UTC

There was a problem with recent images.  Please try the latest.

Comment 2 Qiaoling Tang 2019-07-12 01:07:59 UTC

The ose-logging-rsyslog-v4.2.0-201907082219 is the latest one, will test again once a new image is built.

Comment 3 Rich Megginson 2019-07-12 02:36:40 UTC

Also the cluster-logging-operator image broke rsyslog - the latest should fix it

Comment 5 Qiaoling Tang 2019-07-15 02:18:30 UTC

Verified in ose-cluster-logging-operator-v4.2.0-201907121819 and ose-logging-rsyslog-v4.2.0-201907141446

Comment 6 errata-xmlrpc 2019-10-16 06:33:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922