Bug 1499813

Summary: Fluentd configuration file is not right on non-ops cluster
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.1CC: anli, aos-bugs, jcantril, jkarasek, juzhao, rmeggins
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-07 07:12:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2017-10-09 12:49:22 UTC
Description of problem:
non-ops cluster, there should be filter-post-z-retag-one.conf under /etc/fluent/configs.d/openshift,
but it is filter-post-z-retag-two.conf now.

# oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-x7j99                   1/1       Running   0          6m
logging-es-data-master-ztomkeg9-1-z5mh7   1/1       Running   0          6m
logging-fluentd-6cvbk                     1/1       Running   0          6m
logging-fluentd-k0804                     1/1       Running   0          6m
logging-kibana-1-13sd7                    2/2       Running   0          6m
# oc rsh logging-fluentd-6cvbk
sh-4.2# ls -al /var/lib/fluentd/
total 12
drwxr-xr-x.  2 root root    78 Oct  9 08:34 .
drwxr-xr-x. 15 root root   191 Oct  9 08:28 ..
-rw-r--r--.  1 root root 11641 Oct  9 08:34 buffer-output-es-ops-config.output_ops_tag.b55b1c69e2b63a0a8.log
sh-4.2# ls -al /etc/fluent/configs.d/openshift | grep filter-post
-rw-r--r--. 1 root root  549 Oct  9 08:28 filter-post-z-retag-two.conf

# oc rsh logging-fluentd-k0804
sh-4.2# ls -al /var/lib/fluentd/
total 204
drwxr-xr-x.  2 root root    142 Oct  9 08:35 .
drwxr-xr-x. 15 root root    191 Oct  9 08:28 ..
-rw-r--r--.  1 root root   9554 Oct  9 08:35 buffer-output-es-config.output_tag.b55b1c6cd28c9b5a6.log
-rw-r--r--.  1 root root 123297 Oct  9 08:35 buffer-output-es-ops-config.output_ops_tag.b55b1c6cd290ef66c.log
sh-4.2# ls -al /etc/fluent/configs.d/openshift | grep filter-post
-rw-r--r--. 1 root root  549 Oct  9 08:28 filter-post-z-retag-two.conf


Version-Release number of selected component (if applicable):
# rpm -qa | grep openshift-ansible
openshift-ansible-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-roles-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-docs-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-lookup-plugins-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-filter-plugins-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-playbooks-3.6.173.0.45-1.git.0.dc70c99.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.45-1.git.0.dc70c99.el7.noarch

Images
logging-auth-proxy:v3.6.173.0.47-1
logging-elasticsearch:v3.6.173.0.47-1
logging-curator:v3.6.173.0.47-1
logging-kibana:v3.6.173.0.21-20
logging-fluentd:v3.6.173.0.45-1

How reproducible:
Always 

Steps to Reproduce:
1. Disable ops cluster and deploy logging
2. Check Fluentd configuration file
3.

Actual results:
filter-post-z-retag-two.conf under /etc/fluent/configs.d/openshift

Expected results:
filter-post-z-retag-one.conf under /etc/fluent/configs.d/openshift

Additional info:

Comment 3 Noriko Hosoi 2017-10-09 15:57:01 UTC
Thank you very much, @Junqi.  I think I found a cause of the bug...

https://github.com/openshift/origin-aggregated-logging/pull/709
https://github.com/openshift/origin-aggregated-logging/pull/710

Comment 6 openshift-github-bot 2017-10-09 20:49:36 UTC
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/3933f70b8f71112a7dde6414dc0e0c7e1a5adfba
Merge pull request #709 from nhosoi/bz1499813

Automatic merge from submit-queue.

fluentd/run.sh - fixing a typo in OPS_PORT

https://bugzilla.redhat.com/show_bug.cgi?id=1499813
Bug 1499813 - Fluentd configuration file is not right on non-ops cluster

Comment 7 Junqi Zhao 2017-10-13 01:28:23 UTC
Please change to ON_QA, issue is fixed.
# oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-gg4w5                   1/1       Running   0          14m
logging-es-data-master-8jq1w3mn-1-5vhn2   1/1       Running   0          15m
logging-fluentd-02kdw                     1/1       Running   0          14m
logging-fluentd-3zqn5                     1/1       Running   0          14m
logging-kibana-1-83mv0                    2/2       Running   0          14m
# oc rsh logging-fluentd-02kdw
sh-4.2# ls -al /var/lib/fluentd/
total 876
drwxr-xr-x.  2 root root    214 Oct 12 21:17 .
drwxr-xr-x. 15 root root    191 Oct 12 21:01 ..
-rw-r--r--.  1 root root   8545 Oct 12 21:17 buffer-output-es-config.output_tag.b55b636ab916cc296.log
-rw-r--r--.  1 root root 709449 Oct 11 22:33 buffer-output-es-ops-config.output_ops_tag.b55b505c5336eae80.log
-rw-r--r--.  1 root root 171797 Oct 11 22:33 buffer-output-es-ops-config.output_ops_tag.q55b505c43f37ab8c.log
sh-4.2# ls -al /etc/fluent/configs.d/openshift | grep filter-post
-rw-r--r--. 1 root root  256 Oct 12 21:01 filter-post-z-retag-one.conf

# oc rsh logging-fluentd-3zqn5
sh-4.2# ls -al /var/lib/fluentd/
total 732
drwxr-xr-x.  2 root root    214 Oct 12 21:17 .
drwxr-xr-x. 15 root root    191 Oct 12 21:01 ..
-rw-r--r--.  1 root root 111378 Oct 12 21:17 buffer-output-es-config.output_tag.q55b636cafc97eb9e.log
-rw-r--r--.  1 root root 516582 Oct 11 22:33 buffer-output-es-ops-config.output_ops_tag.b55b505c63d6d827d.log
-rw-r--r--.  1 root root  32038 Oct 11 22:33 buffer-output-es-ops-config.output_ops_tag.q55b505c624644482.log
sh-4.2# ls -al /etc/fluent/configs.d/openshift | grep filter-post
-rw-r--r--. 1 root root  256 Oct 12 21:01 filter-post-z-retag-one.conf

Images:
logging-kibana:v3.6.173.0.49-2
logging-elasticsearch:v3.6.173.0.49-2
logging-fluentd:v3.6.173.0.49-2
logging-auth-proxy:v3.6.173.0.49-2
logging-curator:v3.6.173.0.49-2

Comment 8 Noriko Hosoi 2017-10-13 04:23:17 UTC
Thanks for verifying the bug, @Junqi!!

Comment 9 Noriko Hosoi 2017-10-16 18:39:25 UTC
This is a bug found in the new feature.  No Doc is needed.

> https://bugzilla.redhat.com/show_bug.cgi?id=1499813
> - Missing doc text

Comment 12 errata-xmlrpc 2017-12-07 07:12:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389