Bug 1735548
Summary: | The pipeline_metadata.collector.ipaddr6 is not correct when using rsyslog as log collector. | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> |
Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
Severity: | medium | Docs Contact: | |
Priority: | unspecified | ||
Version: | 4.2.0 | CC: | aos-bugs, nhosoi, rmeggins |
Target Milestone: | --- | ||
Target Release: | 4.3.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-01-23 11:04:29 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Qiaoling Tang
2019-08-01 05:05:33 UTC
"pipeline_metadata": { "collector": { "ipaddr4": "10.0.173.53", $ oc get node -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIMEip-10-0-173-53.eu-west-2.compute.internal Ready worker 127m v1.14.0+c569285e9 10.0.173.53 <none> Red Hat Enterprise Linux CoreOS 42.80.20190731.2 (Ootpa) 4.18.0-80.7.1.el8_0.x86_64 cri-o://1.14.10-0.5.dev.rhaos4.2.gitcf4220b.el8-dev As you can see, the ipaddr4 field is reporting the INTERNAL-IP of the node. This value comes from the Kubernetes Downward API NODE_IPV4: - name: NODE_IPV4 valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP which is specified in the operator here: https://github.com/openshift/cluster-logging-operator/blob/master/pkg/k8shandler/rsyslog.go#L349 {Name: "NODE_IPV4", ValueFrom: &v1.EnvVarSource{FieldRef: &v1.ObjectFieldSelector{FieldPath: "status.hostIP"}}}, the downward api is described here: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api * status.hostIP - the node’s IP, available since v1.7.0-alpha.1 I thought since rsyslog is a daemonset, it would be most useful to report the IP address of the node where it is running. Which IP address are you expecting to be reported? note that fluentd does the same thing with NODE_IPV4: https://github.com/openshift/cluster-logging-operator/blob/master/pkg/k8shandler/fluentd.go#L233 but it ignores it https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/run.sh#L71 So perhaps the bug should be "the fluentd pipeline_metadata.collector.ipaddr4 and pipeline_metadata.collector.ipaddr6 are not correct" We discussed this at the logging scrum today. Since we cannot get the IPv6 address of the node, we should simply omit that field. For rsyslog, this means we should set IPADDR6="" in cluster-logging-operator files/rsyslog/rsyslog.sh then, in 65-viaq-formatting.conf, do this: if strlen(`echo $IPADDR6`) > 0 then { set $!pipeline_metadata!collector!ipaddr6 = `echo $IPADDR6`; } That way, if we figure out how to get the node IPv6 addr, we can change it in rsyslog.sh, and rsyslog will just work. For fluentd, it is much trickier. We'll have to change the viaq plugin to also conditionally set ipaddr6 https://github.com/ViaQ/fluent-plugin-viaq_data_model/blob/master/lib/fluent/plugin/filter_viaq_data_model.rb#L370 which is set here: https://github.com/ViaQ/fluent-plugin-viaq_data_model/blob/master/lib/fluent/plugin/filter_viaq_data_model.rb#L206 from the environment variable. The environment variable is set here: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/run.sh#L72 So we'll need to change run.sh to set IPADDR6="" like rsyslog. Then, change the plugin at line 206 like this: if ENV['IPADDR6'] && ENV['IPADDR6'].length > 0 @ipaddr6 = ENV['IPADDR6'] else @ipaddr6 = nil end then around 370 like this: if @ipaddr6 record['pipeline_metadata'][pipeline_type]['ipaddr6'] = @ipaddr6 end Do we want to include the fix for fluentd in this bug or open a separate bug? (In reply to Noriko Hosoi from comment #7) > Do we want to include the fix for fluentd in this bug or open a separate bug? I would include it in the fix for this bz. Verified with ose-logging-rsyslog/images/v4.2.0-201908091819 and ose-logging-fluentd-v4.2.0-201908091819 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |