Bug 1808055

Summary:	[IPv6] Elasticsearch unable to perform oauth workflow because of hostname verification
Product:	OpenShift Container Platform	Reporter:	Periklis Tsirakidis <periklis>
Component:	Logging	Assignee:	Periklis Tsirakidis <periklis>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.4	CC:	aos-bugs, cstark, jcantril
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:21:52 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1808053
Bug Blocks:	1806995

Description Periklis Tsirakidis 2020-02-27 17:42:29 UTC

This bug was initially created as a copy of Bug #1806995

I am copying this bug because: 



Description of problem:
There are not default pattern in the kibana for both kubeadmin and non-kubeadmin users.

Version-Release number of selected component (if applicable):
Baremetal machine launched by IPv6
4.3.0-0.nightly-2020-02-17-205936-ipv6.1
clusterlogging.4.3.4-202002241021
elasticsearch-operator.4.3.4-202002241021


How reproducible:
Always

Steps to Reproduce:
1. Deploy clusterlogging in IPv6 clusters.
2. Loging kibana via kubeamdin
3. Loging kibana as non-kubeadmin

Actual results:
No default pattern in kibana.

Expected results:
The pod log can be displayed in kibana

Additional info:

Comment 3 Anping Li 2020-03-04 14:36:17 UTC

openshift/ose-elasticsearch-operator:202003021217

1. Logs of rejected error in fluentd
2020-03-04 14:22:18 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="journal.system" time=2020-03-02 17:56:28.847640000 +0000 record={"_STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "_SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "systemd"=>{"t"=>{"BOOT_ID"=>"d519df282bb14d66939a9bcd540048fc", "CAP_EFFECTIVE"=>"3fffffffff", "CMDLINE"=>"/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=fd2e:6f44:5dd8:c956::114 --address=fd2e:6f44:5dd8:c956::114 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --v=3", "COMM"=>"hyperkube", "EXE"=>"/usr/bin/hyperkube", "GID"=>"0", "MACHINE_ID"=>"c44bc23a7e6a4668915dc7f7bf59ee44", "PID"=>"3139265", "SELINUX_CONTEXT"=>"system_u:system_r:unconfined_service_t:s0", "STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "SYSTEMD_CGROUP"=>"/system.slice/kubelet.service", "SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"kubelet.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"hyperkube"}}, "level"=>"info", "message"=>"MASTER_IP=\"fd2e:6f44:5dd8:c956::114\"", "hostname"=>"master-2.ocp-edge-cluster.qe.lab.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"fd2e:6f44:5dd8:c956::114", "ipaddr6"=>"fd01::3:90be:d3ff:fe00:a6\nfe80::6cb7:28ff:fef5:2378", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2020-03-04T10:59:20.457506+00:00", "version"=>"1.7.4 1.6.0"}}, "@timestamp"=>"2020-03-02T17:56:28.847640+00:00", "viaq_index_name"=>".operations.2020.03.02", "viaq_msg_id"=>"NDU2MWMzNDctZWE3Mi00MzlmLWEyM2QtY2JmYzMzZDRkZjgy"}
2020-03-04 14:22:18 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="journal.system" time=2020-03-02 17:56:28.847640000 +0000 record={"_STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "_SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "systemd"=>{"t"=>{"BOOT_ID"=>"d519df282bb14d66939a9bcd540048fc", "CAP_EFFECTIVE"=>"3fffffffff", "CMDLINE"=>"/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=fd2e:6f44:5dd8:c956::114 --address=fd2e:6f44:5dd8:c956::114 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --v=3", "COMM"=>"hyperkube", "EXE"=>"/usr/bin/hyperkube", "GID"=>"0", "MACHINE_ID"=>"c44bc23a7e6a4668915dc7f7bf59ee44", "PID"=>"3139265", "SELINUX_CONTEXT"=>"system_u:system_r:unconfined_service_t:s0", "STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "SYSTEMD_CGROUP"=>"/system.slice/kubelet.service", "SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"kubelet.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"hyperkube"}}, "level"=>"info", "message"=>"if [[ \"${K8S_NODE_IP}\" == \"${MASTER_IP}\" ]]; then", "hostname"=>"master-2.ocp-edge-cluster.qe.lab.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"fd2e:6f44:5dd8:c956::114", "ipaddr6"=>"fd01::3:90be:d3ff:fe00:a6\nfe80::6cb7:28ff:fef5:2378", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2020-03-04T10:59:20.458968+00:00", "version"=>"1.7.4 1.6.0"}}, "@timestamp"=>"2020-03-02T17:56:28.847640+00:00", "viaq_index_name"=>".operations.2020.03.02", "viaq_msg_id"=>"NjAwZDAzOTMtYzA0Ny00NWM2LWFkOGMtN2VkMWU4YWZmMjYw"}
2. e80::a81a:90ff:fe1c:9ee2' is not an IP string literal.

	... 48 more
[2020-03-04T14:29:38,415][DEBUG][o.e.a.b.TransportShardBulkAction] [elasticsearch-cdm-ammnz9zb-1] [.operations.2020.03.03][0] failed to execute bulk item (create) BulkShardRequest [[.operations.2020.03.03][0]] containing [4155] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [pipeline_metadata.collector.ipaddr6]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:468) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:591) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:465) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:484) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:383) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:465) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:484) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:383) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:530) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:507) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:466) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:145) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178) [search-guard-ssl-5.6.16.23-redhat-1.jar:5.6.16.23-redhat-1]
	at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:107) [search-guard-5-5.6.16.19-3-redhat-1.jar:?]
	at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92) [search-guard-ssl-5.6.16.23-redhat-1.jar:5.6.16.23-redhat-1]
	at com.floragunn.searchguard.SearchGuardPlugin$5$1.messageReceived(SearchGuardPlugin.java:493) [search-guard-5-5.6.16.19-3-redhat-1.jar:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.lang.IllegalArgumentException: 'fd01::6:90be:d3ff:fe00:aa
fe80::a81a:90ff:fe1c:9ee2' is not an IP string literal.
	at org.elasticsearch.common.network.InetAddresses.forString(InetAddresses.java:333) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.IpFieldMapper.parseCreateField(IpFieldMapper.java:377) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2]
	... 48 more

3. kibana plugin is red

Comment 5 Anping Li 2020-04-23 05:32:43 UTC

Move to 4.5 as we don't backport ipv6 fix to 4.4.z any more.  If the 4.4 cluster is ready. I will verify on it. and drop a comment.

Comment 6 Periklis Tsirakidis 2020-04-23 06:14:20 UTC

@Anping

The fix for 4.5 landed and is verified by [1]. FWIW, I don't see any official communication that IPv6 is not going to be fixed for 4.4.z anymore. Can you provide more info?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1808053

Comment 8 Anping Li 2020-05-22 14:29:30 UTC

Verified in 4.5. The user can extract logs from Elasticsearch base on their roles. No Hostname error in pods

Comment 11 errata-xmlrpc 2020-07-13 17:21:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409