This bug was initially created as a copy of Bug #1806995 I am copying this bug because: Description of problem: There are not default pattern in the kibana for both kubeadmin and non-kubeadmin users. Version-Release number of selected component (if applicable): Baremetal machine launched by IPv6 4.3.0-0.nightly-2020-02-17-205936-ipv6.1 clusterlogging.4.3.4-202002241021 elasticsearch-operator.4.3.4-202002241021 How reproducible: Always Steps to Reproduce: 1. Deploy clusterlogging in IPv6 clusters. 2. Loging kibana via kubeamdin 3. Loging kibana as non-kubeadmin Actual results: No default pattern in kibana. Expected results: The pod log can be displayed in kibana Additional info:
openshift/ose-elasticsearch-operator:202003021217 1. Logs of rejected error in fluentd 2020-03-04 14:22:18 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="journal.system" time=2020-03-02 17:56:28.847640000 +0000 record={"_STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "_SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "systemd"=>{"t"=>{"BOOT_ID"=>"d519df282bb14d66939a9bcd540048fc", "CAP_EFFECTIVE"=>"3fffffffff", "CMDLINE"=>"/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=fd2e:6f44:5dd8:c956::114 --address=fd2e:6f44:5dd8:c956::114 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --v=3", "COMM"=>"hyperkube", "EXE"=>"/usr/bin/hyperkube", "GID"=>"0", "MACHINE_ID"=>"c44bc23a7e6a4668915dc7f7bf59ee44", "PID"=>"3139265", "SELINUX_CONTEXT"=>"system_u:system_r:unconfined_service_t:s0", "STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "SYSTEMD_CGROUP"=>"/system.slice/kubelet.service", "SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"kubelet.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"hyperkube"}}, "level"=>"info", "message"=>"MASTER_IP=\"fd2e:6f44:5dd8:c956::114\"", "hostname"=>"master-2.ocp-edge-cluster.qe.lab.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"fd2e:6f44:5dd8:c956::114", "ipaddr6"=>"fd01::3:90be:d3ff:fe00:a6\nfe80::6cb7:28ff:fef5:2378", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2020-03-04T10:59:20.457506+00:00", "version"=>"1.7.4 1.6.0"}}, "@timestamp"=>"2020-03-02T17:56:28.847640+00:00", "viaq_index_name"=>".operations.2020.03.02", "viaq_msg_id"=>"NDU2MWMzNDctZWE3Mi00MzlmLWEyM2QtY2JmYzMzZDRkZjgy"} 2020-03-04 14:22:18 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="journal.system" time=2020-03-02 17:56:28.847640000 +0000 record={"_STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "_SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "systemd"=>{"t"=>{"BOOT_ID"=>"d519df282bb14d66939a9bcd540048fc", "CAP_EFFECTIVE"=>"3fffffffff", "CMDLINE"=>"/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=fd2e:6f44:5dd8:c956::114 --address=fd2e:6f44:5dd8:c956::114 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --v=3", "COMM"=>"hyperkube", "EXE"=>"/usr/bin/hyperkube", "GID"=>"0", "MACHINE_ID"=>"c44bc23a7e6a4668915dc7f7bf59ee44", "PID"=>"3139265", "SELINUX_CONTEXT"=>"system_u:system_r:unconfined_service_t:s0", "STREAM_ID"=>"3e8efe6791964d27b4c05630c6f804d5", "SYSTEMD_CGROUP"=>"/system.slice/kubelet.service", "SYSTEMD_INVOCATION_ID"=>"d97d864ae0e142158a8d4f227ff5c9ec", "SYSTEMD_SLICE"=>"system.slice", "SYSTEMD_UNIT"=>"kubelet.service", "TRANSPORT"=>"stdout", "UID"=>"0"}, "u"=>{"SYSLOG_FACILITY"=>"3", "SYSLOG_IDENTIFIER"=>"hyperkube"}}, "level"=>"info", "message"=>"if [[ \"${K8S_NODE_IP}\" == \"${MASTER_IP}\" ]]; then", "hostname"=>"master-2.ocp-edge-cluster.qe.lab.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"fd2e:6f44:5dd8:c956::114", "ipaddr6"=>"fd01::3:90be:d3ff:fe00:a6\nfe80::6cb7:28ff:fef5:2378", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2020-03-04T10:59:20.458968+00:00", "version"=>"1.7.4 1.6.0"}}, "@timestamp"=>"2020-03-02T17:56:28.847640+00:00", "viaq_index_name"=>".operations.2020.03.02", "viaq_msg_id"=>"NjAwZDAzOTMtYzA0Ny00NWM2LWFkOGMtN2VkMWU4YWZmMjYw"} 2. e80::a81a:90ff:fe1c:9ee2' is not an IP string literal. ... 48 more [2020-03-04T14:29:38,415][DEBUG][o.e.a.b.TransportShardBulkAction] [elasticsearch-cdm-ammnz9zb-1] [.operations.2020.03.03][0] failed to execute bulk item (create) BulkShardRequest [[.operations.2020.03.03][0]] containing [4155] requests org.elasticsearch.index.mapper.MapperParsingException: failed to parse [pipeline_metadata.collector.ipaddr6] at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:468) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:591) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:465) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:484) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:383) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:465) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:484) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:383) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:530) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:507) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:466) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:145) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178) [search-guard-ssl-5.6.16.23-redhat-1.jar:5.6.16.23-redhat-1] at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:107) [search-guard-5-5.6.16.19-3-redhat-1.jar:?] at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92) [search-guard-ssl-5.6.16.23-redhat-1.jar:5.6.16.23-redhat-1] at com.floragunn.searchguard.SearchGuardPlugin$5$1.messageReceived(SearchGuardPlugin.java:493) [search-guard-5-5.6.16.19-3-redhat-1.jar:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] Caused by: java.lang.IllegalArgumentException: 'fd01::6:90be:d3ff:fe00:aa fe80::a81a:90ff:fe1c:9ee2' is not an IP string literal. at org.elasticsearch.common.network.InetAddresses.forString(InetAddresses.java:333) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.IpFieldMapper.parseCreateField(IpFieldMapper.java:377) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] ... 48 more 3. kibana plugin is red
Move to 4.5 as we don't backport ipv6 fix to 4.4.z any more. If the 4.4 cluster is ready. I will verify on it. and drop a comment.
@Anping The fix for 4.5 landed and is verified by [1]. FWIW, I don't see any official communication that IPv6 is not going to be fixed for 4.4.z anymore. Can you provide more info? [1] https://bugzilla.redhat.com/show_bug.cgi?id=1808053
Verified in 4.5. The user can extract logs from Elasticsearch base on their roles. No Hostname error in pods
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409