Description of problem: Could not find elasticsearch metrics in Prometheus server: No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]] The elasticsearch pods' metrics are exposed. es pod logs: $ oc exec elasticsearch-cdm-9cn7xpjg-1-7f7d47ddfc-zzcwc -- logs Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-9cn7xpjg-1-7f7d47ddfc-zzcwc -n openshift-logging' to see all of the containers in this pod. [2019-07-03T03:01:32,936][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] initializing ... [2019-07-03T03:01:33,008][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-9cn7xpjg-1] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/nvme1n1)]], net usable_space [9.7gb], net total_space [9.7gb], spins? [possibly], types [ext4] [2019-07-03T03:01:33,008][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-9cn7xpjg-1] heap size [1015.6mb], compressed ordinary object pointers [true] [2019-07-03T03:01:33,010][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] node name [elasticsearch-cdm-9cn7xpjg-1], node ID [oXsX3Ig8T_qdKifM-RyRXw] [2019-07-03T03:01:33,010][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] version[5.6.13], pid[1], build[921f2bd/2018-11-16T16:58:29.974Z], OS[Linux/4.18.0-80.4.2.el8_0.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b04] [2019-07-03T03:01:33,010][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -Dmapper.allow_dots_in_name=true, -Xms1024m, -Xmx1024m, -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof, -Dsg.display_lic_none=false, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.type=unpooled, -Djdk.tls.ephemeralDHKeySize=2048, -Des.path.home=/usr/share/elasticsearch] [2019-07-03T03:01:33,719][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elasticsearch [2019-07-03T03:01:33,719][WARN ][c.f.s.SearchGuardPlugin ] ### LICENSE NOTICE Search Guard ### If you use one or more of the following features in production make sure you have a valid Search Guard license (See https://floragunn.com/searchguard-validate-license) * Kibana Multitenancy * LDAP authentication/authorization * Active Directory authentication/authorization * REST Management API * JSON Web Token (JWT) authentication/authorization * Kerberos authentication/authorization * Document- and Fieldlevel Security (DLS/FLS) * Auditlogging In case of any doubt mail to <sales> ################################### [2019-07-03T03:01:33,720][WARN ][c.f.s.SearchGuardPlugin ] Consider setting -Djdk.tls.rejectClientInitiatedRenegotiation=true to prevent DoS attacks through client side initiated TLS renegotiation. [2019-07-03T03:01:33,743][INFO ][c.f.s.SearchGuardPlugin ] Node [elasticsearch-cdm-9cn7xpjg-1] is a transportClient: false/tribeNode: false/tribeNodeClient: false [2019-07-03T03:01:33,743][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS module not available [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_212 [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8 [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8 [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64 [2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 4.18.0-80.4.2.el8_0.x86_64 [2019-07-03T03:01:33,836][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 43 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] [2019-07-03T03:01:33,839][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 43 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] [2019-07-03T03:01:33,839][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively [2019-07-03T03:01:33,842][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: elasticsearch [2019-07-03T03:01:33,876][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] HTTPS client auth mode OPTIONAL [2019-07-03T03:01:33,876][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: logging-es [2019-07-03T03:01:33,879][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2019-07-03T03:01:33,879][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] [2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1] [2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1] [2019-07-03T03:01:33,906][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [aggs-matrix-stats] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [ingest-common] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-expression] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-groovy] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-mustache] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-painless] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [parent-join] [2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [percolator] [2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [reindex] [2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [transport-netty3] [2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [transport-netty4] [2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded plugin [openshift-elasticsearch] [2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-9cn7xpjg-1] loaded plugin [prometheus-exporter] [2019-07-03T03:01:34,818][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin io.fabric8.elasticsearch.plugin.OpenShiftElasticSearchPlugin [2019-07-03T03:01:34,848][INFO ][i.f.e.p.PluginSettings ] Using kibanaIndexMode: 'shared_ops' [2019-07-03T03:01:34,849][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.app from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json [2019-07-03T03:01:34,852][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.ops from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json [2019-07-03T03:01:34,854][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.empty from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json [2019-07-03T03:01:34,862][INFO ][i.f.e.p.OpenshiftRequestContextFactory] Using kibanaIndexMode: 'shared_ops' [2019-07-03T03:01:34,897][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS valve not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.DlsFlsValveImpl [2019-07-03T03:01:34,898][INFO ][c.f.s.SearchGuardPlugin ] Auditlog not available due to java.lang.ClassNotFoundException: com.floragunn.searchguard.auditlog.impl.AuditLogImpl [2019-07-03T03:01:34,899][INFO ][c.f.s.SearchGuardPlugin ] Privileges interceptor not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl [2019-07-03T03:01:34,987][INFO ][o.e.d.DiscoveryModule ] [elasticsearch-cdm-9cn7xpjg-1] using discovery type [zen] [2019-07-03T03:01:35,327][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] initialized [2019-07-03T03:01:35,327][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] starting ... [2019-07-03T03:01:35,416][INFO ][o.e.t.TransportService ] [elasticsearch-cdm-9cn7xpjg-1] publish_address {10.128.2.21:9300}, bound_addresses {[::]:9300} [2019-07-03T03:01:35,423][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-cdm-9cn7xpjg-1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-07-03T03:01:35,428][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if .searchguard index exists ... [2019-07-03T03:01:35,432][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [elasticsearch-cdm-9cn7xpjg-1] no known master node, scheduling a retry [2019-07-03T03:01:38,460][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:41,461][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:44,463][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:47,464][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:50,466][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:53,467][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:56,470][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:01:59,471][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:02,472][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:05,444][WARN ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] timed out while waiting for initial discovery state - timeout: 30s [2019-07-03T03:02:05,451][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [elasticsearch-cdm-9cn7xpjg-1] publish_address {10.128.2.21:9200}, bound_addresses {[::]:9200} [2019-07-03T03:02:05,451][INFO ][o.e.n.Node ] [elasticsearch-cdm-9cn7xpjg-1] started [2019-07-03T03:02:05,474][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:08,476][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:11,477][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:14,478][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:17,479][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:20,481][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:23,482][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:26,483][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:29,484][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:32,486][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:35,436][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [elasticsearch-cdm-9cn7xpjg-1] timed out while retrying [indices:admin/exists] after failure (timeout [1m]) [2019-07-03T03:02:35,437][ERROR][c.f.s.c.IndexBaseConfigurationRepository] Failure while checking MasterNotDiscoveredException[null] index .searchguard org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:209) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:311) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:238) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1] at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1056) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:576) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] [2019-07-03T03:02:35,442][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elasticsearch-cdm-9cn7xpjg-1] no known master node, scheduling a retry [2019-07-03T03:02:35,487][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:38,488][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:41,489][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:44,491][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:47,492][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again [2019-07-03T03:02:52,205][INFO ][o.e.c.s.ClusterService ] [elasticsearch-cdm-9cn7xpjg-1] detected_master {elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300}, added {{elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300},}, reason: zen-disco-receive(from master [master {elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300} committed version [1]]) [2019-07-03T03:02:55,588][ERROR][c.f.s.c.ConfigurationLoader] No data for config while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.searchguard) [2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for roles while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.searchguard) [2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for rolesmapping while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.searchguard) [2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for internalusers while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.searchguard) [2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for actiongroups while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.searchguard) [2019-07-03T03:03:15,478][WARN ][c.f.s.c.PrivilegesEvaluator] .kibana does not exist in cluster metadata [2019-07-03T03:03:55,572][WARN ][c.f.s.c.IndexBaseConfigurationRepository] Timeout, we just try again in a few seconds ... [2019-07-03T03:03:55,581][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'elasticsearch-cdm-9cn7xpjg-1' initialized [2019-07-03T03:06:32,009][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [2019-07-03T03:06:32,009][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {} [2019-07-03T03:06:32,767][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [2019-07-03T03:06:32,767][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {} [2019-07-03T03:07:01,287][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [2019-07-03T03:07:01,287][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {} [2019-07-03T03:07:02,760][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [2019-07-03T03:07:02,760][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {} Version-Release number of selected component (if applicable): ose-cluster-logging-operator-v4.2.0-201907021819 ose-logging-elasticsearch5-v4.2.0-201907021819 ose-elasticsearch-operator-v4.2.0-201907021819 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.0-0.nightly-2019-07-03-003353 True False 79m Cluster version is 4.2.0-0.nightly-2019-07-03-003353 How reproducible: Always Steps to Reproduce: 1. Deploy logging via OLM 2. check metrics in prometheus server 3. Actual results: could not find elasticsearch metrics in prometheus server Expected results: should be able to find elasticsearch metrics in prometheus server Additional info: Fluentd and rsyslog metrics can be found in Prometheus server
Created attachment 1586976 [details] clusterlogging-bundle
Created attachment 1586977 [details] elasticsearch-bundle
This may be fixed by https://github.com/openshift/origin-aggregated-logging/pull/1695
Please verify this is still an issue now that https://bugzilla.redhat.com/show_bug.cgi?id=1729297 is verified
Verified with ose-logging-elasticsearch5-v4.2.0-201907232219
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922