Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1726573

Summary: Could not find elasticsearch metrics in Prometheus server.
Product: OpenShift Container Platform Reporter: Qiaoling Tang <qitang>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aos-bugs, rmeggins
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:33:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
clusterlogging-bundle
none
elasticsearch-bundle none

Description Qiaoling Tang 2019-07-03 08:17:26 UTC 
Description of problem:
Could not find elasticsearch metrics in Prometheus server: No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]]

The elasticsearch pods' metrics are exposed.

es pod logs:
$ oc exec elasticsearch-cdm-9cn7xpjg-1-7f7d47ddfc-zzcwc -- logs
Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-9cn7xpjg-1-7f7d47ddfc-zzcwc -n openshift-logging' to see all of the containers in this pod.
[2019-07-03T03:01:32,936][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] initializing ...
[2019-07-03T03:01:33,008][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-cdm-9cn7xpjg-1] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/nvme1n1)]], net usable_space [9.7gb], net total_space [9.7gb], spins? [possibly], types [ext4]
[2019-07-03T03:01:33,008][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-cdm-9cn7xpjg-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-07-03T03:01:33,010][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] node name [elasticsearch-cdm-9cn7xpjg-1], node ID [oXsX3Ig8T_qdKifM-RyRXw]
[2019-07-03T03:01:33,010][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] version[5.6.13], pid[1], build[921f2bd/2018-11-16T16:58:29.974Z], OS[Linux/4.18.0-80.4.2.el8_0.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b04]
[2019-07-03T03:01:33,010][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -Dmapper.allow_dots_in_name=true, -Xms1024m, -Xmx1024m, -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof, -Dsg.display_lic_none=false, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.type=unpooled, -Djdk.tls.ephemeralDHKeySize=2048, -Des.path.home=/usr/share/elasticsearch]
[2019-07-03T03:01:33,719][INFO ][c.f.s.SearchGuardPlugin  ] Clustername: elasticsearch
[2019-07-03T03:01:33,719][WARN ][c.f.s.SearchGuardPlugin  ] 
### LICENSE NOTICE Search Guard ###

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging

In case of any doubt mail to <sales>
###################################
[2019-07-03T03:01:33,720][WARN ][c.f.s.SearchGuardPlugin  ] Consider setting -Djdk.tls.rejectClientInitiatedRenegotiation=true to prevent DoS attacks through client side initiated TLS renegotiation.
[2019-07-03T03:01:33,743][INFO ][c.f.s.SearchGuardPlugin  ] Node [elasticsearch-cdm-9cn7xpjg-1] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2019-07-03T03:01:33,743][INFO ][c.f.s.SearchGuardPlugin  ] FLS/DLS module not available
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_212
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64
[2019-07-03T03:01:33,753][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 4.18.0-80.4.2.el8_0.x86_64
[2019-07-03T03:01:33,836][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 43 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2019-07-03T03:01:33,839][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 43 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2019-07-03T03:01:33,839][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2019-07-03T03:01:33,842][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: elasticsearch
[2019-07-03T03:01:33,876][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] HTTPS client auth mode OPTIONAL
[2019-07-03T03:01:33,876][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the first one: logging-es
[2019-07-03T03:01:33,879][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2019-07-03T03:01:33,879][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2019-07-03T03:01:33,880][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2019-07-03T03:01:33,906][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [aggs-matrix-stats]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [ingest-common]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-expression]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-groovy]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-mustache]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [lang-painless]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [parent-join]
[2019-07-03T03:01:33,909][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [percolator]
[2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [reindex]
[2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [transport-netty3]
[2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded module [transport-netty4]
[2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded plugin [openshift-elasticsearch]
[2019-07-03T03:01:33,910][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-9cn7xpjg-1] loaded plugin [prometheus-exporter]
[2019-07-03T03:01:34,818][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin io.fabric8.elasticsearch.plugin.OpenShiftElasticSearchPlugin
[2019-07-03T03:01:34,848][INFO ][i.f.e.p.PluginSettings   ] Using kibanaIndexMode: 'shared_ops'
[2019-07-03T03:01:34,849][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.app from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
[2019-07-03T03:01:34,852][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.ops from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
[2019-07-03T03:01:34,854][INFO ][i.f.e.p.k.IndexMappingLoader] Trying to load Kibana mapping for io.fabric8.elasticsearch.kibana.mapping.empty from plugin: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json
[2019-07-03T03:01:34,862][INFO ][i.f.e.p.OpenshiftRequestContextFactory] Using kibanaIndexMode: 'shared_ops'
[2019-07-03T03:01:34,897][INFO ][c.f.s.SearchGuardPlugin  ] FLS/DLS valve not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.DlsFlsValveImpl
[2019-07-03T03:01:34,898][INFO ][c.f.s.SearchGuardPlugin  ] Auditlog not available due to java.lang.ClassNotFoundException: com.floragunn.searchguard.auditlog.impl.AuditLogImpl
[2019-07-03T03:01:34,899][INFO ][c.f.s.SearchGuardPlugin  ] Privileges interceptor not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl
[2019-07-03T03:01:34,987][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-cdm-9cn7xpjg-1] using discovery type [zen]
[2019-07-03T03:01:35,327][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] initialized
[2019-07-03T03:01:35,327][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] starting ...
[2019-07-03T03:01:35,416][INFO ][o.e.t.TransportService   ] [elasticsearch-cdm-9cn7xpjg-1] publish_address {10.128.2.21:9300}, bound_addresses {[::]:9300}
[2019-07-03T03:01:35,423][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-cdm-9cn7xpjg-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-03T03:01:35,428][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if .searchguard index exists ...
[2019-07-03T03:01:35,432][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [elasticsearch-cdm-9cn7xpjg-1] no known master node, scheduling a retry
[2019-07-03T03:01:38,460][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:41,461][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:44,463][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:47,464][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:50,466][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:53,467][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:56,470][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:01:59,471][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:02,472][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:05,444][WARN ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] timed out while waiting for initial discovery state - timeout: 30s
[2019-07-03T03:02:05,451][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [elasticsearch-cdm-9cn7xpjg-1] publish_address {10.128.2.21:9200}, bound_addresses {[::]:9200}
[2019-07-03T03:02:05,451][INFO ][o.e.n.Node               ] [elasticsearch-cdm-9cn7xpjg-1] started
[2019-07-03T03:02:05,474][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:08,476][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:11,477][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:14,478][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:17,479][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:20,481][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:23,482][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:26,483][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:29,484][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:32,486][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:35,436][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [elasticsearch-cdm-9cn7xpjg-1] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[2019-07-03T03:02:35,437][ERROR][c.f.s.c.IndexBaseConfigurationRepository] Failure while checking MasterNotDiscoveredException[null] index .searchguard
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:209) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:311) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:238) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
	at org.elasticsearch.cluster.service.ClusterService$NotifyTimeout.run(ClusterService.java:1056) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:576) [elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
[2019-07-03T03:02:35,442][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elasticsearch-cdm-9cn7xpjg-1] no known master node, scheduling a retry
[2019-07-03T03:02:35,487][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:38,488][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:41,489][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:44,491][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:47,492][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-cdm-9cn7xpjg-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-cdm-9cn7xpjg-1}{oXsX3Ig8T_qdKifM-RyRXw}{BRHli7fyRdSMZNzYX6RzmQ}{10.128.2.21}{10.128.2.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-07-03T03:02:52,205][INFO ][o.e.c.s.ClusterService   ] [elasticsearch-cdm-9cn7xpjg-1] detected_master {elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300}, added {{elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300},}, reason: zen-disco-receive(from master [master {elasticsearch-cdm-9cn7xpjg-2}{G59Y2r4yQQCgwkBUoOjrHg}{ObHSz9SNS_Wy-a6caag6Xg}{10.129.2.20}{10.129.2.20:9300} committed version [1]])
[2019-07-03T03:02:55,588][ERROR][c.f.s.c.ConfigurationLoader] No data for config while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups]  (index=.searchguard)
[2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for roles while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups]  (index=.searchguard)
[2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for rolesmapping while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups]  (index=.searchguard)
[2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for internalusers while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups]  (index=.searchguard)
[2019-07-03T03:02:55,589][ERROR][c.f.s.c.ConfigurationLoader] No data for actiongroups while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups]  (index=.searchguard)
[2019-07-03T03:03:15,478][WARN ][c.f.s.c.PrivilegesEvaluator] .kibana does not exist in cluster metadata
[2019-07-03T03:03:55,572][WARN ][c.f.s.c.IndexBaseConfigurationRepository] Timeout, we just try again in a few seconds ... 
[2019-07-03T03:03:55,581][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'elasticsearch-cdm-9cn7xpjg-1' initialized
[2019-07-03T03:06:32,009][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]]
[2019-07-03T03:06:32,009][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {}
[2019-07-03T03:06:32,767][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]]
[2019-07-03T03:06:32,767][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {}
[2019-07-03T03:07:01,287][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]]
[2019-07-03T03:07:01,287][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {}
[2019-07-03T03:07:02,760][INFO ][c.f.s.c.PrivilegesEvaluator] No cluster-level perm match for User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac]] [IndexType [index=_all, type=*]] [Action [[cluster:monitor/prometheus/metrics]]] [RolesChecked [gen_kibana_647a750f1787408bf50088234ec0edd5a6a9b2ac, gen_user_647a750f1787408bf50088234ec0edd5a6a9b2ac]]
[2019-07-03T03:07:02,760][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {}

Version-Release number of selected component (if applicable):
ose-cluster-logging-operator-v4.2.0-201907021819
ose-logging-elasticsearch5-v4.2.0-201907021819
ose-elasticsearch-operator-v4.2.0-201907021819

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.2.0-0.nightly-2019-07-03-003353   True        False         79m     Cluster version is 4.2.0-0.nightly-2019-07-03-003353

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging via OLM
2. check metrics in prometheus server
3.

Actual results:
could not find elasticsearch metrics in prometheus server

Expected results:
should be able to find elasticsearch metrics in prometheus server

Additional info:
Fluentd and rsyslog metrics can be found in Prometheus server
Comment 1 Qiaoling Tang 2019-07-03 08:18:30 UTC 
Created attachment 1586976 [details]
clusterlogging-bundle
Comment 2 Qiaoling Tang 2019-07-03 08:18:56 UTC 
Created attachment 1586977 [details]
elasticsearch-bundle
Comment 3 Jeff Cantrill 2019-07-18 06:10:38 UTC 
This may be fixed by https://github.com/openshift/origin-aggregated-logging/pull/1695
Comment 4 Jeff Cantrill 2019-07-24 17:33:34 UTC 
Please verify this is still an issue now that https://bugzilla.redhat.com/show_bug.cgi?id=1729297 is verified
Comment 5 Qiaoling Tang 2019-07-25 01:39:24 UTC 
Verified with ose-logging-elasticsearch5-v4.2.0-201907232219
Comment 6 errata-xmlrpc 2019-10-16 06:33:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922