Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1838929

Summary: ES pod CrashLoopBackOff: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip]
Product: OpenShift Container Platform Reporter: Qiaoling Tang <qitang>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, mifiedle
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:41:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1841892    
Bug Blocks:    

Description Qiaoling Tang 2020-05-22 05:13:37 UTC 
Description of problem:
The ES pod stuck in CrashLoopBackOff status:

$ oc logs -c elasticsearch elasticsearch-cdm-sfz7erjz-1-59559cf7c6-qlvkq 
[2020-05-22 04:53:04,551][INFO ][container.run            ] Begin Elasticsearch startup script
[2020-05-22 04:53:04,555][INFO ][container.run            ] Comparing the specified RAM to the maximum recommended for Elasticsearch...
[2020-05-22 04:53:04,556][INFO ][container.run            ] Inspecting the maximum RAM available...
[2020-05-22 04:53:04,559][INFO ][container.run            ] ES_JAVA_OPTS: ' -Xms512m -Xmx512m'
[2020-05-22 04:53:04,560][INFO ][container.run            ] Copying certs from /etc/openshift/elasticsearch/secret to /etc/elasticsearch//secret
[2020-05-22 04:53:04,565][INFO ][container.run            ] Building required jks files and truststore
Importing keystore /etc/elasticsearch//secret/admin.p12 to /etc/elasticsearch//secret/admin.jks...
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12".
Certificate was added to keystore

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12".
Importing keystore /etc/elasticsearch//secret/elasticsearch.p12 to /etc/elasticsearch//secret/elasticsearch.jks...
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12".
Certificate was added to keystore

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12".
Importing keystore /etc/elasticsearch//secret/logging-es.p12 to /etc/elasticsearch//secret/logging-es.jks...
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12".

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12".
Certificate was added to keystore

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12".
Certificate was added to keystore
Certificate was added to keystore
[2020-05-22 04:53:06,364][INFO ][container.run            ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof
[2020-05-22 04:53:06,365][INFO ][container.run            ] Checking if Elasticsearch is ready
[2020-05-22 04:53:06,366][INFO ][container.run            ] ES_JAVA_OPTS: ' -Xms512m -Xmx512m -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof -Xloggc:/elasticsearch/persistent/elasticsearch/logs/gc.log -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2020-05-22T04:53:07,862][WARN ][o.e.c.l.LogConfigurator  ] [elasticsearch-cdm-sfz7erjz-1] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /etc/elasticsearch/log4j2.properties
[2020-05-22T04:53:08,099][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-cdm-sfz7erjz-1] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/mapper/coreos-luks-root-nocrypt)]], net usable_space [68.1gb], net total_space [79.4gb], types [xfs]
[2020-05-22T04:53:08,099][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-cdm-sfz7erjz-1] heap size [503.6mb], compressed ordinary object pointers [true]
[2020-05-22T04:53:08,100][INFO ][o.e.n.Node               ] [elasticsearch-cdm-sfz7erjz-1] node name [elasticsearch-cdm-sfz7erjz-1], node ID [emNzsEq7RoO5nL2GB3BQkw]
[2020-05-22T04:53:08,101][INFO ][o.e.n.Node               ] [elasticsearch-cdm-sfz7erjz-1] version[6.8.1-SNAPSHOT], pid[1], build[oss/zip/Unknown/Unknown], OS[Linux/4.18.0-147.8.1.el8_1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_252/25.252-b09]
[2020-05-22T04:53:08,101][INFO ][o.e.n.Node               ] [elasticsearch-cdm-sfz7erjz-1] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7079938385813671367, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -Xms512m, -Xmx512m, -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof, -Xloggc:/elasticsearch/persistent/elasticsearch/logs/gc.log, -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log, -Djdk.tls.ephemeralDHKeySize=2048, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=zip]
[2020-05-22T04:53:08,101][WARN ][o.e.n.Node               ] [elasticsearch-cdm-sfz7erjz-1] version [6.8.1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2020-05-22T04:53:08,809][INFO ][o.e.p.p.PrometheusExporterPlugin] [elasticsearch-cdm-sfz7erjz-1] starting Prometheus exporter plugin
[2020-05-22T04:53:08,997][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] ES Config path is /etc/elasticsearch
[2020-05-22T04:53:09,049][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2020-05-22T04:53:09,115][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2020-05-22T04:53:09,118][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-sfz7erjz-1] No alias given, use the first one: elasticsearch
[2020-05-22T04:53:09,133][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] HTTPS client auth mode OPTIONAL
[2020-05-22T04:53:09,134][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-sfz7erjz-1] No alias given, use the first one: logging-es
[2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS Transport Client Provider : JDK
[2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS Transport Server Provider : JDK
[2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS HTTP Provider             : JDK
[2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2]
[2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Enabled TLS protocols for HTTP layer      : [TLSv1.1, TLSv1.2]
[2020-05-22T04:53:09,405][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Clustername: elasticsearch
[2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch has insecure file permissions (should be 0700)
[2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch/scripts has insecure file permissions (should be 0700)
[2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch/secret has insecure file permissions (should be 0700)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/admin.p12 has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/admin.jks has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/elasticsearch.p12 has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/searchguard.key has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/logging-es.p12 has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/key has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/truststore has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/searchguard.truststore has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/index_settings has insecure file permissions (should be 0600)
[2020-05-22T04:53:09,456][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [aggs-matrix-stats]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [analysis-common]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-common]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-geoip]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-user-agent]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-expression]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-mustache]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-painless]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [mapper-extras]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [parent-join]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [percolator]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [rank-eval]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [reindex]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [repository-url]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [transport-netty4]
[2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded module [tribe]
[2020-05-22T04:53:09,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded plugin [opendistro_security]
[2020-05-22T04:53:09,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-cdm-sfz7erjz-1] loaded plugin [prometheus-exporter]
[2020-05-22T04:53:09,474][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml
[2020-05-22T04:53:11,359][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-cdm-sfz7erjz-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:101) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6]
	... 6 more


EO log:
time="2020-05-22T04:55:48Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: dial tcp 172.30.161.35:9200: i/o timeout"
time="2020-05-22T04:55:52Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:56:18Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout"
{"level":"error","ts":1590123378.571056,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2020-05-22T04:56:23Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:56:53Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:57:19Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.161.35:9200: i/o timeout\r\n"
time="2020-05-22T04:57:23Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:57:53Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:58:24Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:58:55Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:59:25Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T04:59:55Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T05:00:20Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: dial tcp 172.30.161.35:9200: i/o timeout"
time="2020-05-22T05:00:25Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T05:00:50Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout"
{"level":"error","ts":1590123650.069007,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2020-05-22T05:00:56Z" level=info msg="Kibana status successfully updated"
time="2020-05-22T05:01:26Z" level=info msg="Kibana status successfully updated"

Version-Release number of selected component (if applicable):
$ oc get csv
NAME                                        DISPLAY                  VERSION              REPLACES   PHASE
clusterlogging.4.5.0-202005201941           Cluster Logging          4.5.0-202005201941              Succeeded
elasticsearch-operator.4.5.0-202005210017   Elasticsearch Operator   4.5.0-202005210017              Succeeded


How reproducible:
Always

Steps to Reproduce:
1. Deploy EO and CLO
2. create clusterlogging intance
3.

Actual results:


Expected results:


Additional info:
no such issue in quay.io/openshift/origin-logging-elasticsearch6:latest and quay.io/openshift/origin-elasticsearch-operator:latest
Comment 1 Jeff Cantrill 2020-05-22 13:05:24 UTC 
Possible solution: https://discuss.elastic.co/t/6-7-docker-crashes-on-ingest-geoip-expected-database-geolite2-asn-mmdb-to-not-exist-in-usr-share-elasticsearch-config-ingest-geoip/174125/4

We may need this as an impetus to ensure we are using the latest 6.x release
Comment 5 Anping Li 2020-05-29 03:51:08 UTC 
The fix wasn't in the latest elasticsearch6 images.

#oc get csv
NAME                                        DISPLAY                  VERSION              REPLACES                                    PHASE
elasticsearch-operator.4.5.0-202005290037   Elasticsearch Operator   4.5.0-202005290037   elasticsearch-operator.4.4.0-202005180840   Succeeded
$ oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-logging-elasticsearch6@sha256:1d2c67ad5a6bbebfc4d44c6e943b3c1727cb33731f67c35e69d4436ff8b46774 --filter-by-os linux/amd64 | grep 'release='
               release=202005201941
$ oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-logging-elasticsearch6:v4.5.0 --filter-by-os linux/amd64 | grep 'release='
               release=202005201941
Comment 7 Anping Li 2020-05-29 09:36:50 UTC 
Move to assigned as no ES6 downstream images.
Comment 9 Anping Li 2020-05-30 12:25:40 UTC 
Verified by csv:clusterlogging.4.5.0-202005291637 and elasticsearch-operator.4.5.0-202005291637
Comment 10 errata-xmlrpc 2020-07-13 17:41:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409