Description of problem: The ES pod stuck in CrashLoopBackOff status: $ oc logs -c elasticsearch elasticsearch-cdm-sfz7erjz-1-59559cf7c6-qlvkq [2020-05-22 04:53:04,551][INFO ][container.run ] Begin Elasticsearch startup script [2020-05-22 04:53:04,555][INFO ][container.run ] Comparing the specified RAM to the maximum recommended for Elasticsearch... [2020-05-22 04:53:04,556][INFO ][container.run ] Inspecting the maximum RAM available... [2020-05-22 04:53:04,559][INFO ][container.run ] ES_JAVA_OPTS: ' -Xms512m -Xmx512m' [2020-05-22 04:53:04,560][INFO ][container.run ] Copying certs from /etc/openshift/elasticsearch/secret to /etc/elasticsearch//secret [2020-05-22 04:53:04,565][INFO ][container.run ] Building required jks files and truststore Importing keystore /etc/elasticsearch//secret/admin.p12 to /etc/elasticsearch//secret/admin.jks... Entry for alias 1 successfully imported. Import command completed: 1 entries successfully imported, 0 entries failed or cancelled Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12". Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12". Certificate was added to keystore Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/admin.jks -destkeystore /etc/elasticsearch//secret/admin.jks -deststoretype pkcs12". Importing keystore /etc/elasticsearch//secret/elasticsearch.p12 to /etc/elasticsearch//secret/elasticsearch.jks... Entry for alias 1 successfully imported. Import command completed: 1 entries successfully imported, 0 entries failed or cancelled Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12". Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12". Certificate was added to keystore Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/elasticsearch.jks -destkeystore /etc/elasticsearch//secret/elasticsearch.jks -deststoretype pkcs12". Importing keystore /etc/elasticsearch//secret/logging-es.p12 to /etc/elasticsearch//secret/logging-es.jks... Entry for alias 1 successfully imported. Import command completed: 1 entries successfully imported, 0 entries failed or cancelled Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12". Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12". Certificate was added to keystore Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/elasticsearch//secret/logging-es.jks -destkeystore /etc/elasticsearch//secret/logging-es.jks -deststoretype pkcs12". Certificate was added to keystore Certificate was added to keystore [2020-05-22 04:53:06,364][INFO ][container.run ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof [2020-05-22 04:53:06,365][INFO ][container.run ] Checking if Elasticsearch is ready [2020-05-22 04:53:06,366][INFO ][container.run ] ES_JAVA_OPTS: ' -Xms512m -Xmx512m -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof -Xloggc:/elasticsearch/persistent/elasticsearch/logs/gc.log -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log' OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N [2020-05-22T04:53:07,862][WARN ][o.e.c.l.LogConfigurator ] [elasticsearch-cdm-sfz7erjz-1] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations: /etc/elasticsearch/log4j2.properties [2020-05-22T04:53:08,099][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-sfz7erjz-1] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/mapper/coreos-luks-root-nocrypt)]], net usable_space [68.1gb], net total_space [79.4gb], types [xfs] [2020-05-22T04:53:08,099][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-sfz7erjz-1] heap size [503.6mb], compressed ordinary object pointers [true] [2020-05-22T04:53:08,100][INFO ][o.e.n.Node ] [elasticsearch-cdm-sfz7erjz-1] node name [elasticsearch-cdm-sfz7erjz-1], node ID [emNzsEq7RoO5nL2GB3BQkw] [2020-05-22T04:53:08,101][INFO ][o.e.n.Node ] [elasticsearch-cdm-sfz7erjz-1] version[6.8.1-SNAPSHOT], pid[1], build[oss/zip/Unknown/Unknown], OS[Linux/4.18.0-147.8.1.el8_1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_252/25.252-b09] [2020-05-22T04:53:08,101][INFO ][o.e.n.Node ] [elasticsearch-cdm-sfz7erjz-1] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7079938385813671367, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -Xms512m, -Xmx512m, -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof, -Xloggc:/elasticsearch/persistent/elasticsearch/logs/gc.log, -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log, -Djdk.tls.ephemeralDHKeySize=2048, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=zip] [2020-05-22T04:53:08,101][WARN ][o.e.n.Node ] [elasticsearch-cdm-sfz7erjz-1] version [6.8.1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production [2020-05-22T04:53:08,809][INFO ][o.e.p.p.PrometheusExporterPlugin] [elasticsearch-cdm-sfz7erjz-1] starting Prometheus exporter plugin [2020-05-22T04:53:08,997][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] ES Config path is /etc/elasticsearch [2020-05-22T04:53:09,049][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2020-05-22T04:53:09,115][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively [2020-05-22T04:53:09,118][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-sfz7erjz-1] No alias given, use the first one: elasticsearch [2020-05-22T04:53:09,133][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] HTTPS client auth mode OPTIONAL [2020-05-22T04:53:09,134][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-sfz7erjz-1] No alias given, use the first one: logging-es [2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS Transport Client Provider : JDK [2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS Transport Server Provider : JDK [2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] TLS HTTP Provider : JDK [2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2] [2020-05-22T04:53:09,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-sfz7erjz-1] Enabled TLS protocols for HTTP layer : [TLSv1.1, TLSv1.2] [2020-05-22T04:53:09,405][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Clustername: elasticsearch [2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch has insecure file permissions (should be 0700) [2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch/scripts has insecure file permissions (should be 0700) [2020-05-22T04:53:09,436][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Directory /etc/elasticsearch/secret has insecure file permissions (should be 0700) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/admin.p12 has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/admin.jks has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/elasticsearch.p12 has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/searchguard.key has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/logging-es.p12 has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/key has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/truststore has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/secret/searchguard.truststore has insecure file permissions (should be 0600) [2020-05-22T04:53:09,437][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] File /etc/elasticsearch/index_settings has insecure file permissions (should be 0600) [2020-05-22T04:53:09,456][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [aggs-matrix-stats] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [analysis-common] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-common] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-geoip] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [ingest-user-agent] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-expression] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-mustache] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [lang-painless] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [mapper-extras] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [parent-join] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [percolator] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [rank-eval] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [reindex] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [repository-url] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [transport-netty4] [2020-05-22T04:53:09,457][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded module [tribe] [2020-05-22T04:53:09,458][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded plugin [opendistro_security] [2020-05-22T04:53:09,458][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-sfz7erjz-1] loaded plugin [prometheus-exporter] [2020-05-22T04:53:09,474][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-sfz7erjz-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2020-05-22T04:53:11,359][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-cdm-sfz7erjz-1] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip] at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?] at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] ... 6 more Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to exist in [/usr/share/elasticsearch/modules/ingest-geoip] at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?] at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:101) ~[?:?] at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?] at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.redhat-6.jar:6.8.1.redhat-6] ... 6 more EO log: time="2020-05-22T04:55:48Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: dial tcp 172.30.161.35:9200: i/o timeout" time="2020-05-22T04:55:52Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:56:18Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout" {"level":"error","ts":1590123378.571056,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2020-05-22T04:56:23Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:56:53Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:57:19Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.161.35:9200: i/o timeout\r\n" time="2020-05-22T04:57:23Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:57:53Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:58:24Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:58:55Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:59:25Z" level=info msg="Kibana status successfully updated" time="2020-05-22T04:59:55Z" level=info msg="Kibana status successfully updated" time="2020-05-22T05:00:20Z" level=warning msg="Unable to list existing templates in order to reconcile stale ones: Get https://elasticsearch.openshift-logging.svc:9200/_template: dial tcp 172.30.161.35:9200: i/o timeout" time="2020-05-22T05:00:25Z" level=info msg="Kibana status successfully updated" time="2020-05-22T05:00:50Z" level=error msg="Error creating index template for mapping app: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout" {"level":"error","ts":1590123650.069007,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"elasticsearch-controller","request":"openshift-logging/elasticsearch","error":"Failed to reconcile IndexMangement for Elasticsearch cluster: Put https://elasticsearch.openshift-logging.svc:9200/_template/ocp-gen-app: dial tcp 172.30.161.35:9200: i/o timeout","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2020-05-22T05:00:56Z" level=info msg="Kibana status successfully updated" time="2020-05-22T05:01:26Z" level=info msg="Kibana status successfully updated" Version-Release number of selected component (if applicable): $ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterlogging.4.5.0-202005201941 Cluster Logging 4.5.0-202005201941 Succeeded elasticsearch-operator.4.5.0-202005210017 Elasticsearch Operator 4.5.0-202005210017 Succeeded How reproducible: Always Steps to Reproduce: 1. Deploy EO and CLO 2. create clusterlogging intance 3. Actual results: Expected results: Additional info: no such issue in quay.io/openshift/origin-logging-elasticsearch6:latest and quay.io/openshift/origin-elasticsearch-operator:latest
Possible solution: https://discuss.elastic.co/t/6-7-docker-crashes-on-ingest-geoip-expected-database-geolite2-asn-mmdb-to-not-exist-in-usr-share-elasticsearch-config-ingest-geoip/174125/4 We may need this as an impetus to ensure we are using the latest 6.x release
The fix wasn't in the latest elasticsearch6 images. #oc get csv NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.4.5.0-202005290037 Elasticsearch Operator 4.5.0-202005290037 elasticsearch-operator.4.4.0-202005180840 Succeeded $ oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-logging-elasticsearch6@sha256:1d2c67ad5a6bbebfc4d44c6e943b3c1727cb33731f67c35e69d4436ff8b46774 --filter-by-os linux/amd64 | grep 'release=' release=202005201941 $ oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-logging-elasticsearch6:v4.5.0 --filter-by-os linux/amd64 | grep 'release=' release=202005201941
Move to assigned as no ES6 downstream images.
Verified by csv:clusterlogging.4.5.0-202005291637 and elasticsearch-operator.4.5.0-202005291637
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409