Bug 1466626
Summary: | Unable to load index mapping for io.fabric8.elasticsearch.kibana.mapping.empty. | ||||||
---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Xia Zhao <xiazhao> | ||||
Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | ||||
Status: | CLOSED ERRATA | QA Contact: | Xia Zhao <xiazhao> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 3.5.1 | CC: | aos-bugs, dvercill, jcantril, jlee, nnosenzo, pdwyer, rhodzelm, rmeggins, rromerom, xiazhao | ||||
Target Milestone: | --- | Keywords: | Regression | ||||
Target Release: | 3.5.z | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: |
Cause: Property missing from configuration file
Consequence: Elasticsearch fails to start and generates a large stack trace.
Fix: Modify installer code to create configuration with required property
Result: Elasticsearch starts as desired
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1472144 (view as bug list) | Environment: | |||||
Last Closed: | 2017-07-27 18:02:01 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1472144 | ||||||
Attachments: |
|
Description
Xia Zhao
2017-06-30 05:47:37 UTC
You can manually add the config entry per the attached PR to workaround Do you only see this error in 3.5? Do you see it in 3.6? Just wondering if this needs to be a blocker for 3.6. Note that the 3.5 images are way behind upstream . . . we need to update soon . . . fixed github PR link - looks like 3.6 has the correct settings already The work around worked fine, test result was added into https://bugzilla.redhat.com/show_bug.cgi?id=1463046#c5 This issue didn't happen on v3.6 images. Verified with the latest ansible package, bug is fixed, es is able to start up well after the ansible deployment, set to verified: # rpm -qa | grep ansible openshift-ansible-docs-3.5.94-1.git.0.1b33481.el7.noarch openshift-ansible-lookup-plugins-3.5.94-1.git.0.1b33481.el7.noarch ansible-2.2.3.0-1.el7.noarch openshift-ansible-3.5.94-1.git.0.1b33481.el7.noarch openshift-ansible-filter-plugins-3.5.94-1.git.0.1b33481.el7.noarch openshift-ansible-roles-3.5.94-1.git.0.1b33481.el7.noarch openshift-ansible-callback-plugins-3.5.94-1.git.0.1b33481.el7.noarch openshift-ansible-playbooks-3.5.94-1.git.0.1b33481.el7.noarch @Xia, 3.5.94 version is released? The latest version of the package in "rhel-7-server-ose-3.5-rpms/x86_64" is 3.5.91. Is there any other work-around to avoid it? Thanks, Jooho Lee. @jooho lee. Manually edit the logging-elasticsearch configmap to add something like: io.fabric8.elasticsearch.kibana.mapping.empty: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json Thanks @Jeff, I did it and I got new issues. After I add the parameter, I got another issue. There are 3 elasticsearch containers and 2 of them run well at first but the other one could not run with following message: ~~~ 2017-07-25 15:52:00 INFO SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available 2017-07-25 15:52:00 INFO SearchGuardPlugin:58 - Clustername: elasticsearch 2017-07-25 15:52:00 INFO SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNode: false/tribeNodeClient: false 2017-07-25 15:52:00 INFO plugins:180 - [Tom Corsi] modules [], plugins [search-guard-ssl, search-guard2], sites [] 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:423 - Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:173 - Config directory is /usr/share/java/elasticsearch/config/, from there the key- and truststore files are resolved relatively 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:142 - sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:144 - sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256] 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:146 - sslHTTPProvider:null with ciphers [] 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:148 - sslTransport protocols [TLSv1.2, TLSv1.1] 2017-07-25 15:52:00 INFO DefaultSearchGuardKeyStore:149 - sslHTTP protocols [TLSv1.2, TLSv1.1] 2017-07-25 15:52:00 INFO transport:99 - [Tom Corsi] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl] Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ... Cannot retrieve cluster state due to: ClusterService was close during health call. This is not an error, will keep on trying ... * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates) * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) ~~~ Moreover, if i delete the 2 es that was running well, the 2 containers also failed to start up with the same error message. Is it totally different issue from this ticket? Thanks, Jooho lee. @joolee, Please do not use this issue as a catch all. Based on the output I see, there is nothing wrong with your cluster. It looks to be in an initialization state, possibly allocating indicies @Jeff, Ok, I think I found another bug about it so I am going to file a new ticket. Thanks, Jooho Lee. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1810 FYI: I just upgraded my 3.5 AWS cluster to the v3.6 via the official upgrade playbook (all-in-one), and I still had to manually add the io.fabric8.elasticsearch.kibana.mapping.empty: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json |