Description of problem: EFK stack installed on IBM Power-based machines has logging-es fail with unsupported architecture Version-Release number of selected component (if applicable): 3.11 How reproducible: Unconfirmed Actual results: [2020-02-24T20:58:33,393][INFO ][i.f.e.p.OpenshiftRequestContextFactory] Using kibanaIndexMode: 'unique' [2020-02-24T20:58:33,447][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS valve not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.DlsFlsValveImpl [2020-02-24T20:58:33,448][INFO ][c.f.s.SearchGuardPlugin ] Auditlog not available due to java.lang.ClassNotFoundException: com.floragunn.searchguard.auditlog.impl.AuditLogImpl [2020-02-24T20:58:33,449][INFO ][c.f.s.SearchGuardPlugin ] Privileges interceptor not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl [2020-02-24T20:58:33,518][INFO ][o.e.d.DiscoveryModule ] [logging-es-data-master-m85sak2x] using discovery type [zen] [2020-02-24T20:58:34,130][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] initialized [2020-02-24T20:58:34,130][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] starting ... [2020-02-24T20:58:34,346][INFO ][o.e.t.TransportService ] [logging-es-data-master-m85sak2x] publish_address {10.2.4.116:9300}, bound_addresses {[::]:9300} [2020-02-24T20:58:34,355][INFO ][o.e.b.BootstrapChecks ] [logging-es-data-master-m85sak2x] bound or publishing to a non-loopback address, enforcing bootstrap checks [2020-02-24T20:58:34,357][ERROR][o.e.b.Bootstrap ] [logging-es-data-master-m85sak2x] node validation exception [1] bootstrap checks failed [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk [2020-02-24T20:58:34,358][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] stopping ... [2020-02-24T20:58:34,723][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] stopped [2020-02-24T20:58:34,723][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] closing ... [2020-02-24T20:58:34,735][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] closed [2020-02-24T20:58:37,308][WARN ][o.e.b.JNANatives ] unable to install syscall filter: java.lang.UnsupportedOperationException: seccomp unavailable: 'ppc64le' architecture unsupported at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:266) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:258) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.16.redhat-2.jar:5.6.16.redhat-2] [2020-02-24T20:58:37,482][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] initializing ... [2020-02-24T20:58:37,561][INFO ][o.e.e.NodeEnvironment ] [logging-es-data-master-m85sak2x] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/mapper/mpathr)]], net usable_space [37.1gb], net total_space [39.2gb], spins? [possibly], types [ext4] [2020-02-24T20:58:37,561][INFO ][o.e.e.NodeEnvironment ] [logging-es-data-master-m85sak2x] heap size [3.9gb], compressed ordinary object pointers [true] [2020-02-24T20:58:37,562][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] node name [logging-es-data-master-m85sak2x], node ID [eStC9bwpTjejfov_rIse3Q] [2020-02-24T20:58:37,562][INFO ][o.e.n.Node ] [logging-es-data-master-m85sak2x] version[5.6.16], pid[1], build[8dc130e/2019-09-10T20:07:09.564Z], OS[Linux/3.10.0-1062.12.1.el7.ppc64le/ppc64le], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08] Expected results: Additional info:
According to the following upstream discussion, this is simply a warning: https://discuss.elastic.co/t/java-lang-unsupportedoperationexception-on-starting-elasticsearch/64548/2 The following blog post discusses how this could be fixed: https://www.clarenceho.net/2016/06/adding-seccomp-support-to-elasticsearch.html Although please note that the relevant code has since been moved multiple times: https://github.com/elastic/elasticsearch/blob/5.6/core/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java#L239 https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java#L238 I cannot say as to how far back upstream would take backports, or if we would carry a backport as a patch in our builds, but the first step would be to get upstream master fixed (and please consider fixing s390x too while you're at it).
Could someone please verify the following: --- a/server/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java +++ b/server/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java @@ -241,6 +241,8 @@ final class SystemCallFilter { ARCHITECTURES = Map.of( "amd64", new Arch(0xC000003E, 0x3FFFFFFF, 57, 58, 59, 322, 317), "aarch64", new Arch(0xC00000B7, 0xFFFFFFFF, 1079, 1071, 221, 281, 277)); + "ppc64le", new Arch(0xC00000A5, 0xFFFFFFFF, 2, 189, 11, 362, 358)); + "s390x", new Arch(0x800000A6, 0x3FFFFFFF, 2, 190, 11, 354, 348)); } /** invokes prctl() from linux libc library */ --
Hi Yaakov, The above change mentioned by you is required to build elasticsearch on Power. Please find the below reference link to build elasticsearch on Power.(validated on Power PC) https://github.com/ppc64le/build-scripts/blob/master/elasticsearch/elasticsearch_7.4_rhel_7.6.sh
Fixing typos: --- a/server/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java +++ b/server/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java @@ -241,6 +241,8 @@ final class SystemCallFilter { ARCHITECTURES = Map.of( "amd64", new Arch(0xC000003E, 0x3FFFFFFF, 57, 58, 59, 322, 317), "aarch64", new Arch(0xC00000B7, 0xFFFFFFFF, 1079, 1071, 221, 281, 277)); + "ppc64le", new Arch(0xC0000015, 0xFFFFFFFF, 2, 189, 11, 362, 358)); + "s390x", new Arch(0x80000016, 0x3FFFFFFF, 2, 190, 11, 354, 348)); } /** invokes prctl() from linux libc library */ --
Submitted upstream PR: https://github.com/elastic/elasticsearch/pull/53239
Upstream has rejected this patch outright because they do not want to even appear to claim support for architectures other than x86_64. Therefore, if we're going to fix this, it will have to be downstream only. Jeff, if I provide the patches for review, would we be willing to take the patch and carry it in our future builds of the elasticsearch jar?
Moving to the multiarch team as I understand they are the ones responsible for testing and ensuring support.
*** Bug 1842412 has been marked as a duplicate of this bug. ***
I had a customer testing this in a cluster on Z (s390x) with OCP 4.4. $ oc patch elasticsearch.logging.openshift.io/elasticsearch --type merge --patch '{"spec":{"managementState": "Unmanaged"}}' $ oc edit cm elasticsearch ------------------------------------------------------- apiVersion: v1 data: elasticsearch.yml: |2- bootstrap.system_call_filter: false cluster: name: ${CLUSTER_NAME} ------------------------------------------------------- After this change, ES fails to start with the following message ~~~ [2020-06-29T09:48:43,603][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [elasticsearch-cdm-1cb87h5k-1] SSL Problem Received close_notify during handshake javax.net.ssl.SSLException: Received close_notify during handshake at sun.security.ssl.Alerts.getSSLException(Alerts.java:214) ~[?:?] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1667) ~[?:?] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1635) ~[?:?] at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1777) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1090) ~[?:?] at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:913) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:783) ~[?:?] ~~~ Anyone knows if this is related to the bootstrap checks mentioned in comment #16?
Hi, do you think we can get full ES node log? Lukáš
@Lukas, Hi, will share the logs within bug 1854472
Created attachment 1701284 [details] openshift-ansible 3.11 patch
Steven, if you could please take attachment 1701284 [details] back to the customer on 3.11 and see if redeploying with that applied on top of openshift-ansible solves their issue?
Hi, The patch seems to have worked allowing the playbook to finish! Thanks!
@Doug - let me know if there is anything I can do to help you with this one.
The fix for this in openshift-ansible appears to have worked using version OCP 3.11.286. Here are the results - http://pastebin.test.redhat.com/902499
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4170