Bug 1628788
| Summary: | [3.11] Fluentd pod crashes with "undefined symbol: rbffi_Closure_Alloc" | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> | |
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.11.0 | CC: | aos-bugs, jcantril, nhosoi, rmeggins | |
| Target Milestone: | --- | Keywords: | Regression | |
| Target Release: | 3.11.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openshift3/ose-logging-fluentd:v3.11.6-1 | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The fix for bz1628371 introduced a badly built shared library missing a symbol.
Consequence: Fluentd crashes with "undefined symbol: rbffi_Closure_Alloc"
Fix: Rebuild the shared library with the correct symbols.
Result: Fluentd does not crash.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1628798 (view as bug list) | Environment: | ||
| Last Closed: | 2018-10-11 07:25:56 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1628798, 1628799 | |||
Likely a dup of "Bug 1628407 - [3.11] Fluentd pods failed to start after an update to 3.9.41 when deny_execmem=1 on nodes"? It's a consequence of trying to revert the revert :P fixed in rubygem-ffi-1.9.25-4.el7_5 - now awaiting ART to rebuild compose and image We talked about this one and determined it could affect a number of people who have the selinux setting as identified in #c1. This can wait to get into the first z-stream drop. (In reply to Jeff Cantrill from comment #4) > We talked about this one and determined it could affect a number of people > who have the selinux setting as identified in #c1. This can wait to get > into the first z-stream drop. You're talking about "Bug 1628407 - [3.11] Fluentd pods failed to start after an update to 3.9.41 when deny_execmem=1 on nodes" Fluentd pod works well with logging-fluentd-v3.11.6-1. Remove keyword 'TestBlocker' Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652 |
Description of problem: Deploy logging on Openshift, check the pods' status under openshift-logging, the logging-fluentd pod failed to start. [root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc get pod NAME READY STATUS RESTARTS AGE logging-es-data-master-ndbb1ikj-1-ll845 2/2 Running 0 9m logging-fluentd-6b6fz 0/1 CrashLoopBackOff 6 10m logging-fluentd-hchfp 0/1 CrashLoopBackOff 6 10m logging-fluentd-jz9m4 0/1 CrashLoopBackOff 6 10m logging-fluentd-mcbzk 0/1 CrashLoopBackOff 6 10m logging-kibana-1-hwm26 2/2 Running 0 11m [root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc logs logging-fluentd-hchfp umounts of dead containers will fail. Ignoring... umount: /var/lib/docker/containers/07a0f08f34aa261b9a28aac5d5d3c26b30ea5d8a497c0e4823d2ca9bee62904b/shm: not mounted umount: /var/lib/docker/containers/0f73eafea5bc260a11310c90199b4127a568e6a69837070cfb32191fcd77527f/shm: not mounted umount: /var/lib/docker/containers/446351086c29793695f43a3b115d1a743c0c7ce45fbdea14aee4db2556ffaf21/shm: not mounted umount: /var/lib/docker/containers/49ef736cfb5e0f47c0751d7c88c4e44df756dd26693fe9515e289ad74477787e/shm: not mounted umount: /var/lib/docker/containers/4f78a626d47c8b56840ab3a564e219796420a2b6cef2f4157699fb5307a16010/shm: not mounted umount: /var/lib/docker/containers/8354c667b0e40a0787b524bad02a4752c92c33a4bf6364600771f09c8f549c79/shm: not mounted umount: /var/lib/docker/containers/b69c450a4d772e79715d7c906dcbc46caccd1cf2fc58490a2d05d3e96db8bd0c/shm: not mounted umount: /var/lib/docker/containers/b8d2372278e09f6959fff0df1d08107341b15b640bd3591e32b243db96e6905f/shm: not mounted umount: /var/lib/docker/containers/bb0fad2806d6797cbc52227ecc677d4e1827a9596b81be23b3e9fcd60a47c777/shm: not mounted umount: /var/lib/docker/containers/c968df3fa99bf92b755702949c2fdf6111553bc6eb581e19cd7974e455b3576e/shm: not mounted umount: /var/lib/docker/containers/e61268c2ebeaa631c83421a5e63ba96ee28c0c65553d0072b0cd840625ce5c38/shm: not mounted 2018-09-13 21:22:43 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" 2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not 2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so: undefined symbol: rbffi_Closure_Alloc - /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:6:in `rescue in <top (required)>' from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:3:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/id128.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal/native.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:2:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:144:in `require' from /usr/share/gems/gems/fluent-plugin-systemd-0.0.11/lib/fluent/plugin/in_systemd.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:173:in `block in try_load_plugin' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `each' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `try_load_plugin' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:130:in `new_impl' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:55:in `new_input' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:154:in `add_source' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:95:in `block in configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `each' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:129:in `configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:103:in `run_configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:489:in `run_configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:174:in `block in start' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `call' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `main_process' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:170:in `start' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/command/fluentd.rb:173:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/fluentd-0.12.43/bin/fluentd:8:in `<top (required)>' from /usr/bin/fluentd:23:in `load' from /usr/bin/fluentd:23:in `<main>' Version-Release number of selected component (if applicable): ose-logging-fluentd:v3.11.4 ansible-2.6.2-1.el7ae.noarch openshift-ansible-playbooks-3.11.4-1.git.0.d727082.el7_5.noarch openshift-ansible-3.11.4-1.git.0.d727082.el7_5.noarch openshift-ansible-roles-3.11.4-1.git.0.d727082.el7_5.noarch How reproducible: Always Steps to Reproduce: 1.Deploy logging on openshift 2. 3. Actual results: fluentd pods failed to start Expected results: all pods could be start and work well Additional info: -e parameters: openshift_logging_install_logging=true openshift_logging_es_cluster_size=1 openshift_logging_elasticsearch_memory_limit=2Gi openshift_logging_elasticsearch_cpu_limit=1000m openshift_logging_es_nodeselector={"role": "node"} openshift_logging_es_allow_external=true