Bug 1628798

Summary: [3.10] Fluentd pod crashes with "undefined symbol: rbffi_Closure_Alloc"
Product: OpenShift Container Platform Reporter: Rich Megginson <rmeggins>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: anli, aos-bugs, juzhao, nhosoi, qitang, rmeggins
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift3/logging-fluentd:v3.10.50-1 Doc Type: Bug Fix
Doc Text:
Cause: The fix for bz1628371 introduced a badly built shared library missing a symbol. Consequence: Fluentd crashes with "undefined symbol: rbffi_Closure_Alloc" Fix: Rebuild the shared library with the correct symbols. Result: Fluentd does not crash.
 Story Points: ---
Clone Of: 1628788
: 1628799 (view as bug list) Environment:
Last Closed: 2018-11-11 16:39:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1628788    
Bug Blocks: 1628799    

Description Rich Megginson 2018-09-14 03:00:40 UTC 
+++ This bug was initially created as a clone of Bug #1628788 +++

Description of problem:
Deploy logging on Openshift, check the pods' status under openshift-logging, the logging-fluentd pod failed to start.

[root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc get pod
NAME                                      READY     STATUS             RESTARTS   AGE
logging-es-data-master-ndbb1ikj-1-ll845   2/2       Running            0          9m
logging-fluentd-6b6fz                     0/1       CrashLoopBackOff   6          10m
logging-fluentd-hchfp                     0/1       CrashLoopBackOff   6          10m
logging-fluentd-jz9m4                     0/1       CrashLoopBackOff   6          10m
logging-fluentd-mcbzk                     0/1       CrashLoopBackOff   6          10m
logging-kibana-1-hwm26                    2/2       Running            0          11m
[root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc logs logging-fluentd-hchfp
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/07a0f08f34aa261b9a28aac5d5d3c26b30ea5d8a497c0e4823d2ca9bee62904b/shm: not mounted
umount: /var/lib/docker/containers/0f73eafea5bc260a11310c90199b4127a568e6a69837070cfb32191fcd77527f/shm: not mounted
umount: /var/lib/docker/containers/446351086c29793695f43a3b115d1a743c0c7ce45fbdea14aee4db2556ffaf21/shm: not mounted
umount: /var/lib/docker/containers/49ef736cfb5e0f47c0751d7c88c4e44df756dd26693fe9515e289ad74477787e/shm: not mounted
umount: /var/lib/docker/containers/4f78a626d47c8b56840ab3a564e219796420a2b6cef2f4157699fb5307a16010/shm: not mounted
umount: /var/lib/docker/containers/8354c667b0e40a0787b524bad02a4752c92c33a4bf6364600771f09c8f549c79/shm: not mounted
umount: /var/lib/docker/containers/b69c450a4d772e79715d7c906dcbc46caccd1cf2fc58490a2d05d3e96db8bd0c/shm: not mounted
umount: /var/lib/docker/containers/b8d2372278e09f6959fff0df1d08107341b15b640bd3591e32b243db96e6905f/shm: not mounted
umount: /var/lib/docker/containers/bb0fad2806d6797cbc52227ecc677d4e1827a9596b81be23b3e9fcd60a47c777/shm: not mounted
umount: /var/lib/docker/containers/c968df3fa99bf92b755702949c2fdf6111553bc6eb581e19cd7974e455b3576e/shm: not mounted
umount: /var/lib/docker/containers/e61268c2ebeaa631c83421a5e63ba96ee28c0c65553d0072b0cd840625ce5c38/shm: not mounted
2018-09-13 21:22:43 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so: undefined symbol: rbffi_Closure_Alloc - /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so (LoadError)
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:6:in `rescue in <top (required)>'
	from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:3:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/id128.rb:1:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal/native.rb:1:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:2:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:144:in `require'
	from /usr/share/gems/gems/fluent-plugin-systemd-0.0.11/lib/fluent/plugin/in_systemd.rb:1:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:173:in `block in try_load_plugin'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `each'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `try_load_plugin'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:130:in `new_impl'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:55:in `new_input'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:154:in `add_source'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:95:in `block in configure'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `each'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `configure'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:129:in `configure'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:103:in `run_configure'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:489:in `run_configure'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:174:in `block in start'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `call'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `main_process'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:170:in `start'
	from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/gems/gems/fluentd-0.12.43/bin/fluentd:8:in `<top (required)>'
	from /usr/bin/fluentd:23:in `load'
	from /usr/bin/fluentd:23:in `<main>'


Version-Release number of selected component (if applicable):

ose-logging-fluentd:v3.11.4

ansible-2.6.2-1.el7ae.noarch
openshift-ansible-playbooks-3.11.4-1.git.0.d727082.el7_5.noarch
openshift-ansible-3.11.4-1.git.0.d727082.el7_5.noarch
openshift-ansible-roles-3.11.4-1.git.0.d727082.el7_5.noarch

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging on openshift
2.
3.

Actual results:
fluentd pods failed to start

Expected results:
all pods could be start and work well

Additional info:
-e parameters:
openshift_logging_install_logging=true
openshift_logging_es_cluster_size=1
openshift_logging_elasticsearch_memory_limit=2Gi
openshift_logging_elasticsearch_cpu_limit=1000m
openshift_logging_es_nodeselector={"role": "node"}
openshift_logging_es_allow_external=true

--- Additional comment from Noriko Hosoi on 2018-09-13 21:45:23 EDT ---

Likely a dup of "Bug 1628407 - [3.11] Fluentd pods failed to start after an update to 3.9.41 when deny_execmem=1 on nodes"?

--- Additional comment from Rich Megginson on 2018-09-13 22:34:36 EDT ---

It's a consequence of trying to revert the revert :P

--- Additional comment from Rich Megginson on 2018-09-13 22:59:45 EDT ---

fixed in rubygem-ffi-1.9.25-4.el7_5 - now awaiting ART to rebuild compose and image
Comment 3 Junqi Zhao 2018-09-30 08:59:46 UTC 
Issue is fixed with logging-fluentd-v3.10.51-1
# oc -n openshift-logging get pod
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-wv427                   1/1       Running   0          1h
logging-es-data-master-sqbxzuzr-1-gc9hr   2/2       Running   0          1h
logging-fluentd-4pr95                     1/1       Running   0          1h
logging-fluentd-kmzvn                     1/1       Running   0          1h
logging-fluentd-rq9xb                     1/1       Running   0          1h
logging-kibana-1-4rgqp                    2/2       Running   0          1h
Comment 5 errata-xmlrpc 2018-11-11 16:39:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2709