+++ This bug was initially created as a clone of Bug #1628798 +++ +++ This bug was initially created as a clone of Bug #1628788 +++ Description of problem: Deploy logging on Openshift, check the pods' status under openshift-logging, the logging-fluentd pod failed to start. [root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc get pod NAME READY STATUS RESTARTS AGE logging-es-data-master-ndbb1ikj-1-ll845 2/2 Running 0 9m logging-fluentd-6b6fz 0/1 CrashLoopBackOff 6 10m logging-fluentd-hchfp 0/1 CrashLoopBackOff 6 10m logging-fluentd-jz9m4 0/1 CrashLoopBackOff 6 10m logging-fluentd-mcbzk 0/1 CrashLoopBackOff 6 10m logging-kibana-1-hwm26 2/2 Running 0 11m [root@qe-qitang-311-gce-master-etcd-nfs-1 ~]# oc logs logging-fluentd-hchfp umounts of dead containers will fail. Ignoring... umount: /var/lib/docker/containers/07a0f08f34aa261b9a28aac5d5d3c26b30ea5d8a497c0e4823d2ca9bee62904b/shm: not mounted umount: /var/lib/docker/containers/0f73eafea5bc260a11310c90199b4127a568e6a69837070cfb32191fcd77527f/shm: not mounted umount: /var/lib/docker/containers/446351086c29793695f43a3b115d1a743c0c7ce45fbdea14aee4db2556ffaf21/shm: not mounted umount: /var/lib/docker/containers/49ef736cfb5e0f47c0751d7c88c4e44df756dd26693fe9515e289ad74477787e/shm: not mounted umount: /var/lib/docker/containers/4f78a626d47c8b56840ab3a564e219796420a2b6cef2f4157699fb5307a16010/shm: not mounted umount: /var/lib/docker/containers/8354c667b0e40a0787b524bad02a4752c92c33a4bf6364600771f09c8f549c79/shm: not mounted umount: /var/lib/docker/containers/b69c450a4d772e79715d7c906dcbc46caccd1cf2fc58490a2d05d3e96db8bd0c/shm: not mounted umount: /var/lib/docker/containers/b8d2372278e09f6959fff0df1d08107341b15b640bd3591e32b243db96e6905f/shm: not mounted umount: /var/lib/docker/containers/bb0fad2806d6797cbc52227ecc677d4e1827a9596b81be23b3e9fcd60a47c777/shm: not mounted umount: /var/lib/docker/containers/c968df3fa99bf92b755702949c2fdf6111553bc6eb581e19cd7974e455b3576e/shm: not mounted umount: /var/lib/docker/containers/e61268c2ebeaa631c83421a5e63ba96ee28c0c65553d0072b0cd840625ce5c38/shm: not mounted 2018-09-13 21:22:43 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" 2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not 2018-09-13 21:22:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so: undefined symbol: rbffi_Closure_Alloc - /usr/lib64/gems/ruby/ffi-1.9.25/lib/ffi_c.so (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:6:in `rescue in <top (required)>' from /usr/share/gems/gems/ffi-1.9.25/lib/ffi.rb:3:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/id128.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal/native.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:2:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:144:in `require' from /usr/share/gems/gems/fluent-plugin-systemd-0.0.11/lib/fluent/plugin/in_systemd.rb:1:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:173:in `block in try_load_plugin' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `each' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:170:in `try_load_plugin' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:130:in `new_impl' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin.rb:55:in `new_input' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:154:in `add_source' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:95:in `block in configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `each' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:92:in `configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:129:in `configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/engine.rb:103:in `run_configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:489:in `run_configure' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:174:in `block in start' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `call' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in `main_process' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:170:in `start' from /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/command/fluentd.rb:173:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/fluentd-0.12.43/bin/fluentd:8:in `<top (required)>' from /usr/bin/fluentd:23:in `load' from /usr/bin/fluentd:23:in `<main>' Version-Release number of selected component (if applicable): ose-logging-fluentd:v3.11.4 ansible-2.6.2-1.el7ae.noarch openshift-ansible-playbooks-3.11.4-1.git.0.d727082.el7_5.noarch openshift-ansible-3.11.4-1.git.0.d727082.el7_5.noarch openshift-ansible-roles-3.11.4-1.git.0.d727082.el7_5.noarch How reproducible: Always Steps to Reproduce: 1.Deploy logging on openshift 2. 3. Actual results: fluentd pods failed to start Expected results: all pods could be start and work well Additional info: -e parameters: openshift_logging_install_logging=true openshift_logging_es_cluster_size=1 openshift_logging_elasticsearch_memory_limit=2Gi openshift_logging_elasticsearch_cpu_limit=1000m openshift_logging_es_nodeselector={"role": "node"} openshift_logging_es_allow_external=true --- Additional comment from Noriko Hosoi on 2018-09-13 21:45:23 EDT --- Likely a dup of "Bug 1628407 - [3.11] Fluentd pods failed to start after an update to 3.9.41 when deny_execmem=1 on nodes"? --- Additional comment from Rich Megginson on 2018-09-13 22:34:36 EDT --- It's a consequence of trying to revert the revert :P --- Additional comment from Rich Megginson on 2018-09-13 22:59:45 EDT --- fixed in rubygem-ffi-1.9.25-4.el7_5 - now awaiting ART to rebuild compose and image
Issue is fixed with logging-fluentd-v3.9.47-1 # oc -n logging get pod NAME READY STATUS RESTARTS AGE logging-curator-1-g4qfr 1/1 Running 0 7m logging-es-data-master-wgfvbxuz-1-mb4t7 2/2 Running 0 6m logging-fluentd-c2n4g 1/1 Running 0 6m logging-fluentd-mtfst 1/1 Running 0 6m logging-kibana-1-865dw 2/2 Running 0 7m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2908