Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1637778

Summary: OSP 14 | procevent collectd plugin for Service Assurance fails to load due to "no buffer space available"
Product: Red Hat OpenStack Reporter: Leonid Natapov <lnatapov>
Component: collectdAssignee: Matthias Runge <mrunge>
Status: CLOSED NOTABUG QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: aasmith, abays, apannu, jbadiapa, lars, mmagr, mrunge, pkilambi, rmccabe
Target Milestone: Upstream M3Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-18 15:57:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1581469    

Description Leonid Natapov 2018-10-10 05:12:43 UTC 
After loading procevent collectd plugin for service assurance I am getting multiple error messages.

The conf file i am using for the procevent plugin:

BaseDir "/var/lib/collectd"
PIDFile "/run/collectd.pid"
Interval 1
 
LoadPlugin procevent
 
<Plugin "procevent">
  BufferLength 10000
  RegexProcess "/^.*nova.*$/"
  RegexProcess "/^.*neutron.*$/"
</Plugin>
----


The messages I am getting:
-----
[2018-10-09 17:22:27] plugin_load: plugin "procevent" successfully loaded.
[2018-10-09 17:22:28] procevent plugin: socket receive error: 105
[2018-10-09 17:22:28] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:22:28] read-function of plugin `procevent' failed. Will suspend it for 2.000 seconds.
[2018-10-09 17:22:29] TYPE_VALUES_NAMES_MAPPING for (memcached, df) does not have names, fallback to indexes. Please report a bug to collectd-gnocchi
[2018-10-09 17:22:29] TYPE_VALUES_NAMES_MAPPING for (memcached, memcached_octets) does not have names, fallback to indexes. Please report a bug to collectd-gnocchi
[2018-10-09 17:22:31] procevent plugin: socket receive error: 105
[2018-10-09 17:22:31] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:22:31] read-function of plugin `procevent' failed. Will suspend it for 2.000 seconds.
[2018-10-09 17:22:31] procevent plugin: socket receive error: 105
[2018-10-09 17:22:33] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:22:33] read-function of plugin `procevent' failed. Will suspend it for 4.000 seconds.
[2018-10-09 17:22:35] procevent plugin: socket receive error: 105
[2018-10-09 17:22:37] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:22:37] read-function of plugin `procevent' failed. Will suspend it for 8.000 seconds.
[2018-10-09 17:22:38] procevent plugin: socket receive error: 105
[2018-10-09 17:22:45] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:22:45] read-function of plugin `procevent' failed. Will suspend it for 16.000 seconds.
[2018-10-09 17:22:48] procevent plugin: socket receive error: 105
[2018-10-09 17:23:01] procevent plugin: The interface thread had a problem. Restarting it.
[2018-10-09 17:23:01] read-function of plugin `procevent' failed. Will suspend it for 32.000 seconds.
Comment 1 Andrew Bays 2018-10-10 13:27:04 UTC 
Yesterday I thought this might be due to a difference in netlink headers between the kernel's libraries and the user space libraries, but that doesn't seem plausible now after doing some research.  I found this enlightening comment on a Linux netlink forum where someone was experiencing a similar problem:

"I also don't think it is actually related to the size of the buffer used in the recvfrom() call, because if that's too small it should not result in an error, the message would just get truncated.  Perhaps the socket buffer is running full before the thread can read and process messages."

I noticed that the Collectd configuration in this scenario had an Interval value of 1.  If the system was exceptionally busy in terms of process activity (forking, exiting, starting and everything else netlink reports for processes), then it is possible that the socket buffer was completely filled before the plugin's read thread could get around to reading the messages.  In that case, an errno 105 "no buffer space available" would occur when the thread tried to allocate a buffer on the socket.

All this being said, it would be interesting to try setting the Collect Interval to something much smaller, such as "0.005" (5 ms).  In the meantime, I will try to induce heavy process activity in my local environment and see if I can reproduce the error with a higher Collectd Interval.
Comment 7 Matthias Runge 2019-02-18 15:57:25 UTC 
Based on #c6, I'm going to close this.