Bug 1732918

Summary: collectd is not reporting VM statistics in metrics store
Product: Red Hat Enterprise Virtualization Manager Reporter: amashah
Component: ovirt-engine-metricsAssignee: Shirly Radco <sradco>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Lukas Svaty <lsvaty>
Severity: high Docs Contact:
Priority: high    
Version: 4.3.4CC: guillaume.pavese, lleistne, pelauter, sradco
Target Milestone: ovirt-4.3.7Keywords: ZStream
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-26 13:54:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Metrics RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description amashah 2019-07-24 17:22:22 UTC 
Description of problem:

collectd is running on RHV-H, metrics-store shows hosts but does not show any VM data.

Initially collectd was failing, due to BZ 1717954 - this was resolved by reinstalling the host and removing /etc/collectd.d/libvirt.conf and restarting collectd. 

Following this collectd is running but with errors [1].

Version-Release number of selected component (if applicable):

rhvm-4.3.4.3-0.1


Actual results:
VM data is not visible in metrics-store

Expected results:
VM data should be in metrics-store

Additional info:
[1]
~~~
* collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-16 17:29:47 EDT; 5 days ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 70410 (collectd)
    Tasks: 11
   CGroup: /system.slice/collectd.service
           `-70410 /usr/sbin/collectd

Jul 20 10:42:07 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/85976/stat" failed.
Jul 20 15:13:47 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/54125/stat" failed.
Jul 20 20:28:17 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/41493/stat" failed.
Jul 20 22:35:07 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/970/stat" failed.
Jul 20 23:16:47 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/16790/stat" failed.
Jul 21 07:47:17 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/45178/stat" failed.
Jul 21 15:19:07 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/18713/stat" failed.
Jul 21 22:04:07 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/52281/stat" failed.
Jul 22 04:35:07 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/52740/stat" failed.
Jul 22 08:07:37 04.removed.local collectd[70410]: read_file_contents: Reading file "/proc/51040/stat" failed.
~~~


~~~
* collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-16 17:29:40 EDT; 5 days ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 80231 (collectd)
    Tasks: 11
   CGroup: /system.slice/collectd.service
           `-80231 /usr/sbin/collectd

Jul 16 17:29:40 02.removed.local collectd[80231]: plugin_load: plugin "aggregation" successfully loaded.
Jul 16 17:29:40 02.removed.local collectd[80231]: plugin_load: plugin "processes" successfully loaded.
Jul 16 17:29:40 02.removed.local collectd[80231]: plugin_load: plugin "write_syslog" successfully loaded.
Jul 16 17:29:40 02.removed.local collectd[80231]: write_syslog plugin: Invalid configuration option: MessageFormat.
Jul 16 17:29:40 02.removed.local collectd[80231]: Systemd detected, trying to signal readyness.
Jul 16 17:29:40 02.removed.local systemd[1]: Started Collectd statistics daemon.
Jul 16 17:29:40 02.removed.local collectd[80231]: virt plugin: reader virt-0 initialized
Jul 16 17:29:40 02.removed.local collectd[80231]: Initialization complete, entering read-loop.
Jul 20 11:37:40 02.removed.local collectd[80231]: write_syslog plugin: send failed with status 32 (Broken pipe)
Jul 20 11:37:40 02.removed.local collectd[80231]: write_syslog plugin: error with wr_send_message
~~~


~~~
* collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-16 17:30:25 EDT; 5 days ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 84138 (collectd)
    Tasks: 11
   CGroup: /system.slice/collectd.service
           `-84138 /usr/sbin/collectd

Jul 22 06:01:27 05.removed.local collectd[84138]: getting the disk params count: Domain not found: no domain with matching uuid 'e5bfe9f9-bca9-45aa-b17f-57fc8dc77b90' (removed)
Jul 22 06:01:27 05.removed.local collectd[84138]: virt plugin: lv_domain_block_info failed
Jul 22 06:01:27 05.removed.local collectd[84138]: virt failed to get stats for block device (vda) in domain removed
Jul 22 06:01:27 05.removed.local collectd[84138]: libvirt: QEMU Driver error : Domain not found: no domain with matching uuid 'e5bfe9f9-bca9-45aa-b17f-57fc8dc77b90' (removed)
Jul 22 06:01:27 05.removed.local collectd[84138]: getting the disk params count: Domain not found: no domain with matching uuid 'e5bfe9f9-bca9-45aa-b17f-57fc8dc77b90' (removed)
Jul 22 06:01:27 05.removed.local collectd[84138]: virt plugin: lv_domain_block_info failed
Jul 22 06:01:27 05.removed.local collectd[84138]: virt failed to get stats for block device (vdb) in domain removed
Jul 22 06:01:27 05.removed.local collectd[84138]: libvirt: QEMU Driver error : Domain not found: no domain with matching uuid 'e5bfe9f9-bca9-45aa-b17f-57fc8dc77b90' (removed)
Jul 22 06:01:27 05.removed.local collectd[84138]: virt plugin: virDomainInterfaceStats failed
Jul 22 06:01:27 05.removed.local collectd[84138]: virt failed to get interface stats for device (vnet1) in domain removed
~~~
Comment 6 Daniel Gur 2019-08-28 13:12:34 UTC 
sync2jira
Comment 7 Daniel Gur 2019-08-28 13:16:46 UTC 
sync2jira
Comment 11 Sandro Bonazzola 2019-09-26 13:54:49 UTC 
Closing with insufficient data. Please reopen if you can provide needed info.
Comment 12 Guillaume Pavese 2020-07-09 18:10:20 UTC 
I am seeing the same bug
I'm running ovirt 4.3.7, with latest rsylog-8.24.0-52 and collectd-5.10.0-2

I also can't make the dashboards work in my own elastic/kibana
As OP, I see the following errors for rsyslogd :


systemd[1]: Starting Collectd statistics daemon...
collectd[21629]: plugin_load: plugin "disk" successfully loaded.
collectd[21629]: plugin_load: plugin "load" successfully loaded.
collectd[21629]: plugin_load: plugin "nfs" successfully loaded.
collectd[21629]: plugin_load: plugin "entropy" successfully loaded.
collectd[21629]: plugin_load: plugin "interface" successfully loaded.
collectd[21629]: plugin_load: plugin "uptime" successfully loaded.
collectd[21629]: plugin_load: plugin "virt" successfully loaded.
collectd[21629]: plugin_load: plugin "cpu" successfully loaded.
collectd[21629]: plugin_load: plugin "memory" successfully loaded.
collectd[21629]: plugin_load: plugin "swap" successfully loaded.
collectd[21629]: plugin_load: plugin "df" successfully loaded.
collectd[21629]: plugin_load: plugin "aggregation" successfully loaded.
collectd[21629]: plugin_load: plugin "processes" successfully loaded.
collectd[21629]: plugin_load: plugin "write_syslog" successfully loaded.
collectd[21629]: plugin_load: plugin "network" successfully loaded.
collectd[21629]: Systemd detected, trying to signal readiness.
systemd[1]: Started Collectd statistics daemon.
collectd[21629]: virt plugin: reader virt-0 initialized
collectd[21629]: Initialization complete, entering read-loop.
collectd[21629]: write_syslog plugin: send failed with status -1 (Connection reset by peer)
collectd[21629]: write_syslog plugin: error with ws_send_message