Description of problem: Sample from viaq metrics store Sample from metrics store: { "_index": "project.ovirt-metrics-ls-engine1.144e12cb-9ed8-11e7-a4ba-001a4a803f01.2017.09.21", "_type": "com.redhat.viaq.common", "_id": "AV6lGud3RgVbj9Bljir7", "_score": null, "_source": { "kubernetes": { "container_name": "mux-mux", "namespace_id": "144e12cb-9ed8-11e7-a4ba-001a4a803f01", "namespace_name": "ovirt-metrics-ls-engine1", "pod_name": "mux" }, "hostname": "************.redhat.com", "@timestamp": "2017-09-21T17:42:27+02:00", "pipeline_metadata": { "normalizer": { "received_at": "2017-09-21T15:42:36.382421+00:00", "name": "fluentd", "ipaddr6": "************", "inputname": "fluent-plugin-systemd", "version": "0.12.39 1.6.0", "ipaddr4": "***********" } }, "ovirt": { "entity": "engine" }, "tag": "project.ovirt-metrics-ls-engine1", "collectd": {}, <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< MISSING "ipaddr4": "************" }, "fields": { "@timestamp": [ 1506008547000 ], "pipeline_metadata.normalizer.received_at": [ 1506008556382 ] }, "highlight": { "tag": [ "@kibana-highlighted-field@/kibana-highlighted-field@-@kibana-highlighted-field@metrics@/kibana-highlighted-field@-@kibana-highlighted-field@ls@/kibana-highlighted-field@-@kibana-highlighted-field@engine1@/kibana-highlighted-field@" ] }, "sort": [ 1506008547000 ] } Version-Release number of selected component (if applicable): ovirt-engine-metrics-1.1.1-0.0.master.20170919065728.el7.centos.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy ViaQ setup 2. Connect engine to it 3. Check for samples collectd part Actual results: "collectd": {} Expected results: should contain info collected from collectd plugin Additional info: Collectd and fluentd seems to be working without any issues on engine machine. [root@ls-engine1 ~]# service collectd status Redirecting to /bin/systemctl status collectd.service ● collectd.service - Collectd statistics daemon Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/collectd.service.d └─postgresql.conf Active: active (running) since Thu 2017-09-21 16:29:17 CEST; 1h 17min ago Docs: man:collectd(1) man:collectd.conf(5) Process: 22213 ExecStartPre=/usr/share/ovirt-engine-metrics/setup/bin/create_collectd_pg_pass.sh (code=exited, status=0/SUCCESS) Main PID: 22261 (collectd) CGroup: /system.slice/collectd.service └─22261 /usr/sbin/collectd Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "df" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "aggregation" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "processes" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "uptime" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "postgresql" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: plugin_load: plugin "write_http" successfully loaded. Sep 21 16:29:17 test-example.com collectd[22261]: Systemd detected, trying to signal readyness. Sep 21 16:29:17 test-example.com systemd[1]: Started Collectd statistics daemon. Sep 21 16:29:17 test-example.com collectd[22261]: Initialization complete, entering read-loop. Sep 21 16:29:17 test-example.com collectd[22261]: Successfully connected to database engine (user engine) at server localhost:5432 (server version: 9.5.7, protocol version: 3, pid: 22275) [root@ls-engine1 ~]# service fluentd status Redirecting to /bin/systemctl status fluentd.service ● fluentd.service - Fluentd Loaded: loaded (/usr/lib/systemd/system/fluentd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-09-21 16:28:30 CEST; 1h 18min ago Docs: http://www.fluentd.org/ Main PID: 21621 (fluentd) CGroup: /system.slice/fluentd.service ├─21621 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf └─21627 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf Sep 21 16:28:31 test-example.com fluentd[21621]: <server> Sep 21 16:28:31 test-example.com fluentd[21621]: host mux.viaq.com Sep 21 16:28:31 test-example.com fluentd[21621]: port 24284 Sep 21 16:28:31 test-example.com fluentd[21621]: </server> Sep 21 16:28:31 test-example.com fluentd[21621]: </match> Sep 21 16:28:31 test-example.com fluentd[21621]: </ROOT> Sep 21 16:28:31 test-example.com fluentd[21621]: 2017-09-21 16:28:31 +0200 [debug]: listening http on localhost:9880 Sep 21 16:28:31 test-example.com fluentd[21621]: 2017-09-21 16:28:31 +0200 [info]: following tail of /var/log/ovirt-engine/engine.log Sep 21 16:28:32 test-example.com fluentd[21621]: 2017-09-21 16:28:32 +0200 [info]: connection established to mux.viaq.com Sep 21 16:28:32 test-example.com fluentd[21621]: 2017-09-21 16:28:32 +0200 fluent.info: {"message":"connection established to mux.viaq.com"}
The _source was removed for these fields and all of the fields for metrics should be removed according to elastic recommendation. I opened a bug about that: bug 1494912 This prevents seeing the records in the discovery tab but we can still use the fields to create charts and graphs and filter by them. Please let me know if you need assistance in how to verify this RFE.
Hi Shirly, few questions from my side: 1. Why close this bug and open completely new one when it can be just altered/moved to OCP team? 2. Where can we see the data from collectd now in samples, since _source was deprecated? 3. Do you have representation of data from collectd in elasticsearch/kibana somewhere summed up on feature page, as is not included in this one[1]? [2] https://www.ovirt.org/develop/release-management/features/metrics/metrics-store-collected-metrics/
(In reply to Lukas Svaty from comment #2) > Hi Shirly, > > few questions from my side: > > 1. Why close this bug and open completely new one when it can be just > altered/moved to OCP team? As I mentioned this was requested due to elastic recommendation for metrics. It is not a bug. > 2. Where can we see the data from collectd now in samples, since _source was > deprecated? Either use a non ocp setup. Also in graphite for example ypu can;t see the values. You can create a grap and get the values from it. > 3. Do you have representation of data from collectd in elasticsearch/kibana > somewhere summed up on feature page, as is not included in this one[1]? > > [2] > https://www.ovirt.org/develop/release-management/features/metrics/metrics- > store-collected-metrics/
re-opening, if it's deprecated from elasticsearch side than we need to find another way to populated the data so they will be queryable in Kibana Discovery tab. This is necessary for the QA process as well as for the customers for a quick check of data correctness. Also raising priority, as this is currently the only easy way to query data from metrics store and it's blocking only supported metrics store (ViaQ). We have a WA for QA on non-Viaq setup [1], but unless you want to push it to customers, or tell Common Logging team that we won't test the integration with them I suggest finding a solution here. [1] https://github.com/StLuke/ovirt-metrics-store/blob/master/playbooks/single-engine-store.yml
When did you see this is deprecated from elasticsearch? The same behavior is with elastic native metrics solution - metricbeat. The solution to this issue should be pre-defined dashboards.
https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-template.html See the setup.template.settings._source : setup.template.name: "metricbeat" setup.template.fields: "fields.yml" setup.template.overwrite: false setup.template.settings: _source.enabled: false
My mistake, not deprecated but disabled. What can we do about this? At the moment metrics store is unusable without samples not including _source. As you suggested in Bug 1494912 creating a predefined dashboard would solve this.
(In reply to Lukas Svaty from comment #7) > My mistake, not deprecated but disabled. > > What can we do about this? At the moment metrics store is unusable without > samples not including _source. > > As you suggested in Bug 1494912 creating a predefined dashboard would solve > this. For metrics testing please create example dashboard and not via discovery tab.