Bug 1436087

Summary: [RFE] Update the postgresql collectd plugin queries
Product: [oVirt] ovirt-engine-metrics Reporter: Shirly Radco <sradco>
Component: RFEsAssignee: Shirly Radco <sradco>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Svaty <lsvaty>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 1.0.1CC: bugs, lsvaty, sradco, stirabos, ylavi
Target Milestone: ovirt-4.1.3Keywords: FutureFeature
Target Release: ---Flags: rule-engine: ovirt-4.1+
rule-engine: exception+
ylavi: planning_ack+
rule-engine: devel_ack+
lsvaty: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-06 14:02:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Metrics RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1402901, 1456459    

Description Shirly Radco 2017-03-27 06:40:01 UTC
Description of problem:
We would like to collect the data that will help our users to gain an insight to the postgresql performance and be able to act upon them.

We want to include the following queries:
table_states, disk_io, disk_usage, custom_deadlocks

Comment 1 Lukas Svaty 2017-05-10 12:15:13 UTC
dont see any postgres stats under query:
collectd.plugin: postgresql

tested in ovirt-engine-metrics-1.0.3-1.el7ev.noarch

this is the content of template updated in patch, there is no postgresql specification thus I believe these patches did not get through

[root@ls-engine1 /]# cat /usr/share/ovirt-engine-metrics/setup/ansible/roles/ovirt_collectd/ovirt_engine/templates/builtins-conf-for-ovirt-engine.conf.j2
# The following are added to retrieve additional host info
LoadPlugin disk
#Disk plugin with defaults only

LoadPlugin cpu

<Plugin cpu>
        ReportByState true
        ReportByCpu true
        ValuesPercentage true
</Plugin>

LoadPlugin memory

<Plugin memory>
        ValuesAbsolute true
        ValuesPercentage true
</Plugin>

LoadPlugin load
#Load plugin with defaults only

LoadPlugin nfs
#NFS plugin no config values

LoadPlugin entropy
LoadPlugin swap

LoadPlugin df

<Plugin df>
       # ignore rootfs; else, the root file-system would appear twice, causing
       # one of the updates to fail and spam the log
        FSType rootfs
       # ignore the usual virtual / temporary file-systems
        FSType sysfs
        FSType proc
        FSType devtmpfs
        FSType devpts
        FSType tmpfs
        FSType fusectl
        FSType cgroup
        IgnoreSelected true
        ValuesAbsolute true
        ValuesPercentage true
</Plugin>

LoadPlugin interface

LoadPlugin aggregation

<Plugin aggregation>
   <Aggregation>
     Plugin "cpu"
     Type "percent"

     GroupBy "Host"
     GroupBy "TypeInstance"

     CalculateSum true
     CalculateAverage true
   </Aggregation>
</Plugin>

LoadPlugin processes

<Plugin processes>
     Process "fluentd"
     Process "collectd"
     ProcessMatch "ovirt-engine" "ovirt-engine\.xml"
</Plugin>


moving back to post as this is not available in 4.1.2

Comment 2 Lukas Svaty 2017-06-21 13:18:00 UTC
Can you specify which stats do we expect here?

At the moment only these values are reported:

these value types:
"type": "pg_db_size"
"type": "pg_blks", "
"type": "pg_n_tup_g"
"type": "pg_xact"

last two with different type_instances

Example sample:
{
  "_index": "ovirt-metrics-2017.06.21",
  "_type": "fluentd",
  "_id": "AVzKmJd-GNi9AvMt77i4",
  "_score": null,
  "_source": {
    "hostname": "ls-example.com",
    "collectd": {
      "dstypes": [
        "derive"
      ],
      "interval": 10,
      "plugin": "postgresql",
      "plugin_instance": "engine",
      "type": "pg_blks",
      "type_instance": "heap_hit",
      "postgresql": {
        "pg_blks": 2.69932371916975
      }
    },
    "tag": "project.ovirt-metrics-engine",
    "ovirt": {
      "entity": "engine"
    },
    "ipaddr4": "10.34.63.61",
    "ipaddr6": "2620:52:0:223c:21a:4aff:fe80:3f6c",
    "@timestamp": "2017-06-21T14:20:03+02:00"
  },
  "fields": {
    "@timestamp": [
      1498047603000
    ]
  },
  "highlight": {
    "collectd.plugin": [
      "@kibana-highlighted-field@postgresql@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1498047603000
  ]
}

Also this one should be either ONQA or moved to 4.1.4.
Do we consider storing information in tables (ovirt data)?

Comment 3 Shirly Radco 2017-06-22 07:22:55 UTC
Yes. These are the expected values.
I'll add documentation on all collcted metrics.
Postgres metrics are based on https://github.com/astro/collectd/blob/master/src/postgresql_default.conf

Comment 4 Lukas Svaty 2017-06-22 09:39:50 UTC
thus moving to verified in ovirt-engine-metrics-1.0.4.2-1.el7ev.noarch