Bug 1419858 - Provide ovirt_env_name and engine_fqdn to be used in the fluentd.conf on the hosts and engine machines
Summary: Provide ovirt_env_name and engine_fqdn to be used in the fluentd.conf on the ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-metrics
Classification: oVirt
Component: RFEs
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.1.3
: 1.0.4.3
Assignee: Shirly Radco
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On: 1464394
Blocks: oVirt-Metrics-and-Logs 1456459 1462500
TreeView+ depends on / blocked
 
Reported: 2017-02-07 09:09 UTC by Shirly Radco
Modified: 2017-07-06 13:11 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-06 13:11:48 UTC
oVirt Team: Metrics
rule-engine: ovirt-4.1?
rule-engine: exception+
rule-engine: planning_ack?
rule-engine: devel_ack+
lsvaty: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 77756 0 master MERGED Pass ENGINE_FQDN 2017-06-18 19:51:03 UTC
oVirt gerrit 77757 0 master MERGED Add ovirt_env_name 2017-06-18 07:26:09 UTC

Description Shirly Radco 2017-02-07 09:09:27 UTC
Description of problem:

As a requirement of the common data model, each metric needs to have the ovirt environment identifier, so if data from several ovirt environments is collected to the same central metrics store it will be saved to different indexes and can be used to filter by it.

For that we need to add to the metrics the identifier to the ansible script and to the fluentd.conf.

The identifier can be:
1. The ovirt engine fqdn
2. We can generate a random uuid
3. Ask the user to provide the name.

Comment 1 Yaniv Kaul 2017-02-12 16:19:38 UTC
I think we should go with the FQDN.
I'm not sure what we'll do with the UUID, and I don't want to ask the user anything.

I'm aware that FQDNs change from time to time. We'll need to add it to the rename tool later on.

Comment 2 Yaniv Lavi 2017-03-20 10:13:22 UTC
FQDN should also be part of the document as a searchable field in any case.

Comment 3 Rich Megginson 2017-03-20 15:36:34 UTC
(In reply to Yaniv Dary from comment #2)
> FQDN should also be part of the document as a searchable field in any case.

The "hostname" field is one of the default top level fields in the common data model.  ovirt should fill this in with the FQDN.  It is a searchable field.  It is not part of the OpenShift/common logging namespace name, nor part of the Elasticsearch index name.

Comment 4 Shirly Radco 2017-03-20 19:33:34 UTC
(In reply to Rich Megginson from comment #3)
> (In reply to Yaniv Dary from comment #2)
> > FQDN should also be part of the document as a searchable field in any case.
> 
> The "hostname" field is one of the default top level fields in the common
> data model.  ovirt should fill this in with the FQDN.  It is a searchable
> field.  It is not part of the OpenShift/common logging namespace name, nor
> part of the Elasticsearch index name.

Yes. I'll add it to the common data model under the "ovirt" tree.
It would be called ovirt_engine_hostname / ovirt_engine_fqdn.
Yaniv, Which one is preferred or should we add both?

Comment 5 Shirly Radco 2017-03-23 08:17:40 UTC
(In reply to Shirly Radco from comment #4)
> (In reply to Rich Megginson from comment #3)
> > (In reply to Yaniv Dary from comment #2)
> > > FQDN should also be part of the document as a searchable field in any case.
> > 
> > The "hostname" field is one of the default top level fields in the common
> > data model.  ovirt should fill this in with the FQDN.  It is a searchable
> > field.  It is not part of the OpenShift/common logging namespace name, nor
> > part of the Elasticsearch index name.
> 
> Yes. I'll add it to the common data model under the "ovirt" tree.
> It would be called ovirt_engine_hostname / ovirt_engine_fqdn.
> Yaniv, Which one is preferred or should we add both?


1. engine FQDN.
2. ovirt_env_name  -  must be a valid OpenShift namespace identifier - alphanum + "-" character, cannot begin with "-" or a number, and cannot end with a “-”, up to 10 characters.

Comment 6 Yedidyah Bar David 2017-03-23 08:37:13 UTC
(In reply to Shirly Radco from comment #5)
> 1. engine FQDN.

OK.

> 2. ovirt_env_name  -  must be a valid OpenShift namespace identifier -
> alphanum + "-" character, cannot begin with "-" or a number, and cannot end
> with a “-”, up to 10 characters.

Where do we get it from? User configurable? I think that for now we can expect it to be added by the user in the config.yml file, and create a random one if/when we create a tool to help create this file (see bug 1405812). OK?

Comment 7 Shirly Radco 2017-03-23 08:58:13 UTC
ovirt_env_name should be user configurable. Can be in the config.yml file.

If the name will not be complaint with the openshift requirements, the central Fluentd will fail to parse the records and no records will be passed to Elasticsearch.

I think having a name check must be part of this bug.

Not sure if we want to have a default name. Yaniv?

Comment 8 Shirly Radco 2017-03-28 10:55:28 UTC
(In reply to Shirly Radco from comment #7)
> ovirt_env_name should be user configurable. Can be in the config.yml file.
> 
> If the name will not be complaint with the openshift requirements, the
> central Fluentd will fail to parse the records and no records will be passed
> to Elasticsearch.
> 
> I think having a name check must be part of this bug.
> 
> Not sure if we want to have a default name. Yaniv?

At scrub meeting we discussed it and engine-setup needs to ask the name and provide the default.
Yaniv suggests "engine".

Comment 9 Shirly Radco 2017-04-04 08:49:36 UTC
** Update - ovirt_env_name - must be a valid OpenShift namespace identifier - alphanum + "-" character, cannot begin with "-" or a number, and cannot end with a “-”, up to 49 characters.
The length is opentshift limitation for the total text length. Total length should be up to 63 characters. "ovirt-metrics-" is 14 characters.

Comment 10 Shirly Radco 2017-06-12 12:52:58 UTC
We still need to limit the 'ovirt_env_name' to 49 characters. Moving back to New.

Comment 11 Shirly Radco 2017-06-18 07:25:14 UTC
Added a separate bug 1462500, to handle the ovirt_env_name validation.
This bug only handle the ovirt_env_name field population.

Comment 12 Lukas Svaty 2017-06-22 10:04:48 UTC
Is this idea hidden just for functionality with viaq or is it shown in the lables? I am unable to query for such field in kibana.

If its indeed just for communication Im inclined to add CodeChange and do Sanity only. 

Kibana seems to be reporting correctly in ovirt-engine-metrics-1.0.4.2-1.el7ev.noarch

Comment 13 Shirly Radco 2017-06-22 11:04:04 UTC
In the config.yml you should add the ovirt_env_name.
This configures the elasticsearch index names.
The default is "engine" which create ovirt-metrics-engine and ovirt-logs-engine prefixes.

It is saved in the "_index" field as well as under "kubernetes" namespace as "namespace_name".

Example:

"kubernetes": {
      "namespace_name": "ovirt-metrics-engine",
      "pod_name": "mux",
      "namespace_id": "b130f429-3560-11e7-96f0-001a4a23128a",
      "container_name": "mux-mux"
    }

For engine_fqdn, it is available for hosts and vms records.
It is nested under "ovirt" namspace.

Yaniv, does engine_fqdn required also in engine records?

Comment 14 Lukas Svaty 2017-06-22 12:05:03 UTC
Few issues with this one:

1. setting up ovirt_env_name does not make it to samples stored in elasticsearch and previewd by kibana 

my config file:
local_fluentd_ca_cert_path: /root/mux-ca.crt
fluentd_shared_key: secre-tgenerated
fluentd_fluentd_host: lsvaty-vm1.example.com
ovirt_env_name: lsvaty_test-ovirt                 

sample from kibana:

{
  "_index": ".operations.2017.06.22",
  "_type": "com.redhat.viaq.common",
  "_id": "AVzPrnBw-JHtk37G4A56",
  "_score": null,
  "_source": {
    "docker": {
      "container_id": "4d0fb93f0ae1091705331f9ca516e235ddb7c72394f9f80787a1f10dcb028822"
    },
    "kubernetes": {
      "namespace_name": "default",
      "pod_name": "docker-registry-1-m3b4p",
      "pod_id": "3086295e-5660-11e7-94be-001a4a803f01",
      "labels": {
        "deployment": "docker-registry-1",
        "deploymentconfig": "docker-registry",
        "docker-registry": "default"
      },
      "host": "lsvaty-vm1.vm1.example.com",
      "namespace_id": "60747ef7-565e-11e7-94be-001a4a803f01",
      "container_name": "registry"
    },
    "hostname": "lsvaty-vm1.example.com",
    "message": "10.128.0.1 - - [22/Jun/2017:12:02:08 +0000] \"GET /healthz HTTP/2.0\" 200 0 \"\" \"Go-http-client/2.0\"",
    "level": "6",
    "pipeline_metadata": {
      "collector": {
        "ipaddr4": "99.999.9.99",
        "ipaddr6": "ffff::ffff:ffff:ffff:ffff",
        "inputname": "fluent-plugin-systemd",
        "name": "fluentd openshift",
        "received_at": "2017-06-22T12:02:08.945607+00:00",
        "version": "0.12.29 1.4.0"
      }
    },
    "@timestamp": "2017-06-22T12:02:08.945607+00:00"
  },
  "fields": {
    "pipeline_metadata.collector.received_at": [
      1498132928945
    ],
    "@timestamp": [
      1498132928945
    ]
  },
  "sort": [
    1498132928945
  ]
}


2. we do not support hidden options, please document it in config.yml.example

Comment 15 Red Hat Bugzilla Rules Engine 2017-06-22 12:05:09 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 19 Lukas Svaty 2017-06-26 14:26:27 UTC
sample contains the tag, however not in _index as specificed, Shirly?
also we need to specifcy the format of this string, what special characters are supported?? @#$%^&*

{
  "_index": "ovirt-metrics-2017.06.26",  <==== no tag here
  "_type": "fluentd",
  "_id": "AVzkx__1GiB38QjWfBZc",
  "_score": null,
  "_source": {
    "hostname": "ls-engine1.example.com",
    "collectd": {
      "dstypes": [
        "gauge"
      ],
      "interval": 10,
      "plugin": "cpu",
      "plugin_instance": "2",
      "type": "percent",
      "type_instance": "user",
      "cpu": {
        "percent": 1.20240480961924
      }
    },
    "tag": "project.ovirt-metrics-lsvaty_test-ovirt", <===========
    "ovirt": {
      "entity": "engine"
    },
    "ipaddr4": "...",
    "ipaddr6": "::",
    "@timestamp": "2017-06-26T16:21:28+02:00"
  },
  "fields": {
    "@timestamp": [
      1498486888000
    ]
  },
  "sort": [
    1498486888000
  ]
}


In config.yml.example you can see:
[root@ls-engine1 ovirt-engine-metrics]# cat config.yml.example config.yml
# This is an example file for oVirt metrics configuration.
# To use this file, copy it to /etc/ovirt-engine-metrics/config.yml and edit.

# Address of the central fluentd machine
fluentd_fluentd_host: fluentd-server.example.com

# Shared secret on the central fluentd machine
fluentd_shared_key: my_shared_key

# Path to the cert of the CA used to sign central fluentd cert
# To use the engine internal CA, use:
# local_fluentd_ca_cert_path: /etc/pki/ovirt-engine/ca.pem
local_fluentd_ca_cert_path: /path/to/fluentd_ca_cert.pem

# Environment name. Can be used to identify data collected in a single central
# store sent from more than one oVirt engine.
# Maximum field length is 49 characters.
# For more details see:
# https://bugzilla.redhat.com/1419858
# TODO replace above link with some nicer page on ovirt.org
ovirt_env_name: engine


Do we have a bug for this TODO: ?

Comment 20 Shirly Radco 2017-06-26 18:31:47 UTC
Bug 1462500


Note You need to log in before you can comment on or make changes to this bug.