Red Hat Bugzilla – Bug 1371530
[RFE][Metrics Store] Install Collectd and fluentd with relevant plugins
Last modified: 2017-03-09 02:46:09 EST
Description of problem: As part of the the metrics store solution we will need to have relevant Collectd packages installed during host deploy.
Moving to ovirt-host-deploy. Didi, if collectd needs to be packaged, please open a separate bug on ovirt-distribution component.
*** Bug 1371524 has been marked as a duplicate of this bug. ***
What is currect status? Will this be part of 4.1?
(In reply to Jiri Belka from comment #3) > What is currect status? Will this be part of 4.1? Yes.
We decided to only install, during host-deploy, and configure separately - using ansible. Changing summary accordingly. The list of packages we need: collectd collectd-netlink collectd-virt collectd-write_http fluentd rubygem-fluent-plugin-rewrite-tag-filter rubygem-fluent-plugin-secure-forward
Still missing fluentd and plugins
Thus you wanna change this to track only collectd and create another bug or wait for the fluentd and plugins pathces?
Moving to MODIFIED, hopefully that's enough. collectd-disk was split into its own package only very recently. If other plugins will go the same path, we'll have to add their packages in later patches/bugs.
So if i understand correctly this "RFE" is just about dependency of engine on these packages so they would be installed. Or are we setting them up some way? Anything specific that should be caught within the testplan?
The current plan is: 1. Install collectd/fluentd + plugins on hosts by host-deploy. That's current bug. 2. Have them updated by the update manager. 3. Have a central metrics store machine, running elasticsearch and fluentd, installed and managed using ocp (openshift). This is handled by the Common Logging team. 4. Provide some means for the user to tell the engine where this metrics store machine is. This can be done using engine-config, and be stored in vdc_options, still no final decision AFAIK. 5. Once the above is ready, configure the hosts using ansible. I am currently working on this one, also using [1] and [2]. I am not aware of open bugs for any of the above except for current bug. Shirly - are there any? Is there anything else I am missing? Please open relevant bugs as needed. Thanks. [1] https://github.com/centos-opstools/opstools-ansible [2] https://gerrit.ovirt.org/66999
Yaniv - please see comment 10. Please open bugs as needed. For (4.) please decide how it should behave, and what values we need to store. Ee do want to use secure forwarding. Current fluentd plugin has several different ways to do this. We need to decide what we want, and perhaps as a result add more values to allow setting/storing. Specifically, if we want to use on the hosts something like: <match **> @type secure_forward shared_key $METRICS_STORE_SECRET self_hostname $HOSTNAME secure true ca_cert_path $METRICS_STORE_CACERT_PATH <server> host $METRICS_STORE_HOST port $METRICS_STORE_PORT # default 24284 </server> </match> Then we need: 1. $METRICS_STORE_HOST 2. $METRICS_STORE_PORT 3. $METRICS_STORE_SECRET - needs to be stored securely, need to discuss this 4. $METRICS_STORE_CACERT_PATH - Basically 3 options: 4.1. Enforce using the engine CA. Then we do not need to ask for this, and the hosts will already have certs signed by it, but the user will have to make fluentd on the metrics store accept this. 4.2. Enforce using the CA on the metrics store. IIUC OCP already has such a thing built in, so will probably be easy to configure fluentd to use it, but then the user will have sign with it all the hosts' certs. 4.3. Use a 3rd party CA. Then user has to do everything - both on the metrics store and on the hosts. 4.4. Allow the user to choose. If so, we should know this and behave accordingly - if using the engine CA, we can use it, but if different, the user has to do stuff on the hosts before ansible configures them. 5. Turn on/off - perhaps we want the user to turn this on/off. "This" is the cron job that will configure the hosts using ansible, not actual metrics forwarding. If you want to allow on/off for metrics forwarding, that's another bug. HOSTNAME will be set by ansible There are other, more complex options, such as using different user/password pairs for different hosts.
We agreed that the engine config tool will contain these parameters values and that there will be a value on if to use the engine CA or it is externally provided. By default the engine CA will be used.
Opened various bugs, you can see details by checking dependencies of the tracker bug 1402901.
Didi are you sure no doc-text is needed for this RFE?
both are installed / updated by package manager on host update, thus verified in rhevm-4.1.0.2-0.1.el7.noarch configuration and deployment handled by BZ#1405813
(In reply to Sandro Bonazzola from comment #14) > Didi are you sure no doc-text is needed for this RFE? Not sure, wrote something anyway.
Emma, considering that in 4.1 we do not have yet "_a_ central metrics store", perhaps change to something like: In this release, when installing or reinstalling hosts, the collectd and fluentd packages are now installed, including the relevant plugins. These can be used to send statistics to a central metrics store.