Description of problem: While following https://access.redhat.com/articles/4921101, the script 'configure_ovirt_machines_for_metrics.sh' fails as RHVH-4.4 is missing 'rsyslog-gnutls' package.  Version-Release number of selected component (if applicable): ovirt-engine-metrics-1.4.2.2-1.el8ev.noarch How reproducible: 100% Steps to Reproduce: 1. Follow https://access.redhat.com/articles/4921101 to send logs to external ELK server from RHV-4.4 2. Run the script 'configure_ovirt_machines_for_metrics.sh' Actual results: Playbook fails with following error: ~~~ Failed to install some of the specified packages <host.example.com> (0, b'', b'') fatal: [host.example.com]: FAILED! => { "changed": false, "failures": [ "No package rsyslog-gnutls available." ], ~~~ Expected results: Script should finish without error Additional info:
On RHV-H host we deliver only packages, which are really required using RPM dependency tree. When I looked at rhel-system-roles it doesn't depend on any RPM package, which is installed by logging linux-system-role: rsyslog https://github.com/linux-system-roles/logging/blob/4610e63d5dca37b605737ae2a4543c2594015eb0/roles/rsyslog/vars/main.yml#L5 rsyslog-gnutls ca-certificates https://github.com/linux-system-roles/logging/blob/4610e63d5dca37b605737ae2a4543c2594015eb0/roles/rsyslog/vars/main.yml#L13 rsyslog-mmnormalize rsyslog-mmjsonparse libfastjson liblognorm libestr https://github.com/linux-system-roles/logging/blob/master/roles/rsyslog/vars/inputs/ovirt/main.yml#L8 @Rich, are there any additional dependencies, which we have missed? To determine which packages should be included in RHV-H package, we use dependency tree of ovirt-host package and here we define additional dependencies for ovirt-engine-metrics: rsyslog rsyslog-elasticsearch rsyslog-mmjsonparse rsyslog-mmnormalize libfastjson liblognorm libestr https://github.com/oVirt/ovirt-host/blob/master/ovirt-host.spec#L100 So we will need to add missing packages there
> When I looked at rhel-system-roles it doesn't depend on any RPM package, which is installed by logging linux-system-role: I don't want to get too far into the weeds here explaining how rhel-system-roles works, because it isn't germane to the problem, but I think I should clear up a couple of things: 1) the rhel-system-roles package is installed _on the controller node_ - the node from which you will run ansible-playbook. So it doesn't make sense to have the rhel-system-roles RPM package have dependencies on things like rsyslog which will pull them into the _controller_ node. The dependencies of things like the logging role are for the _managed_ nodes. So RPM dependencies don't "work" here. 2) If somehow we did include all of the dependencies for all possible optional packages that system roles users could install, it would amount to about 200+ dependencies. > So we will need to add missing packages there The problem is that we (the system roles team) assumed that any/all packages provided by the underlying OS would be available to managed nodes. That is, if you are using the logging role to manage a RHEL7 or RHEL8 managed node, then the full suite of packages for RHEL7 or RHEL8 in the base OS subscription would be available for that managed node. However, we were incorrect in that assumption. For the RHV case, it seems you have a select subset of those packages available for RHV RHEL7 and RHV RHEL8 nodes. So our assumption was wrong. I think the system roles team will have to change the logging role to make sure that rsyslog-gnutls is not installed by default, and make it so that the role will install rsyslog-gnutls will only be installed if its functionality is required.
(In reply to Rich Megginson from comment #8) > > When I looked at rhel-system-roles it doesn't depend on any RPM package, which is installed by logging linux-system-role: > > I don't want to get too far into the weeds here explaining how > rhel-system-roles works, because it isn't germane to the problem, but I > think I should clear up a couple of things: > > 1) the rhel-system-roles package is installed _on the controller node_ - the > node from which you will run ansible-playbook. So it doesn't make sense to > have the rhel-system-roles RPM package have dependencies on things like > rsyslog which will pull them into the _controller_ node. The dependencies > of things like the logging role are for the _managed_ nodes. So RPM > dependencies don't "work" here. My bad, sorry. So do we have a list of top level package dependencies that the role installs on the control node? Then we can make sure that those packages (and of course their RPM dependencies) will be included on RHV-H image. > > 2) If somehow we did include all of the dependencies for all possible > optional packages that system roles users could install, it would amount to > about 200+ dependencies. We are interested only on top level dependencies, all transitional dependencies should be handled by RPM dependencies > > > So we will need to add missing packages there > > The problem is that we (the system roles team) assumed that any/all packages > provided by the underlying OS would be available to managed nodes. That is, > if you are using the logging role to manage a RHEL7 or RHEL8 managed node, > then the full suite of packages for RHEL7 or RHEL8 in the base OS > subscription would be available for that managed node. > > However, we were incorrect in that assumption. For the RHV case, it seems > you have a select subset of those packages available for RHV RHEL7 and RHV > RHEL8 nodes. So our assumption was wrong. > > I think the system roles team will have to change the logging role to make > sure that rsyslog-gnutls is not installed by default, and make it so that > the role will install rsyslog-gnutls will only be installed if its > functionality is required. So again, we don't have a problem to add rsyslog-gnutls into RHV-H, we just need to know if this is mandatory dependency or not. But if this is required for secure connections from rsyslog to elasticsearch, then I think we should add it to RHV-H anyway. The only strange thing is that even though our supported configuration described at https://access.redhat.com/articles/4921101 mentions certificates, we haven't seen the issue with missing rsyslog-gnutls before. But the KCS doesn't mention if it was tested with RHEL-H or RHV-H hosts.
(In reply to Martin Perina from comment #9) > (In reply to Rich Megginson from comment #8) > > > When I looked at rhel-system-roles it doesn't depend on any RPM package, which is installed by logging linux-system-role: > > > > I don't want to get too far into the weeds here explaining how > > rhel-system-roles works, because it isn't germane to the problem, but I > > think I should clear up a couple of things: > > > > 1) the rhel-system-roles package is installed _on the controller node_ - the > > node from which you will run ansible-playbook. So it doesn't make sense to > > have the rhel-system-roles RPM package have dependencies on things like > > rsyslog which will pull them into the _controller_ node. The dependencies > > of things like the logging role are for the _managed_ nodes. So RPM > > dependencies don't "work" here. > > My bad, sorry. So do we have a list of top level package dependencies that > the role installs on the control node? Then we can make sure that those > packages (and of course their RPM dependencies) will be included on RHV-H > image. The packages needed for the control node should already be listed as "Requires:" in the rhel-system-roles RPM package. The packages needed for the managed nodes would be extremely difficult to parse out of the roles. The packages needed for the logging role should not have changed to break the RHV use case, and we will revert it. > > > > > 2) If somehow we did include all of the dependencies for all possible > > optional packages that system roles users could install, it would amount to > > about 200+ dependencies. > > We are interested only on top level dependencies, all transitional > dependencies should be handled by RPM dependencies See above. > > > > > > So we will need to add missing packages there > > > > The problem is that we (the system roles team) assumed that any/all packages > > provided by the underlying OS would be available to managed nodes. That is, > > if you are using the logging role to manage a RHEL7 or RHEL8 managed node, > > then the full suite of packages for RHEL7 or RHEL8 in the base OS > > subscription would be available for that managed node. > > > > However, we were incorrect in that assumption. For the RHV case, it seems > > you have a select subset of those packages available for RHV RHEL7 and RHV > > RHEL8 nodes. So our assumption was wrong. > > > > I think the system roles team will have to change the logging role to make > > sure that rsyslog-gnutls is not installed by default, and make it so that > > the role will install rsyslog-gnutls will only be installed if its > > functionality is required. > > So again, we don't have a problem to add rsyslog-gnutls into RHV-H, we just > need to know if this is mandatory dependency or not. But if this is required > for secure connections from rsyslog to elasticsearch, then I think we should > add it to RHV-H anyway. The only strange thing is that even though our > supported configuration described at > https://access.redhat.com/articles/4921101 mentions certificates, we haven't > seen the issue with missing rsyslog-gnutls before. But the KCS doesn't > mention if it was tested with RHEL-H or RHV-H hosts. I believe rsyslog-gnutls is not a requirement for RHV-H, so we will change the logging role so that it is not installed by default, but only when the user chooses some functionality that requires it. I believe right now that this is only imfwd/omfwd/imrelp/omrelp, which RHV-H does not use. I believe omelasticsearch uses openssl for crypto, not gnutls. In addition, we should add a test to rsyslog that will check to see if we have inadvertently changed the packages we install by default, so that this doesn't happen again.
(In reply to Rich Megginson from comment #11) > I believe rsyslog-gnutls is not a requirement for RHV-H, so we will change > the logging role so that it is not installed by default, but only when the > user chooses some functionality that requires it. I believe right now that > this is only imfwd/omfwd/imrelp/omrelp, which RHV-H does not use. I believe > omelasticsearch uses openssl for crypto, not gnutls. Confirmed. The package rsyslog-gnutils is not used for sending logs to Elasticsearch. One possibility is RHV is getting the metrics from collectd via imtcp. It could enable gnutls. But currently, it's not used nor available. (To make it happen, we need the enhancement in the logging role. That's being said rsyslog-gnutls package is not needed for RHV.) > In addition, we should add a test to rsyslog that will check to see if we > have inadvertently changed the packages we install by default, so that this > doesn't happen again. +1
(In reply to Noriko Hosoi from comment #12) > (In reply to Rich Megginson from comment #11) > > I believe rsyslog-gnutls is not a requirement for RHV-H, so we will change > > the logging role so that it is not installed by default, but only when the > > user chooses some functionality that requires it. I believe right now that > > this is only imfwd/omfwd/imrelp/omrelp, which RHV-H does not use. I believe > > omelasticsearch uses openssl for crypto, not gnutls. > > Confirmed. The package rsyslog-gnutils is not used for sending logs to > Elasticsearch. > > One possibility is RHV is getting the metrics from collectd via imtcp. It > could enable gnutls. But currently, it's not used nor available. (To make it > happen, we need the enhancement in the logging role. That's being said > rsyslog-gnutls package is not needed for RHV.) > > > In addition, we should add a test to rsyslog that will check to see if we > > have inadvertently changed the packages we install by default, so that this > > doesn't happen again. > > +1 OK, thanks for confirmation. We will change this bug as a testonly bug and waiting for a fix from rhel-system-roles package
Verified on: ovirt-engine-4.4.6.6-0.10.el8ev.noarch ovirt-engine-metrics-1.4.2.2-1.el8ev.noarch rhv-4.4.6-7 Metrics automation ran successfully on rhvh hosts (collectd,rsyslog and configuration were successfully set in the hosts)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: RHV Manager security update (ovirt-engine) [ovirt-4.4.6]), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2179