Bug 1286684 - running virt-who in big vmware environments results in to fast growing logfiles - candlepin + foreman
Summary: running virt-who in big vmware environments results in to fast growing logfil...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.1.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On: 1313036 1503573
Blocks: 1296845
TreeView+ depends on / blocked
 
Reported: 2015-11-30 13:47 UTC by Marcel Gazdík
Modified: 2019-12-16 05:07 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1313036 (view as bug list)
Environment:
Last Closed: 2018-02-21 17:11:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Marcel Gazdík 2015-11-30 13:47:58 UTC
Description of problem:
  In a big environment the information about each processed machine reported by virt-who leads to large log files.

Version-Release number of selected component (if applicable):
Candlepin and foreman in sat 6.1.X

How reproducible:
  

Actual results:
  There is line for each reported VM in candlepin log and the whole report in the foreman log. 

Expected results:
  Candlepin:
    Instead of having line by host reporting we can have a grouping line like:
      "Processed XX host reported by YY"

  Foreman:
    Do not print the request into log file. I would suggest to have similar behavior like I suggested above. Just to have a note about received report.

The current behavior should be then available in debug mode instead of info mode.

Additional info:
  related bug: 1281715

Comment 2 Christian Horn 2015-12-15 09:17:45 UTC
No changes with 6.1.4.  Any ideas to improve this?

Comment 3 Chris Snyder 2015-12-15 19:30:42 UTC
One way to reduce the log output of candlepin would be to add the following lines to your /etc/candlepin/candlepin.conf file:
"""
log4j.logger.org.candlepin.resource.ConsumerResource.checkForGuestsUpdate=WARN
log4j.logger.org.candlepin.resource.HypervisorResource=WARN
"""

Please note this will stop not only the undesirable log messages but the vast majority of log messages coming from ConsumerResource and HypervisorResource.

Also note that this configuration file might get changed by Katello.

Comment 4 Christian Horn 2015-12-16 10:07:31 UTC
With ConsumerResource=WARN in place, one virt-who commit is only causing these messages:

2015-12-16 10:26:53,967 [req=316948b0-2b97-4f4b-a38b-cf5227f89ea6, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/hypervisors?owner=Vodafone&env=Development/cv-VF-All_Locations-Capsule-RHEL7-Essentials
2015-12-16 10:28:02,794 [req=316948b0-2b97-4f4b-a38b-cf5227f89ea6, org=Vodafone] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=68835

...but we would actually appreciate to see if the action succeeded.

* Setting the following two values in /etc/candlepin/candlepin.conf makes
  candlepin.log be more readable and not growing as much as before:
    log4j.logger.org.candlepin.resource.ConsumerResource=WARN
    log4j.logger.org.candlepin.resource.HypervisorResource=WARN
* After the logging change we miss out on statistics of the update, as we
  only see that the request was processed OK.

We think the following would be best:
- the current messages like 
     "org.candlepin.resource.ConsumerResource - Updating 1 guest IDs."
  is what should be belonging to DEBUG level.
  Suggestion: change the level these are logged to from INFO to DEBUG.
- We could then set loglevel to INFO instead of WARN.
  We would nontheless want to see if things break horribly, so
  "updated X hypervisors and Y consumers" messages could be output to INFO.

With these 2 changes, we could run INFO loglevel, and would have the amount down to reasonable level.

Comment 5 Barnaby Court 2016-06-01 18:44:27 UTC
This has been fixed upstream in Candlepin 2.0.11-1 which is targetted for Sat 6.3

Comment 15 Jan Hutař 2017-10-18 05:51:23 UTC
Note: Candlepin's counterpart of this bug is bug 1313036 which references this commit:

https://github.com/candlepin/candlepin/commit/edcd7ba0ae95edd77bacc6cc6e6c420e1383fcc5

Comment 16 Jan Hutař 2017-10-18 05:58:14 UTC
For Foremanf part, this issue, linked in the case, is probably relevant:

http://projects.theforeman.org/issues/13499

Comment 22 Bryan Kearney 2018-02-21 16:45:19 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336

Comment 23 Bryan Kearney 2018-02-21 17:11:41 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.