Bug 1286684 - running virt-who in big vmware environments results in to fast growing logfiles - candlepin + foreman
running virt-who in big vmware environments results in to fast growing logfil...
Status: ON_QA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Candlepin (Show other bugs)
6.1.5
x86_64 Linux
unspecified Severity high (vote)
: GA
: --
Assigned To: satellite6-bugs
Jan Hutař
: Triaged
Depends On: 1503573 1313036
Blocks: 1296845
  Show dependency treegraph
 
Reported: 2015-11-30 08:47 EST by Marcel Gazdík
Modified: 2017-10-18 08:47 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1313036 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Marcel Gazdík 2015-11-30 08:47:58 EST
Description of problem:
  In a big environment the information about each processed machine reported by virt-who leads to large log files.

Version-Release number of selected component (if applicable):
Candlepin and foreman in sat 6.1.X

How reproducible:
  

Actual results:
  There is line for each reported VM in candlepin log and the whole report in the foreman log. 

Expected results:
  Candlepin:
    Instead of having line by host reporting we can have a grouping line like:
      "Processed XX host reported by YY"

  Foreman:
    Do not print the request into log file. I would suggest to have similar behavior like I suggested above. Just to have a note about received report.

The current behavior should be then available in debug mode instead of info mode.

Additional info:
  related bug: 1281715
Comment 2 Christian Horn 2015-12-15 04:17:45 EST
No changes with 6.1.4.  Any ideas to improve this?
Comment 3 Chris Snyder 2015-12-15 14:30:42 EST
One way to reduce the log output of candlepin would be to add the following lines to your /etc/candlepin/candlepin.conf file:
"""
log4j.logger.org.candlepin.resource.ConsumerResource.checkForGuestsUpdate=WARN
log4j.logger.org.candlepin.resource.HypervisorResource=WARN
"""

Please note this will stop not only the undesirable log messages but the vast majority of log messages coming from ConsumerResource and HypervisorResource.

Also note that this configuration file might get changed by Katello.
Comment 4 Christian Horn 2015-12-16 05:07:31 EST
With ConsumerResource=WARN in place, one virt-who commit is only causing these messages:

2015-12-16 10:26:53,967 [req=316948b0-2b97-4f4b-a38b-cf5227f89ea6, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/hypervisors?owner=Vodafone&env=Development/cv-VF-All_Locations-Capsule-RHEL7-Essentials
2015-12-16 10:28:02,794 [req=316948b0-2b97-4f4b-a38b-cf5227f89ea6, org=Vodafone] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=68835

...but we would actually appreciate to see if the action succeeded.

* Setting the following two values in /etc/candlepin/candlepin.conf makes
  candlepin.log be more readable and not growing as much as before:
    log4j.logger.org.candlepin.resource.ConsumerResource=WARN
    log4j.logger.org.candlepin.resource.HypervisorResource=WARN
* After the logging change we miss out on statistics of the update, as we
  only see that the request was processed OK.

We think the following would be best:
- the current messages like 
     "org.candlepin.resource.ConsumerResource - Updating 1 guest IDs."
  is what should be belonging to DEBUG level.
  Suggestion: change the level these are logged to from INFO to DEBUG.
- We could then set loglevel to INFO instead of WARN.
  We would nontheless want to see if things break horribly, so
  "updated X hypervisors and Y consumers" messages could be output to INFO.

With these 2 changes, we could run INFO loglevel, and would have the amount down to reasonable level.
Comment 5 Barnaby Court 2016-06-01 14:44:27 EDT
This has been fixed upstream in Candlepin 2.0.11-1 which is targetted for Sat 6.3
Comment 15 Jan Hutař 2017-10-18 01:51:23 EDT
Note: Candlepin's counterpart of this bug is bug 1313036 which references this commit:

https://github.com/candlepin/candlepin/commit/edcd7ba0ae95edd77bacc6cc6e6c420e1383fcc5
Comment 16 Jan Hutař 2017-10-18 01:58:14 EDT
For Foremanf part, this issue, linked in the case, is probably relevant:

http://projects.theforeman.org/issues/13499

Note You need to log in before you can comment on or make changes to this bug.