Bug 1281715
| Summary: | running virt-who in big vmware environments results in to fast growing logfiles | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Christian Horn <chorn> | |
| Component: | virt-who | Assignee: | Radek Novacek <rnovacek> | |
| Status: | CLOSED ERRATA | QA Contact: | gaoshang <sgao> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 6.8 | CC: | egolov, gxing, hsun, jherrman, mgazdik, mshimura, ovasik, rbalakri, rnovacek, sgao, shihliu | |
| Target Milestone: | pre-dev-freeze | |||
| Target Release: | 6.8 | |||
| Hardware: | All | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | virt-who-0.16-1.el6 | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, virt-who was very verbose when writing to its log file. As a result, the log file grew very quickly, especially in large environments. Now, virt-who logs significantly less information in non-debug mode, and the log file thus grows slower.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1292736 1355624 (view as bug list) | Environment: | ||
| Last Closed: | 2016-05-10 23:56:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1292736, 1296845, 1355624 | |||
|
Description
Christian Horn
2015-11-13 09:20:59 UTC
I think it would be reasonable to limit number of VM infos logged without the debug enabled. My suggestion is change the logging format a bit: * without the DEBUG enabled, log something like this: [INFO] Sending update in hosts-to-guests mapping: 4 hypervisor with 42 guest found * with the DEBUG enabled, next log entry with appear containing whole host-to-guest association JSON (basically the same we have now in the INFO entry) This will reduce the log file size without DEBUG enabled significantly. And for debugging purposes full JSON can be enabled. You you agree with this solution? Suggestions are welcome. Moving this bug to the virt-who component. Looks good to me. The current logging is way to verbose for "actually debug is disabled in config". Yep, this looks good. Suggested change is applied upstream: https://github.com/virt-who/virt-who/commit/5109d5c22e18eda8cbfc6649127c515161cced2c Fixed by rebase to virt-who-0.16-1.el6. It hurts us having to wait until rhel6.8GA for this. We are applying https://github.com/virt-who/virt-who/commit/5109d5c22e18eda8cbfc6649127c515161cced2c now manually, but it hurts ofcourse having to track this change for future updates. Will the fixed virt-who be in satellite channels earlier than rhel6.8GA? We don't plan to fix it currently in the satellite channel, but it could be done if needed. I would recommend to alter logrotate configuration for /var/log/rhsm/rhsm.log to control size of that file. (In reply to Radek Novacek from comment #11) > We don't plan to fix it currently in the satellite channel, but it could be > done if needed. Ok. > I would recommend to alter logrotate configuration for > /var/log/rhsm/rhsm.log to control size of that file. We do, but we consider this a workaround as it requires customers first to notice the logging which is more verbose than required (i.e. in having filesystems filling up), and then implement the workaround. We should aim at logging the right amount in the first place for best user experience. We have documented this in a kbase now. Verified it on virt-who-0.16-4.el6.noarch since virt-who won't generate host/guest mapping JSON data in log but show "Sending update in hosts-to-guests mapping for config "env/cmdline": X hypervisors and X guests found" when disable VIRTWHO_DEBUG.
Verified version:
virt-who-0.16-4.el6.noarch
subscription-manager-1.16.8-2.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64
Verified process:
1 Register system to satellite6.1
2 Configure virt-who run at hyperv mode and disable VIRTWHO_DEBUG.
[root@sgi-xe500-01 rhsm]# cat /etc/sysconfig/virt-who | grep -v ^# | grep -v ^$
VIRTWHO_DEBUG=0
VIRTWHO_INTERVAL=2
VIRTWHO_HYPERV=1
VIRTWHO_HYPERV_OWNER=7715246
VIRTWHO_HYPERV_ENV=7715246
VIRTWHO_HYPERV_SERVER=10.73.5.212
VIRTWHO_HYPERV_USERNAME=administrator
VIRTWHO_HYPERV_PASSWORD=Welcome1
3 Restart virt-who service and check virt-who's log
[root@sgi-xe500-01 rhsm]# service virt-who restart
Stopping virt-who: [ OK ]
Starting virt-who: [ OK ]
[root@sgi-xe500-01 rhsm]# tail -f /var/log/rhsm/rhsm.log
2016-02-25 22:25:28,841 [INFO] @virtwho.py:305 - Report for config "env/cmdline" haven't changed, not sending
2016-02-25 22:25:30,134 [INFO] @virtwho.py:630 - Using reporter_id='sgi-xe500-01.rhts.eng.bos.redhat.com'
2016-02-25 22:25:30,137 [INFO] @virtwho.py:722 - Using configuration "env/cmdline" ("hyperv" mode)
2016-02-25 22:25:44,800 [INFO] @subscriptionmanager.py:185 - Sending update in hosts-to-guests mapping for config "env/cmdline": 1 hypervisors and 1 guests found
4 Configure virt-who run at hyperv mode and enable VIRTWHO_DEBUG.
[root@sgi-xe500-01 rhsm]# cat /etc/sysconfig/virt-who | grep -v ^# | grep -v ^$
VIRTWHO_DEBUG=1
VIRTWHO_INTERVAL=2
VIRTWHO_HYPERV=1
VIRTWHO_HYPERV_OWNER=7715246
VIRTWHO_HYPERV_ENV=7715246
VIRTWHO_HYPERV_SERVER=10.73.5.212
VIRTWHO_HYPERV_USERNAME=administrator
VIRTWHO_HYPERV_PASSWORD=Welcome1
5 Restart virt-who service and check virt-who's log
[root@sgi-xe500-01 rhsm]# service virt-who restart
Stopping virt-who: [ OK ]
Starting virt-who: [ OK ]
[root@sgi-xe500-01 rhsm]# tail -f /var/log/rhsm/rhsm.log
2016-02-25 22:18:38,419 [virtwho.main DEBUG] MainProcess(3369):MainThread @subscriptionmanager.py:hypervisorCheckIn:186 - Host-to-guest mapping: {
"564D80F1-61F0-8510-87BF-57F9EC6122D7": [
{
"guestId": "8329390B-8359-F646-9176-27D738580E81",
"state": 5,
"attributes": {
"active": 0,
"hypervisorVersion": "6.3.9600.16404",
"virtWhoType": "hyperv",
"hypervisorType": "hyperv"
}
}
]
}
Result:
1.VIRTWHO_DEBUG enabled, virt-who log will containing whole host-to-guest association JSON
2.VIRTWHO_DEBUG disabled, virt-who log won't containing whole host-to-guest association JSON, it reduce the log file size significantly.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0859.html |