Bug 1080107

Jake,

- can you please suply the exact vdsm version,
- is it RHEL or RHEVH hypervisor ?

Barak,

Sorry about that, thought I had included vdsm. Anyways:

vdsm-4.13.2-0.11.el6ev


This does appear on RHEL hosts. On my own RHEV-H hypervisors I do not see libvirt being logged to /var/log at all on RHEV-H 20140217.

Hi Jake,

(In reply to Jake Hunsaker from comment #0)
> Description of problem:
> 
> logrotate is not rotating the /var/log/libvirtd.log logs correctly. However,
> we also see /var/log/libvirt/libvirtd.log 
> 
> Version-Release number of selected component (if applicable):
> 
> rhevm-3.3.1
> libvirt-0.10.2-29.el6_5.4
> 
> How reproducible:
> 100%
> 
> Additional info:
> 
> /var/log/libvirtd.log is not being rotated, however it appears that
> information is now being logged to /var/log/libvirt/libvirtd.log as well
> (and this does appear to be rotating and the logrotate conf file for it
> specifies the right location)

Based on the vdsm version you provided (vdsm-4.13.2-0.11.el6ev) VDSM configure in /etc/libvirt/libvirtd.conf the log_outputs to /var/log/libvirt/libvirtd.log.

In the past, VDSM have been creating /var/log/libvirtd.log to track libvirt
calls. VDSM is stable these days and we don't need to keep tracking libvirt calls and we do not use this log anymore, that's why we do not keep rotating it. In your case, probably your RHEL/vdsm get updated and the old log still there right?

> 
> We have one of two situations:
> 
> 1) /etc/logrotate.d/libvirt is incorrectly pointing to /var/log/libvirt
> rather than /var/log for the libvirtd.log log file, so the one at /var/log
> is not rotating and will grow un-restrained

From RHEV-H 20140217 I can see:

/etc/libvirt/libvirtd.conf pointing log_outputs to /var/log/libvirt/libvirtd.log /etc/logrotate,d/libvirtd also pointing to rotate /etc/logrotate.d/libvirtd
Also checked that the cron job exists: /etc/cron.d/vdsm-libvirt-logrotate

> 
> 2) We are logging in two places, and that needs to be corrected.

We shouldn't, please check /etc/libvirt/libvirtd.conf

If you prefer you can attach the below files that we can confirm:

- /etc/libvirt/libvirtd.conf
- /etc/logrotate.d/libvirtd

Jake, please attach libvirtd.conf. Was this system upgraded from 3.2 to 3.3? If so, I suspect that restarting libvirtd (with `pkill libvirtd`) would convince it to log to the new location.

During upgrade, we should have restarted libvirtd after editing its configuration file. If we failed to do so, that's the bug.

Hi Jake,

Could you please give details how this upgrade/update was made?
Which version of vdsm-4.10.2 you had before?

I tried myself here via yum and it works.

vdsm-4.10.2-30.1
to
vdsm-4.13.2-0.11

Here my test:

I had installed vdsm-4.10.2-30.1 in the host

Then executed:
# yum update ./vdsm-4.13.2-0.11*.rpm
<stop/start> vdsm

# vi /etc/libvirt/libvirtd.conf
## beginning of configuration section by vdsm-4.13.0
<snip>
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
</snip>


Thanks!

Hi Douglas,

The log file size is actually quite big:

$ du -sh  var/log/libvirtd.log.xz 
979M	var/log/libvirtd.log.xz

I tested it also and get the same results as douglas says in comment #6 .
Maybe I miss it, but I can't find in this bugzilla the exact vdsm version that you used before the upgrade. it's specified in vdsm.log if you still have it.

From the versions in comment #6 this bug is not reproducible as far as I see. I tried to modified libvirtd.conf and upgrade and the new location of libvirtd.log was changed properly to /var/log/libvirt/libvirtd.log when starting vdsmd service.

I try to assume the bug is in the specific 4.10.x version that you use. but could be a chance that some other change in other conf file might lead to that. it'll be much nicer to remove the version suspicion before continue with playing with 4.10.x versions and conf files until it'll occur.

Can you check if you have the information please?

Thanks.

(In reply to Yaniv Bronhaim from comment #17)


I no longer have the logs on my local system and they've been purged from the dropbox unfortunately. The cases have also been closed out since the customer stopped responding.

Can you confirm that its not the same case as Bug 1115997 ? do you have the logrotate.d/libvirt configuration file to verify that ?

3.3.4 is out, please retarget to a valid release or request 3.3.5 to be added.

Marking as duplicate with ovirt-node upgrade issue. please reopen if this case also exists in other flows.

*** This bug has been marked as a duplicate of bug 1115997 ***

Same problem after update

from: RHEV-H 6.5 20140407.0.el6ev
to: RHEV-H 6.5 20140624.0.el6ev


Info from old (20140407.0.el6ev) version:

[root@lbch-02 ~]# cat /etc/logrotate.d/libvirtd
# vdsm
/var/log/libvirtd.log {
    rotate 100
    missingok
    copytruncate
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
# end vdsm
/var/log/libvirt/libvirtd.log {
        weekly
        missingok
        rotate 4
        compress
        delaycompress
        copytruncate
        minsize 100k
}

[root@lbch-02 ~]# cat /etc/libvirt/libvirtd.conf |grep log_outputs
# log_outputs="3:syslog:libvirtd"
log_outputs="1:file:/var/log/libvirtd.log"


Info from new (20140624.0.el6ev) version:

[root@lbch-01 ~]# cat /etc/logrotate.d/libvirtd
## beginning of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
    rotate 100
    missingok
    copytruncate
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
## end of configuration section by vdsm
## beginning of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
    rotate 100
    missingok
    copytruncate
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
## end of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
        weekly
        missingok
        rotate 4
        compress
        delaycompress
        copytruncate
        minsize 100k
}

[root@lbch-01 ~]# cat /etc/libvirt/libvirtd.conf |grep log_outputs
# log_outputs="3:syslog:libvirtd"
log_outputs="1:file:/var/log/libvirtd.log"




On updated host (lbch-01, version 20140624.0.el6ev) libvirtd.log is not rotated and /var/log partition was completely filled in 3 days after update...

Hi Milan,

Please give your input to BZ#1115997. We are currently working on that bugzilla.

Thanks,
Uday