Description of problem: Seen on various times in multiple hosts, /var/log/vdsm.log ownership is changed to root:root. after investigating logs, its seems that log rotate file [1] run at the same time the vdsm.log file changed. disabling log rotate fixed the problem and file ownership isn't changed anymore. [1] /etc/cron.hourly/vdsm-logrotate Version-Release number of selected component (if applicable): How reproducible: it's a race, so it doesn't happen all the time. Steps to Reproduce: 1.install vdsm on a host 2.activate log rotate (or change values to less then 15M) 3.run supervdsm commands Actual results: vdsm.log has root:root ownership. Expected results: vdsm.log stays with vdsm:kvm ownership Additional info:
So if I understand correctly every logrotate the vdsm log changes ownership ?
no it's not. like i said, it's a race and it will happen only if log rotate runs exactly the same time as supervdsm writes to the log. not easy to reproduce, but i can assure that once i disabled log rotate the problems stopped.
Added to tracker patch that supposed to create the new file with vdsm as owner. Eyal - can you add to /etc/logrotate.d/vdsm in the below scope: /var/log/vdsm/*.log { rotate 100 missingok size 15M compress compresscmd /usr/bin/xz uncompresscmd /usr/bin/unxz compressext .xz create 644 vdsm kvm <---- This line. } And continue your usual integration tests. Inform me if this scenario appears again.
it will be better if you can prepare a patch with the change and we'll build a custom vdsm to deploy on all the hosts with it.
I did - please check https://gerrit.eng.lab.tlv.redhat.com/#/c/6519/ - base rhev-3.2
I don't think that logrotate is a solution for this, as I can expect that after clearing /var/log/ programs will work correctly. logrotate happens once in a day or so, and is optional component at most distributions. A better solution is to create the log in proper ownership if running as root.
(In reply to comment #7) > I don't think that logrotate is a solution for this, as I can expect that > after clearing /var/log/ programs will work correctly. > > logrotate happens once in a day or so, and is optional component at most > distributions. > > A better solution is to create the log in proper ownership if running as > root. Indeed, the current solution is raceful - if vdsm.log is removed (by logrotate or otherwise), supervdsm may be the first to create it, as root:root, and silently drop all following vdsm logs. If we cannot as supervdsm to create the file as vdsm:kvm, we should have it write to a log file of its own.
That's right.. The first thought was to separate the logs. I agree that we should separate between logs that different level of users write to it. The case of that patch was to solve error\race that logrotate can cause.. deleting the file manually it something else to handle with..
(In reply to comment #10) > deleting the file > manually it something else to handle with.. No it is not, it is the exact issue, log rotate deletes the files, just like manual delete. Logrotate is support utility which is optional, building a solution around it is not correct.
re-open, happened today on our jenkins machines with vdsm build vdsm-4.10.2-10.0.el6ev.x86_64. seems like the race still exists, had to change the permission manually.
Just reporting in, that this bug hit a RHEL 6.4 Hypervisor added to a POSIXfs Data Center, as a single Hypervisor. Everything was working fine for a day,and then on the second day, I noticed the Hypervisor was in non-responsive state, and the Storage Domain and Data Center was brought down. The Storage Domain and Data Center was brought down probably because the non-responsive Hypervisor was the lone Hypervisor in the Data Center. Investigation showed that the vdsm log file ownership was wrong, and changing it back resolved the issue. ----------------------------------------------------------------- # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.4 (Santiago) # rpm -qa | grep vdsm vdsm-python-4.10.2-1.6.el6.x86_64 vdsm-cli-4.10.2-1.6.el6.noarch vdsm-4.10.2-1.6.el6.x86_64 vdsm-xmlrpc-4.10.2-1.6.el6.noarch # ls -l /var/log/vdsm/vdsm.log* -rw-r--r--. 1 root root 930 Mar 1 22:01 /var/log/vdsm/vdsm.log -rw-r--r--. 1 vdsm kvm 681976 Mar 1 22:01 /var/log/vdsm/vdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 745700 Mar 1 17:01 /var/log/vdsm/vdsm.log.2.xz -rw-r--r--. 1 vdsm kvm 746640 Mar 1 10:01 /var/log/vdsm/vdsm.log.3.xz -rw-r--r--. 1 vdsm kvm 753100 Mar 1 03:01 /var/log/vdsm/vdsm.log.4.xz -rw-r--r--. 1 vdsm kvm 743516 Feb 28 20:01 /var/log/vdsm/vdsm.log.5.xz ----------------------------------------------------------------- - rejy (rmc)
Saggi, Yaniv, What logger configuration does the superVdsm uses ? Can we control the file ownership through the logger configuration ? If this is not the case than I don't see any other solution than separating the log files.
As mentioned since comment 9, the case that causes this is that the log file was deleted manually, and the root process continues the writing before vdsm while both still running and using the already opened logger. The logger doesn't check anything before writing to the file, it creates the file with the user that writes and continue. In that case, if we want to handle manual deletion, the only solution I see is to have 2 logs. Saggi, do you have another point of view to raise?
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag. Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information: * Cause: What actions or circumstances cause this bug to present. * Consequence: What happens when the bug presents. * Fix: What was done to fix the bug. * Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug. For further details on the Cause, Consequence, Fix, Result format please refer to: https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes Thanks in advance.
setting 3.1.z stream flag according to comment 16
(In reply to comment #18) > setting 3.1.z stream flag according to comment 16 Oded, what is the impact on the end user?
(In reply to comment #19) > (In reply to comment #18) > > setting 3.1.z stream flag according to comment 16 > > Oded, what is the impact on the end user? When it happen , the vdsmd service is down until the user chown vdsm:kvm vdsm.log and start the service again .
Couldn't reprocude, closing as VERIFIED as of vdsm-4.10.2-1.10.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html